Mining NSS data for enhancement An example from Art & Design Mantz Yorke [email protected] Higher Education Academy Conference “Higher Education

Mining NSS data for enhancementAn example from Art & Design

Mantz [email protected]

Higher Education Academy Conference“Higher Education Surveys for Enhancement”

Nottingham, 18 May 2010

This presentation illustrates one way in which NSS data have been used

in the interests of enhancement

It ought to be adaptable to other situations

Art & Design makes a substantial contribution to the national economy.

It is a national success story.

Yet when it is viewed through the lens of the NSS, it does not come out well in comparison with other subject areas (save in respect of Assessment & Feedback).

Question 1: Why?

The paradox of Art & Design

Judged by NSS scores, some institutions do much better than others.

Question 2: Why?

Variation in the provision of Art & Design

I am critical of the NSS, on various conceptual and technical grounds.

It is problematic as a basis for informing the public.

It is however more useful in its ancillary role as a prompt to enhancing the student experience. • Even if it’s a gross ‘measure’, we can still get messages from it.

Institutions are ill-advised to ‘chase the ratings’: rather, they need to look underneath the scores to identify points at which enhancement-oriented activity might most profitably be targeted.

Clearing some (substantial) brushwood

The NSS is part of a political agenda: it is here to stay (for the time being, at least – but the focus of interest is beginning to switch to student engagement, following the use of NSSE in the US and AUSSE in Australia).

So we have to live with it, even if we don’t like it.

Politics

Art & Design and the NSS

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q220

20

40

60

80

100

Biology (25)English (56)Fine Art (44)Design Studies (54)Cine & Photo (35)

Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS

Selected subject comparisons, NSS 2009(New universities and colleges in England and Wales)

Tim

etab

leC

hang

esO

rgan

isat

ion

Expl

aini

ngIn

tere

stin

gEn

thus

iast

icSt

imul

atin

g

Prom

ptD

etai

led

Cla

rifie

d

Spec

ialis

t equ

ipm

ent

NB Means are unweighted


20

40

60

80

100

Biology (25)English (56)Fine Art (44)Design Studies (54)Cine & Photo (35)


Selected subject comparisons, NSS 2009(New universities and colleges in England and Wales)

Tim

etab

leC

hang

esO

rgan

isat

ion

Expl

aini

ngIn

tere

stin

gEn

thus

iast

icSt

imul

atin

g

Prom

ptD

etai

led

Cla

rifie

d

Spec

ialis

t equ

ipm

ent

NB Means are unweighted

GLAD, the HE Academy ADM Subject Centre and the HEAD Trust combined in supporting a study of the reasons underlying the relatively poor performance of Art & Design since the inception of the NSS.

The study had quantitative and qualitative aspects: only the former is represented in this presentation. (See the HEA-ADM Subject Centre website for the report of the study.)

The NSS and Art & Design – a case study

Disaggregating to institutions withstudio-based provision

Design Studies is used as an illustration: similar results are found for

Fine Art and Cine & Photo Studies

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 660

20

40

60

80

100

Overall Satisfaction (Q22), Design Studies, 2009% agree

Specialist A&D institution

• Some institutions do (consistently) better than others• No ‘specialist institution effect’• No particular size effect

Digging deeper into the NSS data

Three examples


20

40

60

80

100

PQRST

Prom

ptD

etai

led

Cla

rifie

d

Tim

etab

leC

hang

esO

rgan

isat

ion

Expl

aini

ngIn

tere

stin

gEn

thus

iast

icSt

imul

atin

g

Spec

ialis

t equ

ipm

ent

Design Studies, selected comparisons (NSS 2009)


Nov

el p

robl

ems

Cle

ar c

riter

ia


20

40

60

80

100

GHJKLMN

Fine Art: selected comparisons (NSS 2009)


Prom

ptD

etai

led

Cla

rifie

d

Tim

etab

leC

hang

esO

rgan

isat

ion

Expl

aini

ngIn

tere

stin

gEn

thus

iast

icSt

imul

atin

g

Spec

ialis

t equ

ipm

ent

Nov

el p

robl

ems


20

40

60

80

100

ABCDEF

English: selected comparisons (NSS 2009)


Prom

ptD

etai

led

Cla

rifie

d

Tim

etab

leC

hang

esO

rgan

isat

ion

Expl

aini

ngIn

tere

stin

gEn

thus

iast

icSt

imul

atin

g

Spec

ialis

t equ

ipm

ent

Nov

el p

robl

ems

Some possible determinants of the NSS outcomes

Specific institutional and/or departmental causes (e.g. relocation, staffing difficulties).

Programme leadership and/or management.

Pedagogic approach in Art & Design.

Curricular positioning of Art & Design in (some) multi-disciplinary institutions.

Balance of FT and PT staff.

Student expectations v. experiences: ‘The Deal’.

A&D students having to be more self-organising than many others as regards booking spaces and resources; self-organising may cut across formal timetabling.

Some methodological points

The illustrations are based on raw data.

‘Residuals’ make allowances for various institutional factors, such as entry profile, but their interpretation and presentation to others require statistical sophistication.

The raw data may be rougher than one would like, because they are affected by local considerations and the limits of the NSS as an instrument.

One year’s NSS scores may therefore be unreliable.

Nevertheless, rough and ready data may be ‘good enough’ to act as prompts to enhancement.

Warning

Shaping the data into a usable form is a bit of a slog.

In conclusion

NSS data show relative strengths and weaknesses.

The ‘quality literature’ asserts that there’s almost always something that can be improved, by

• doing things better or • doing things differently.

The important thing is to identify those aspects of provision where enhancement activity is most likely to pay off, and to focus on them.

There are unlikely to be many quick fixes.

Documents

Mining NSS data for enhancement An example from Art & Design Mantz Yorke [email protected] Higher Education Academy Conference “Higher Education