Embed Size (px)
Citation preview
Mining NSS data for enhancementAn example from Art & Design
Mantz [email protected]
Higher Education Academy Conference“Higher Education Surveys for Enhancement”
Nottingham, 18 May 2010
This presentation illustrates one way in which NSS data have been used
in the interests of enhancement
It ought to be adaptable to other situations
Art & Design makes a substantial contribution to the national economy.
It is a national success story.
Yet when it is viewed through the lens of the NSS, it does not come out well in comparison with other subject areas (save in respect of Assessment & Feedback).
Question 1: Why?
The paradox of Art & Design
Judged by NSS scores, some institutions do much better than others.
Question 2: Why?
Variation in the provision of Art & Design
I am critical of the NSS, on various conceptual and technical grounds.
It is problematic as a basis for informing the public.
It is however more useful in its ancillary role as a prompt to enhancing the student experience. • Even if it’s a gross ‘measure’, we can still get messages from it.
Institutions are ill-advised to ‘chase the ratings’: rather, they need to look underneath the scores to identify points at which enhancement-oriented activity might most profitably be targeted.
Clearing some (substantial) brushwood
The NSS is part of a political agenda: it is here to stay (for the time being, at least – but the focus of interest is beginning to switch to student engagement, following the use of NSSE in the US and AUSSE in Australia).
So we have to live with it, even if we don’t like it.
Politics
Art & Design and the NSS
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q220
20
40
60
80
100
Biology (25)English (56)Fine Art (44)Design Studies (54)Cine & Photo (35)
Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS
Selected subject comparisons, NSS 2009(New universities and colleges in England and Wales)
Tim
etab
leC
hang
esO
rgan
isat
ion
Expl
aini
ngIn
tere
stin
gEn
thus
iast
icSt
imul
atin
g
Prom
ptD
etai
led
Cla
rifie
d
Spec
ialis
t equ
ipm
ent
NB Means are unweighted
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q220
20
40
60
80
100
Biology (25)English (56)Fine Art (44)Design Studies (54)Cine & Photo (35)
Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS
Selected subject comparisons, NSS 2009(New universities and colleges in England and Wales)
Tim
etab
leC
hang
esO
rgan
isat
ion
Expl
aini
ngIn
tere
stin
gEn
thus
iast
icSt
imul
atin
g
Prom
ptD
etai
led
Cla
rifie
d
Spec
ialis
t equ
ipm
ent
NB Means are unweighted
GLAD, the HE Academy ADM Subject Centre and the HEAD Trust combined in supporting a study of the reasons underlying the relatively poor performance of Art & Design since the inception of the NSS.
The study had quantitative and qualitative aspects: only the former is represented in this presentation. (See the HEA-ADM Subject Centre website for the report of the study.)
The NSS and Art & Design – a case study
Disaggregating to institutions withstudio-based provision
Design Studies is used as an illustration: similar results are found for
Fine Art and Cine & Photo Studies
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 660
20
40
60
80
100
Overall Satisfaction (Q22), Design Studies, 2009% agree
Specialist A&D institution
• Some institutions do (consistently) better than others• No ‘specialist institution effect’• No particular size effect
Digging deeper into the NSS data
Three examples
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q220
20
40
60
80
100
PQRST
Prom
ptD
etai
led
Cla
rifie
d
Tim
etab
leC
hang
esO
rgan
isat
ion
Expl
aini
ngIn
tere
stin
gEn
thus
iast
icSt
imul
atin
g
Spec
ialis
t equ
ipm
ent
Design Studies, selected comparisons (NSS 2009)
Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS
Nov
el p
robl
ems
Cle
ar c
riter
ia
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q220
20
40
60
80
100
GHJKLMN
Fine Art: selected comparisons (NSS 2009)
Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS
Prom
ptD
etai
led
Cla
rifie
d
Tim
etab
leC
hang
esO
rgan
isat
ion
Expl
aini
ngIn
tere
stin
gEn
thus
iast
icSt
imul
atin
g
Spec
ialis
t equ
ipm
ent
Nov
el p
robl
ems
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q220
20
40
60
80
100
ABCDEF
English: selected comparisons (NSS 2009)
Teaching Asst & Fdback Acad Sup Org & Man Learn Res Pers Dev OvS
Prom
ptD
etai
led
Cla
rifie
d
Tim
etab
leC
hang
esO
rgan
isat
ion
Expl
aini
ngIn
tere
stin
gEn
thus
iast
icSt
imul
atin
g
Spec
ialis
t equ
ipm
ent
Nov
el p
robl
ems
Some possible determinants of the NSS outcomes
Specific institutional and/or departmental causes (e.g. relocation, staffing difficulties).
Programme leadership and/or management.
Pedagogic approach in Art & Design.
Curricular positioning of Art & Design in (some) multi-disciplinary institutions.
Balance of FT and PT staff.
Student expectations v. experiences: ‘The Deal’.
A&D students having to be more self-organising than many others as regards booking spaces and resources; self-organising may cut across formal timetabling.
Some methodological points
The illustrations are based on raw data.
‘Residuals’ make allowances for various institutional factors, such as entry profile, but their interpretation and presentation to others require statistical sophistication.
The raw data may be rougher than one would like, because they are affected by local considerations and the limits of the NSS as an instrument.
One year’s NSS scores may therefore be unreliable.
Nevertheless, rough and ready data may be ‘good enough’ to act as prompts to enhancement.
Warning
Shaping the data into a usable form is a bit of a slog.
In conclusion
NSS data show relative strengths and weaknesses.
The ‘quality literature’ asserts that there’s almost always something that can be improved, by
• doing things better or • doing things differently.
The important thing is to identify those aspects of provision where enhancement activity is most likely to pay off, and to focus on them.
There are unlikely to be many quick fixes.
