League Tables:valuable market information

or dangerous nonsense?

Dr Paul Greatrix, Registrar,The University of Nottingham

Dr Tony Rich, Registrar and Secretary,University of Essex

League Tables

• Background

• Who wants to know?• Who are the providers of this valuable

information?

• Mad, bad and dangerous

• Doing it yourself

• Measuring the unmeasurable?

Background

• The US experience

• Regulatory interest

• The Times (1992)

• They sell papers...• ...and, some would suggest, create perverse

incentives

Who wants to know?

• The Government

• The State

• The Funding Councils

• Potential Students

• Alumni

• Parents, teachers, advisors

• Employers

• Overseas sponsors

• Jo Public

• The Universities

Who’s responsible for providing this valuable data?

• The Times

• Sunday Times

• Financial Times

• The Higher

• Daily Telegraph

• The Guardian

• Shanghai Jiao Tong University

• HEFCE

• BBC Online

• others...

• and, of course, Government

Invaluable information

• For all stakeholders

• Intelligent decision-making

• Better than prejudice

• Reflecting the realities of the market place

• We have a right to know

• We aren’t stupid

League tables are a bad thing...

“The silly season that marks the publication of University league tables is nonsensical and illogical. As any New Scientist knows, letters into numbers; quality into quantity won’t go. League tables are simplistic, divisive and undermine the qualitative nature of a University’s work”

But we’ll use them anyway!

“Having said that, I’m not ashamed to report that we came a very creditable 79th overall, with my own department rating a particularly good score for research

- and as I remarked to the Dean, you can’t get much better than that.”

Dangerous...

• Criteria used do not reflect quality of education

• Historical data

• Variation over time

• Scores are institutional averages – mask strengths

• Distorting effect of weightings and scalings and data manipulation

• Many of criteria used are inter-related (ie not independent)

...extremely dangerous...

• Apples and elephants and paperclips

• Perverse incentives

• Hugely political – Government interest

• Open to manipulation

• Spurious precision – there’s no such thing as a good league table

• Serious consequences for universities, departments, staff and students

The Times 2006

1 Oxford2 Cambridge3 Imperial4 LSE5 UCL6 Loughborough7 Bristol8 Warwick9 Bath10 Durham

11 Edinburgh12 Royal Holloway13 Aston14 Nottingham15 York16 Cardiff17= UEA17= King’s19= SOAS, Leicester, St Andrews

Sunday Times 2006

1 Cambridge2 Oxford3 LSE4 Imperial5 UCL6 Warwick7 York8 Durham9 Bristol10 St Andrews

11 Bath12 Nottingham13 King’s14 Edinburgh15 Manchester16 Southampton17 Loughborough18 Exeter19 UEA20 Sheffield

The Guardian 2006

1 Cambridge2 Oxford3 LSE4 UCL5 Imperial6 SOAS7 King’s8 Warwick9 Bath10 Edinburgh

11 Nottingham12 Surrey13 Bristol14 Manchester15 York16 Birmingham17 Leeds18 Goldsmiths19 Aston20 Royal Holloway

Financial Times 2003

1 Cambridge2 Oxford3 Imperial4 LSE5 UCL6 Warwick7 Bristol8 York9 Nottingham10 King’s

11 UMIST12 Bath13 Loughborough14 Edinburgh15 Manchester16 Birmingham17 SOAS18 Leicester19 St Andrews20 Southampton

‘Poll of polls’(Daily Telegraph 2003)

1 Cambridge2 Oxford3 Imperial4 LSE5 Warwick6= Nottingham6= UCL8 York9 Bristol10 Manchester

11 SOAS12 Bath13 Birmingham14 Loughborough15 King’s16 = Durham16= St Andrews18= Southampton18= Edinburgh20 Sheffield

Times HigherWorld Rankings 2006

2 Cambridge (3)3 Oxford (4)9 Imperial (13)17 LSE (11=)25 UCL (28)33= Edinburgh (30)40 Manchester (35)46= King’s (73=)

64= Bristol (49)70= SOAS (103=)73 Warwick (77=)81 Glasgow (101=)90= Birmingham

(143)85 Nottingham (97)99= Queen Mary

(112)

Shanghai Jiao Tong 2006

2 Cambridge (2)10 Oxford (10)23 Imperial (23)26 UCL (26)50 Manchester (53)52 Edinburgh (47)

62 Bristol (64)69 Sheffield (65)79 Nottingham (83)83 King’s (80)90 Birmingham (98)

And a completely different approach:UEL: TQ in the South East 2005

Surplus of good practice over recommendations(in QAA audit reports)

1 East London 42 King's College London 23 Brunel 24 Queen Mary, London 05 Kent 06 Hertfordshire 07 Royal Holloway -18 London South Bank -39 Greenwich -610 Essex -611 Anglia Polytechnic -9

Different approaches 1

• The Times– 9 indicators, z-scoring used– Teaching weighted @ 1.5 (now using only NSS)– Research weighted @ 1.5

• The Financial Times (if they ever do again)– 15 indicators, z-scoring used– Weighting of T and R, both 15%

• The Sunday Times– 9 indicators - student satisfaction weighted x1.5 (NSS);

teaching excellence x1.0– A level scores weighted x2.5 and research weighted x2– Also – Heads’ assessment x1 (up from 0.5 in 2005) and drop-

out rate (variable)

Different approaches 2

• The Guardian– 7 indicators in subject tables (research ratings do not

feature, no longer using TQA but not adopted NSS)– T staff qualifications weighted @ 15% in each subject

table– Entry qualifications and SSR @ 20%– Spend per student and Value-added @ 10%– Destinations @ 17%– ‘inclusiveness’ @ 8%– Overall table in 2006 is an average of subject tables

• The Daily Telegraph– just the one indicator (keeps things nice and simple)

International approaches 1

Shanghai Jiao Tong: Ranking of World Universities

– 6 indicators covering: quality of education; quality of faculty; research output; performance relative to size

– Includes: Nobel and Fields winners among alumni and staff; highly cited researchers; articles in Nature and Science; articles in citation indices.

International approaches 2

Times Higher World University Rankings

– Peer review: 40%

– Citations per Faculty member and SSR: each 20%

– Recruiter review: 10%

– Proportions of international Faculty and students: each 5%

The critical factors(by frequency of appearance in the tables)

• Subject review/teaching quality results/NSS

• A level scores

• Staff:Student Ratios

• Graduate employment rates

• RAE results

• Spending on library and computing

• Degree classifications

• Completion rates

One-offs(only appeared in one UK table)

• Spending on facilities

• Access

• Value-added

• %age of graduates entering professional training

• %age of graduates entering higher degree

• %age of postgraduate research students

• %age of taught postgraduate students

• Income from industry

• Income from research

• Applications to places

• Inclusiveness

An indicator too far(not covered in the UK tables - yet)

• Alumni giving

• Academic staff pay

• Percentage of full-time academic staff

• Citations

• Brand impact

But international tables will increasingly influence methodologies of UK tables

Conclusions

• They aren’t going to go away

• The international dimension will become increasingly significant

• They can and will be used by many different groups – but can be dangerous in the wrong hands

• Handle with great care!