Sustainability in University Rankings · Sustainability in University Rankings 21 October 2013, Venice (Italy) ... Results are from the research project: “Le università nel nuovo

Sustainability in University Rankings

21 October 2013, Venice (Italy)

Rankings and Sustainability

Stefano PALEARI

University of Bergamo, Rector and CRUI, President

Results are from the research project: “Le università nel nuovo secolo: nuovi modelli per una società in trasformazione”

Team Higher Education UnBG-CRUI

Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability

• Sustainability issues are to be contextualized

– Why do we use measures (rankings) in Higher Education?

– Sustainability in Higher Education

– Measures (Rankings) and Sustainability

Our approach

2


• Efficient allocation of limited resources

• Accountability towards funders and society

– Universities are (also) financed by the state

• Transparency of results

• Main World University Rankings:

– Academic Ranking of World Universities (ARWU) - 2003

– Quacquarelli Symonds (QS) - 2004

– Times Higher Education Ranking (THE) – 2010

Why do we need measures for HE?

3


ARWU

4

Criteria Indicator Weight

Quality of EducationAlumni of an institution winning Nobel Prizes and

Fields Medals10%

Staff of an institution winning Nobel Prizes and

Fields Medals20%

Highly cited researchers in 21 broad subject

categories20%

Papers published in Nature and Science* 20%

Papers indexed in Science Citation Index-expanded

and Social Science Citation Index20%

Per Capita Performance Per capita academic performance of an institution 10%

Total 100%

Quality of Faculty

Research Output


QS

5

Criteria Indicator Weight

Teaching Faculty Student Ratio 20%

Academic Peer Review from global survey 40%

Recruiter Review from global survey 10%

Research Citations per Faculty from Sciverse Scopus 20%

Percentage of intanational students 5%

Percentage of intanational staff 5%

Total 100%

Reputation

International Orientation


THE

6

Criteria Indicator Weight 2011

Reputational survey - teaching 15% 15% =

PhD awards per academic 6% 6% =

Undergraduates admitted per academic 4,5% 4,5% =

Income per academic 2,25% 2,25% =

PhD awards/ bachelor's awards 2,25% 2,25% =

Citations ResearchCitations impact (normalised avarage citations per

paper) from Scopus32,5% 30% ↓

Reputational survey - research 19,5% 18% ↓

Research income (scaled) 5,25% 6% ↑

Papers per academic and research staff 4,5% 6% ↑

Ratio of international to domestic staff 0,75% 2,5% ↑

Ratio of international to domestic students 3% 2,5% ↓

Proportion of internationally co-authored research

papers 2% 2,5% ↑

Industry Income - innovationResearch income from industry (per academic

staff)2,5% 2,5% =

Total 100% 100%

Weight 2012

Research - volume, income and

reputation

International Outlook - staff,

students and research

Teaching - the learning

environment


• Methodological issues

– Subjective measures

• i.e.: Subjectivity of weights and indicators

– Biased measures

• Rankings favour hard science and publications in English language

– Reputation vs. Contemporaneous performance measures

• Advantage for big universities

• Advantage for historical universities (Nobel prize winners)

– Measures affected by exogenous factors

• Total income of institutions

Worldwide University Ranking

7


World University Ranking Correlation

• Spearman coefficient:

• High level of correlation among top-100 ranked institutions

(Sample: 137 universities listed in all the three rankings)

8

ARWU 2011 THE 2011 QS 2011

ARWU 2011 1,000

THE 2011 0,684 1,000

QS 2011 0,421 0,614 1,000

Spearman correlation Matrix in year 2011 and among different rankings and among the same ranking in 2011 and 2012

di: difference among the position of the i university

n: number of universities listed

ARWU 0,994

THE 0,957

QS 0,989

2011-2012 Correlation


• Top 10 universities in Times Higher Education Ranking 2012

9

“Best” University according to THE 1/2

Rank Università Students Faculty Student / Faculty

1 California Institute of Technology 2.175 425 5

2 University of Oxford 20.466 4.962 4

2 Stanford University 15.666 1.910 8

4 Harvard University 21.229 2.242 9

5 Massachusetts Institute of Technology (MIT) 10.384 1.009 10

6 Princeton University 7.567 1.172 6

7 University of Cambridge 18.396 2.957 6

8 Imperial College London 15.215 3.594 4

9 University of California, Berkeley 36.142 2.393 15

10 University of Chicago 15.438 2.168 7

Average students number "top 10" 16.268

Average students per faculty "top 10" 7

Avarage students per faculty in Italy 30

Top 10 universities in Times Higher Education Ranking 2012


• Top university by country in THE 2012

“Best” University according to THE 2/2

10

Istituzione Studenti DocentiStudent to

faculty ratio

% studenti

internazionali

% graduate

students

California Institute of Technology 2.130 425 5,0 25,0% 55,4%

University of Oxford 18.703 4.962 3,8 29,2% 35,8%

ETH Zürich 14.116 1.150 12,3 33,1% 50,2%

University of Toronto 71.100 10.650 6,7 10,3% 18,8%

University of Tokyo 27.821 4.719 5,9 7,6% 49,5%

University of Melbourne 34.046 3.595 9,5 28,5% 27,3%

National University of Singapore 30.244 5.252 5,8 36,3% 21,0%

University of Hong Kong 15.863 3.046 5,2 30,0% 41,7%

Karolinska Institute 7.051 1.371 5,1 7,1% 51,6%

Peking University 38.228 2.900 13,2 5,6% 51,0%

Ludwig-Maximilians-Universität München 49.180 3.576 13,8 14,1% 14,0%

Pohang University of Science and Technology 3.217 429 7,5 1,6% 48,0%

Katholieke Universiteit Leuven 32.756 6.781 4,8 15,5% 47,2%

École Normale Supérieure 2.700 800 3,4 11,1% 77,8%

Utrecht University 30.344 2.904 10,4 6,3% 12,9%

25.167 3.504 7,5 17,4% 40,1%


• Leading position of US universities: 17 out of 20, and 57 of

the top 100 in the ARWU 2011/2012

• Considering the top 500 universities, Europe with 202

overtakes America with 182 top-ranked institutions

• In Europe, with a predominant focus on broad-based and

regionally distributed support for a fairly large number of

universities, resources were more evenly distributed

• Key issue: What is Excellence?

– E.g. the University of Göttingen does rank very highly in

rankings, but it is not successful in the German Initiative on

Excellence

Europe vs. US

11


• Use of a sole indicator to evaluate multi-output organisation

– They do not consider emerging aspects as sustainability and

diversity of institutions

• They measure only measurable facts

– Measure only what is measurable, not all what matters

– Describe the whole only through a part

• In sports…

• F1: who determines the result? the driver, the car, the team…

• Does one component really represent the whole?

Why do current rankings fail in measuring sustainability?

12


• If universities are to challenge, rankings should refer to

universities in the same “league”,

• By contrast they do not discriminate:

– Different budget

– Different recruitment procedures

• Human capital is the most important productive factor into a

university!

– Different tuition fees

• Once again… in sports…

– NBA: the same salary cap for all the teams (with fines for those

not complying)

Do rankings improve (a fair) university competition?

13

Stefano Paleari, Venice, 21/10/2013 Rankings and sustainability 14

Funding of European HE

Source EUA’s Public Funding Observatory (June 2012)

• Many countries reduced

public funding to their

tertiary education

systems from the

beginning of the crisis

• Reduction of funding

mainly in the countries

in the south and east of

Europe

Trends in Public Funding to Higher Education Europe over the period 2008-2012


* Including EU Funds

Funding per citizen (2008-2012)

15

Fonte Elaborazione propria dati EUA’s Public Funding Observatory e Word Bank Statistics

CountryFunding 2012 -

(mln €)

Population 2011

('000)

Funding per

citizen

Change 2008-

2012

Change 2008-2012

Inflation-adjusted

Norway 3.621 4.953 731 22,0% 21,0%

Sweden 6.235 9.449 660 22,0% 21,0%

Germany 24.900 81.798 304 23,0% 20,0%

France 19.800 65.434 303 8,8% 6,4%

Iceland 87 319 273 13,0% 7,2%

Ireland 1.236 4.576 270 -20,0% -21,0%

Austria 2.169 8.424 257 15,0% 13,0%

Netherland 3.232 16.693 194 10,0% 7,5%

Spain* 7.258 46.175 157 -9,5% -11,0%

UK 9.815 62.744 156 -10,0% -13,0%

Italy 6.633 60.724 109 -12,0% -14,0%

Croatia 369 4.403 84 5,3% 1,8%

Slovakia 447 5.398 83 2,1% -1,5%

Poland* 3.015 38.534 78 12,0% 8,6%

Czech Reupublic 802 10.496 76 -14,0% -17,0%

Lithuania 189 3.030 62 -19,0% -22,0%

Portugal 602 10.557 57 -1,5% -4,1%

Hungury 542 9.972 54 -20,0% -24,0%

Greece 200 11.300 18 -25,0% -25,0%

Belgium - French Community 585 n.a. 0 19,0% 16,0%


Funding development: Italy

16

Fonte EUA’s Public Funding Observatory


Funding development: other countries

17

Fonte EUA’s Public Funding Observatory


• If rankings have to increase efficiency:

– They need to be an instrument, they should not be the objective!

– Universities should be empowered the instruments to correct

their performances

• i.e.: more autonomy in recruitment

Do rankings help universities to work more efficiently?

18


• To do more with less?

– Sharp declining of academic staff

• Permanent academic staff went back to the level of 2001

– Dramatic declining of non-academic staff

– Increase in Student to Faculty Ratio

Italy: Consequence of funding drop

19

Source: Reprocessing data from MIUR e CINECA database

State universities 2012 2011 2010 2009 2008 Δ Δ %

Bachelor & Master students n.a. 1.625.787 1.665.060 1.684.726 1.699.038 -73.251 -4,3%

Tenure Academic staff 52.456 53.901 55.199 58.307 60.254 -7.798 -12,9%

Full Professors 13.841 14.532 15.169 17.174 18.218 -4.377 -24,0%

Associate Professors 15.435 15.884 16.229 16.858 17.547 -2.112 -12,0%

Tenure Researchers 23.180 23.485 23.801 24.275 24.489 -1.309 -5,3%

Non-tenured researchers 1.770 1.049 732 408 304 1.466 482,2%

Non-academic staff 55.810 57.459 58.966 61.873 69.916 -14.106 -20,2%

Studenti to Faculty Ratio n.a 30 30 29 28


• Advantages

– Better transparency of performance evaluation

– Measure excellence in different missions and levels

– User-oriented

– Evaluation not only at institutional level, but also at cource level

• Drawbacks:

– Still developing (first release in February 2014)

– Costly for institution to collect and provide information and data

Something new: the U-Multiranking

20


• Efficiency oriented

– Including at least one measure which considers both inputs

(staff, funding) and outputs

• Importance of network

– What does really matter? To be the first, or to add value to the

system?

– Importance of diversity

• Promote best practices, rather then the use of synthetic (and

biased) rankings

How can we improve rankings?

21


• Set the «ultimate goal» for the HE system, and set rankings

accordingly

– Is the HE system designed to train the top students, or to

improve the quality of the system as a whole?

– Is the HE system designed to mitigate the tendency to inequality

in the social system?

How can we improve the HE system as a whole?

22


• Sustainability is…

– Efficiency

– The role of the university in the society

– Competing in the same field (classifying universities)

– Competing with the same rules

– Setting long-term goals

Conclusions

23

Documents

Sustainability in University Rankings · Sustainability in University Rankings 21 October 2013, Venice (Italy) ... Results are from the research project: “Le università nel nuovo