UNIVERSITIES EVALUATIONS AND RANKINGS

Philippe VINCKERector of the Université Libre de Bruxelles

Evolution of higher Education

New Actors of higher education and research

Increasing mobility of students and researchers

Accountability of the universities, transparency

Evaluations, comparisons, rankings

Criticisms of the existing rankings (1)

Competencies of the authors of the rankings

Impossibility, for the reader, to reconstruct and verify the results (rankings are not« scientific »)

No information about the goals, the intended uses, the aimed public

Precise definition of « university » : are they all comparable ?

Choice of the criteria and of their relative importance

Research Education Costs Services Social aspects National context, legislation Financial ressources

Choice of the indicators Data validation

Criticisms of the existing rankings (2)

Criticisms of the existing rankings (3)

Bibliometry Quality of the data Discrimination among the scientific fields Different traditions (journals, books, proceedings,

number of authors, time span of valid research) Supremacy of the publications in English Which indicators ? (IF, citation index, h index, …)

Experts Do they exist ? How to choose them ? Which questions ? How to treat the answers ?

Numerical « manipulations » (1)

How is it possible to imagine that complex objects such as universities can be characterized by one number ?

The weighted mean can exclude good candidates

Example : A 41 97B 100 38C 68 68

Curious effects of normalization

(1) (2)

A 2000 500

B 1120 175

C 400 370

D 1600 45

E 880 240

F 160 435

G 1360 110

H 640 305

(1) (2) Global score

A 100 100 100

B 56 35 45,5

C 20 74 47

D 80 9 44,5

E 44 48 46

F 8 87 47,5

G 68 22 45

H 32 61 46,5RANKING : A, F, C, H, E,B,G,D

(1) (2)

A 1600 500

B 1120 175

C 400 370

D 1600 45

E 880 240

F 160 435

G 1360 110

H 640 305

(1) (2) Global score

A 100 100 100

B 70 35 52,5

C 25 74 49,5

D 100 9 54,5

E 55 48 51,5

F 10 87 48,5

G 85 22 53,5

H 40 61 50,5

RANKING : A,D,G,B,E,H,C,F

Before : A, F, C, H, E, B, G, D

One modification of the score of A on one criterion.No change in the scores of the other universities

After : A, D, G, B, E, H, C, F

Inverse ranking !!

Other comments

Rankings are contested but used

Rankings have an influence on reality

Excesses are possible (financial bonus, or incitements,…)

Standardization effect

Conclusions

The rankings relayed by the media are not scientifically valid at this stage

Evaluation of research and higher education is a necessity

But it must be realized by competent people in the context of a clear policy and with explicit goals

There does not exist a unique method applicable in all institutions

Main questions (1)

Wich « objects » ?

Universities (definition?) Education programmes Diploma’s Research centers Research programmes …

Main questions (2)

What does one want to do ?

To compare To select the «  best(s) » To rank To define « homogene » categories To detect strong and weak points To assign ressources …

Main questions (3)

For whom ? External autorities, government, … Potential partners (universities, research centers,

companies,…) External teachers or researchers Potential students Funding agencies Sponsors Public opinion, media Alumni Internal authorities Internal teachers or researchers Internal students …

Main questions (4)

For each « situation » (characterized by the answers to the 3 previous questions):

Which indicators ?

Quality of the data ?

Numerical treatment of the data !

Different approaches for different concrete questions

Choose the « best » education programme for this student ? (« best » for him)

Allocate financial resources to research centers Select the universities which could be « good »

partners for this company Identify the strong points of these universities for

students interested in studies in that field

Necessity of an interactive decision-aid toolbox for each possible user and question.