Martin Ince Communications Limited The World University Rankings in 2008

Martin Ince Communications LimitedMartin Ince Communications Limited

The World University The World University Rankings in 2008Rankings in 2008


The Challenges of University The Challenges of University RankingRanking

Presentation by Martin IncePresentation by Martin Ince

Contributing editor, THE, Contributing editor, THE,

QS-Apple 2008 ConferenceQS-Apple 2008 Conference

Yonsei University, KoreaYonsei University, Korea


Times Higher EducationTimes Higher Education

Since 1971Since 1971 Weekly newspaper formerly associated Weekly newspaper formerly associated

with The Times [of London]with The Times [of London] Now in magazine formatNow in magazine format Group including TESGroup including TES Online at Online at www.timeshighereducation.co.uk

since 1994since 1994


Why rank universities?Why rank universities?

Interest in ranking things and peopleInterest in ranking things and people- HospitalsHospitals- SchoolsSchools- Local authoritiesLocal authorities- Rich lists; Britain, world, Asians, Rich lists; Britain, world, Asians,

footballersfootballers- Universities: The TimesUniversities: The Times


Does it matter?Does it matter?

Some of these rankings are mainly Some of these rankings are mainly for entertainment, eg Rich Listsfor entertainment, eg Rich Lists

Others are seriousOthers are serious- Bad marks for school or hospitalBad marks for school or hospital- Likewise for a university departmentLikewise for a university department- Over 1000 “failing” public bodies in Over 1000 “failing” public bodies in

UK – schools, police forces etc UK – schools, police forces etc


National RankingsNational Rankings The TimesThe Times- produced by John O’Leary, former produced by John O’Leary, former

editor of THEeditor of THE- Institutions as well as subjectsInstitutions as well as subjectsCriteria for subjects include:Criteria for subjects include:- Teaching quality- Teaching quality- Research qualityResearch quality- Entry standardsEntry standards- EmployabilityEmployability


National rankings (2)National rankings (2)

Criteria for institutions includeCriteria for institutions include- Teaching standardsTeaching standards- Staff/student ratioStaff/student ratio- Library spendingLibrary spending- Facilities spending Facilities spending - Good degreesGood degrees- JobsJobs- Research Research


The US ComparisonThe US Comparison US News and World Report US News and World Report

“America’s Best Colleges”“America’s Best Colleges”- Mainly about how likely you are to Mainly about how likely you are to

graduategraduate- Also student experience eg class sizeAlso student experience eg class size- However, many other tables eg However, many other tables eg

liberal arts, business, engineering liberal arts, business, engineering collegescolleges

- Likewise McLean’s (Canada) et al Likewise McLean’s (Canada) et al


Richness of dataRichness of data

Main table in US News has 19 main Main table in US News has 19 main columns of datacolumns of data

Likewise Times Good University Likewise Times Good University GuideGuide

How can we do this globally?How can we do this globally? Why would we want to?Why would we want to?


Why world rankings?Why world rankings?

Long overdue: higher education has Long overdue: higher education has always been very internationalalways been very international

Unique position of THE and QSUnique position of THE and QS Universities becoming more globalUniversities becoming more global Knowledge the real factor in Knowledge the real factor in

international competitivenessinternational competitiveness Increasing desire for comparative Increasing desire for comparative

informationinformation


Why world rankings (2)?Why world rankings (2)?

GATSGATS EU and BolognaEU and Bologna 3 million students outside home country3 million students outside home country Forecast to be 5 million by 2010Forecast to be 5 million by 2010 BTA BTA UK as a major source and destinationUK as a major source and destination UK as major collaboratorUK as major collaborator UK universities opening in China and UK universities opening in China and

elsewhereelsewhere


The five flowsThe five flows

GoodsGoods ServicesServices MoneyMoney PeoplePeople InformationInformation


And Tony says soAnd Tony says so

““In 10 years we will think nothing of In 10 years we will think nothing of students going off to university students going off to university anywhere in the world”anywhere in the world”

Tony Blair at the Labour Party Tony Blair at the Labour Party conference, September 2006conference, September 2006


World interestWorld interest

Interest from governments – UK Interest from governments – UK

Treasury, Lambert Treasury, Lambert EU, GermanyEU, Germany Shanghai Jiao TongShanghai Jiao Tong HEEACTHEEACT OECD from 2010OECD from 2010


And of growing importance..And of growing importance..

Soft power conveying cultural Soft power conveying cultural importance of a nationimportance of a nation

Cheaper and more effective than an Cheaper and more effective than an armyarmy

Prestigious universities mean Prestigious universities mean international regardinternational regard

Even among non-academicsEven among non-academics


Soft power (2)Soft power (2)

British Council, Goethe Institut, British Council, Goethe Institut, Instituto Cervantes Instituto Cervantes

China recognises this: Confucius China recognises this: Confucius Institutes, Shanghai rankingsInstitutes, Shanghai rankings

People like the place where they People like the place where they studied: US, UK, Australiastudied: US, UK, Australia

They absorb its valuesThey absorb its values And they pay to be thereAnd they pay to be there


How to do it?How to do it?

Audience includes –Audience includes – Internationally mobile studentsInternationally mobile students Internationally mobile staffInternationally mobile staff Internationally mobile moneyInternationally mobile money- Focus on:Focus on:- TeachingTeaching- ResearchResearch- International orientationInternational orientation


What are we measuring?What are we measuring?

Data on 500 institutions with Data on 500 institutions with significant citationssignificant citations

Have to teach undergraduatesHave to teach undergraduates Have to teach in at least two areasHave to teach in at least two areas


Peer reviewPeer review

Peer review is the way academic Peer review is the way academic value is measured value is measured

We decided to make it the We decided to make it the centrepiece of this rankingcentrepiece of this ranking

It is the least understood aspect of It is the least understood aspect of our workour work

So here is the explanationSo here is the explanation


Peer review (2)Peer review (2)

Total 5,101 over three yearsTotal 5,101 over three years International spreadInternational spread Subject spreadSubject spread Active academicsActive academics 40 per cent of possible score40 per cent of possible score


The peersThe peers

Gathered by QS list buildingGathered by QS list building 41 per cent in Europe41 per cent in Europe 30 per cent the Americas30 per cent the Americas 29 per cent Asia Pacific29 per cent Asia Pacific

• Aggregate no more than three yearsAggregate no more than three years


The questionThe question

Online surveyOnline survey The top 30 universities in the topics The top 30 universities in the topics

they know aboutthey know about- Arts and humanitiesArts and humanities- Social sciencesSocial sciences- ScienceScience- BiomedicineBiomedicine- Technology Technology


PlussesPlusses

SimpleSimple UnderstandableUnderstandable Robust because valid sampleRobust because valid sample Self-correcting if large enough Self-correcting if large enough

samplesample Hard to cheatHard to cheat


PlussesPlusses

Faculty-level response can be seenFaculty-level response can be seen Full subject coverageFull subject coverage Especially arts and humanitiesEspecially arts and humanities Also business, education, lawAlso business, education, law Mitigates citation issueMitigates citation issue


MinusesMinuses

Biases includeBiases include- AgeAge- SizeSize- Name Name

- BeijingBeijing- LoughboroughLoughborough


EmployersEmployers

Another group who know about university Another group who know about university qualityquality

Innovation in 2005Innovation in 2005 Mainly private sectorMainly private sector At 10 per cent of totalAt 10 per cent of total Therefore academics cut from 50 to 40 per Therefore academics cut from 50 to 40 per

centcent Will be improved in future yearsWill be improved in future years 2007 sample 1,4712007 sample 1,471


Employers are:Employers are:

Americas 43 per centAmericas 43 per cent Europeans 32 per centEuropeans 32 per cent Asia Pacific 25 per centAsia Pacific 25 per cent

• Again, maximum three years Again, maximum three years aggregationaggregation


Quantitative measuresQuantitative measures

Aim to measure universities in terms Aim to measure universities in terms ofof

- Student commitmentStudent commitment- Research commitmentResearch commitment- International commitment and International commitment and

competitivenesscompetitiveness


How to do thisHow to do this

Extensive data Extensive data gathering exercisegathering exercise

By UK firm QSBy UK firm QS Mix of data sourcesMix of data sources NationalNational InstitutionalInstitutional Direct contactDirect contact


First quantitative criterion…First quantitative criterion…

Staff/student ratioStaff/student ratio Classic measure of commitment to Classic measure of commitment to

teachingteaching 20 per cent of final score20 per cent of final score


How international?How international?

Two criteria rated at 5 per cent eachTwo criteria rated at 5 per cent each Is this somewhere where people want to Is this somewhere where people want to

be?be? StaffStaff StudentsStudents- Again raises issuesAgain raises issues

- Visiting scholars?Visiting scholars?- EU cross-border students?EU cross-border students?- Doing full courses?Doing full courses?- Geography advantageGeography advantage


CitationsCitations

Classic measure of research qualityClassic measure of research quality Use Scopus dataUse Scopus data


Citations (2)Citations (2)

Citations per staff member to see Citations per staff member to see density of brain powerdensity of brain power

Not citations per paperNot citations per paper Avoids medical and US biasAvoids medical and US bias


Citations (3)Citations (3)

Well-understood biasWell-understood bias- against non-English publicationagainst non-English publicationagainst arts and humanitiesagainst arts and humanitiesagainst national-oriented topics against national-oriented topics

This accounts for the final 20 per centThis accounts for the final 20 per cent


Changes in 2007Changes in 2007

Scopus: better at AsiaScopus: better at Asia Also more transparentAlso more transparent Better at non-English sourcesBetter at non-English sources Deeper Deeper


Changes (2)Changes (2)

Z score:Z score:

Measure distance from centreMeasure distance from centre

Normalise top to 100Normalise top to 100

Effects outliers such as CalTech, LSEEffects outliers such as CalTech, LSE


Changes (3)Changes (3)

Harder to cheatHarder to cheatMain effects in South AsiaMain effects in South Asia

FTEs more rigorously definedFTEs more rigorously defined

These soft changes bigger effect than These soft changes bigger effect than methodology changesmethodology changes

More understanding of odd claimsMore understanding of odd claims


What did we find?What did we find?

HarvardHarvard The US – 57 in top 200The US – 57 in top 200 Yale Oxbridge logjam at 2Yale Oxbridge logjam at 2


But…But…

The top 200 includes universities in The top 200 includes universities in 28 states28 states

US, UK, Australia, NetherlandsUS, UK, Australia, Netherlands Korea, China, Japan, Taiwan, Korea, China, Japan, Taiwan,

Singapore, Hong KongSingapore, Hong Kong Continental EuropeContinental Europe Developing world: Unam, Brazil (2), Developing world: Unam, Brazil (2),

Cape Town 200=Cape Town 200=


International commitmentInternational commitment

US shows up badlyUS shows up badly HKUST top for international staff, LSE HKUST top for international staff, LSE

in sixth placein sixth place London School of Economics top for London School of Economics top for

international studentsinternational students Yale among few US with international Yale among few US with international

staffstaff


Peer reviewPeer review

Berkeley the winnerBerkeley the winner Cambridge, Oxford, Stanford, Cambridge, Oxford, Stanford,

Harvard popularHarvard popular Well-liked universities all over the Well-liked universities all over the

worldworld Little evidence of patriotism biasLittle evidence of patriotism bias US, UK, Australia, Japan, China, US, UK, Australia, Japan, China,

Canada, Singapore dominate the top Canada, Singapore dominate the top 2020


CitationsCitations

Medical faculty is a big plusMedical faculty is a big plus Or major biomedical incomeOr major biomedical income CalTech the winner by some CalTech the winner by some

distance, then Stanford, MIT, ENSdistance, then Stanford, MIT, ENS Big country effect is at work hereBig country effect is at work here


Staff/student ratioStaff/student ratio

Winner CalTech, USWinner CalTech, US US, French, Swiss, etc institutions all US, French, Swiss, etc institutions all

well placedwell placed Harvard shows badly here at 15thHarvard shows badly here at 15th Asian and European universities well-Asian and European universities well-

placedplaced Weak correlation with research – but Weak correlation with research – but

not zeronot zero


Take home messageTake home message

Small variations don’t matter muchSmall variations don’t matter much A position can be gained by many A position can be gained by many

combinations of weakness and combinations of weakness and strengthstrength

eg, many Asian institutions do well eg, many Asian institutions do well despite scoring low on citationsdespite scoring low on citations

Will improve if more visible Will improve if more visible publicationspublications


Things that don’t workThings that don’t work

Library spendingLibrary spending Course costCourse cost CompletionCompletion Entry standardsEntry standards WealthWealth Alumni givingAlumni giving


Comparison with Shanghai Jiao TongComparison with Shanghai Jiao Tong

Not a newspaperNot a newspaper Nobel + Fields prizes Nobel + Fields prizes

Nobel 6/9 in 2007Nobel 6/9 in 2007

Fields 4/6 in 21Fields 4/6 in 21stst century century These used twiceThese used twice Science and NatureScience and Nature Science and Social Science citationsScience and Social Science citations 500 rather than our 200500 rather than our 200


Comparison with Shanghai Jiao TongComparison with Shanghai Jiao Tong

But:But:

Ends up looking rather similar near Ends up looking rather similar near the top the top

Share 133/200Share 133/200


And with HEEACTAnd with HEEACT

Science focusScience focus Of main policy interestOf main policy interest Avoids Nobel Prize biasAvoids Nobel Prize bias Likewise Science and NatureLikewise Science and Nature Use of ESI means some gapsUse of ESI means some gaps

- 127 from 540 in THE/QS- 127 from 540 in THE/QS


ResponseResponse

More work than writing the thingMore work than writing the thing Last year about 30 newspaper Last year about 30 newspaper

articles in Mexico alonearticles in Mexico alone Interest from media, universities etc Interest from media, universities etc

across Europe and Asiaacross Europe and Asia Less from the USLess from the US Political response – Ireland, Malaysia, Political response – Ireland, Malaysia,

SwitzerlandSwitzerland


Job creationJob creation

More people looking at it than doing More people looking at it than doing itit

Leiden, CEPESLeiden, CEPES IREGIREG All highly valuableAll highly valuable


Types of responseTypes of response

Reject the whole ideaReject the whole idea Complain about their positionComplain about their position Think it is about rightThink it is about right Wonder how to do betterWonder how to do better Compare Shanghai et alCompare Shanghai et al


Asia Pacific top universitiesAsia Pacific top universities

Australia, ANU, 16Australia, ANU, 16 Japan, Tokyo, 17Japan, Tokyo, 17 Singapore, NUS, 33Singapore, NUS, 33 China, Peking, 36China, Peking, 36 Hong Kong, Chinese University, 38Hong Kong, Chinese University, 38 New Zealand, Auckland, 50New Zealand, Auckland, 50 Korea, Seoul National, 51Korea, Seoul National, 51 Taiwan, Taiwan National, 102Taiwan, Taiwan National, 102


In the top 200 In the top 200

Australia 12Australia 12 China 6China 6 Hong Kong 4Hong Kong 4 Japan 11Japan 11 Korea 2Korea 2 New Zealand 3New Zealand 3 Singapore 2Singapore 2 Taiwan 1Taiwan 1


Future Change..Future Change..

These are good figuresThese are good figures Compare France (5), Italy (2), Spain Compare France (5), Italy (2), Spain

(1)(1) Japan 11, Germany 12Japan 11, Germany 12 In the top 500 have more institutions In the top 500 have more institutions

that may emergethat may emerge


In top 201-500In top 201-500

TaiwanTaiwan 8 8 366-499366-499 Thailand Thailand 5 5 223-463223-463 Philippines Philippines 2 2 398, 451398, 451 NZ NZ 3 234-319 3 234-319 Malaysia Malaysia 5 246-416 5 246-416


In top 201-500In top 201-500

Korea Korea 9 9 233-493 233-493 Japan Japan 2121 209-484209-484 Indonesia Indonesia 3 3 360-395360-395 IndiaIndia 9 254-492 9 254-492 ChinaChina 7 209-461 7 209-461 Australia 9 205-374Australia 9 205-374


Asian improvementAsian improvement

Already very internationalAlready very international Well-liked by employers and peersWell-liked by employers and peers Adequate staff numbersAdequate staff numbers Citations: more neededCitations: more needed


How to do betterHow to do better

Publish more in the right placesPublish more in the right places Be more internationalBe more international Be better represented academically Be better represented academically

around the worldaround the world Have better employer linksHave better employer links Have enough staff to teach your Have enough staff to teach your

studentsstudents


The future The future Important for staff, students, parents, Important for staff, students, parents,

management, employersmanagement, employers Important for governments – culturally as Important for governments – culturally as

well as economicallywell as economically Important for businessImportant for business Important globally, eg for the EUImportant globally, eg for the EU Management tool:Management tool:- How do we look?How do we look?- Who to talk to?Who to talk to?- What are our ambitions?What are our ambitions?


Future developmentsFuture developments

New dataNew data- Any suggestions?Any suggestions?- Subject specific?Subject specific?- Management indicatorsManagement indicators

Refine existing data, eg from employersRefine existing data, eg from employersMore global reach, eg AfricaMore global reach, eg AfricaNew analysesNew analysesNew entrantsNew entrants


The bookThe book

Published 2006 and again early 2008Published 2006 and again early 2008 Yours from www.topuniversities.comYours from www.topuniversities.com 500+ institutions including articles 500+ institutions including articles

on the top 100 and shorter details on on the top 100 and shorter details on the restthe rest

Data in groups after 200Data in groups after 200


……really the last slidereally the last slide

Thanks toThanks toOrganisers of QS-AppleOrganisers of QS-Apple

Ann Mroz, editor of THEAnn Mroz, editor of THENunzio Quacquarelli, QSNunzio Quacquarelli, QSBen Sowter, QSBen Sowter, QS

Mandy Mok, QS AsiaMandy Mok, QS AsiaStaff of ScopusStaff of Scopus

Documents

Martin Ince Communications Limited The World University Rankings in 2008