Richard Holmes€¦ · Web viewRichard Holmes. The QS World University Rankings and THE World University Rankings by Subject Group and Subject. The QS World University Rankings

Richard Holmes

The QS World University Rankings and THE World University Rankings by Subject Group and Subject

The QS World University Rankings

History

The QS World University Rankings began life in 2004 as the Times Higher Education Supplement (THES) – Quacquarelli Symonds (QS) World University Rankings. THES was then a weekly newspaper which was not a part of the Times newspaper although at that time it had the same owner. QS Quacquarelli Symonds Ltd (QS) was a company founded in 1990 that specialised in the recruitment and admission of international students.

The arrangement was that QS collected and processed the data while THES decided on the weighting assigned to the various indicators. In the first edition the rankings gave a weighting of 50% to a survey of academic opinion, 20 % to faculty student ratio, 20 % to citations per faculty, 5 % to international students, and 5% to international faculty.

In 2005 QS reduced the weighting for the academic survey to 40% and added an employer survey with a 10% weighting.

In 2007 QS introduced several changes. They switched from Thomson Reuters to Scopus as a source of bibliometric data. They also introduced Z-scores and switched from head counts to full time equivalents for faculty and student data. After that there was no formal announcement of methodological changes, although there have been some modifications.

At the end of 2009, THE announced that it had ended its partnership with QS and would team up with data and media giant Thomson Reuters to publish the Times Higher Education World University Rankings.

QS inherited the data and methodology of the rankings and has continued to publish them as the QS World University Rankings

Publisher

QS Quacquarelli Symonds, a private company with HQ in the UK and branches in New York, Boston, Washington DC, Paris, Singapore, Stuttgart, Sydney, Shanghai, Johannesburg and Alicante, is a “global provider of specialist higher education and careers information and solutions.”

Frequency

Once a year in September

Publication Mode

Online: www.topuniversities.com. Supplementary material is available with registration. Currently, some material is not available. Analysis and more data are available at http://www.iu.qs.com/.

Ranking Products

World University Rankings, World University Rankings by Subject, Faculty (subject group) Rankings, BRICS, Asian, Latin America, Best Student Cities, Under 50 universities. Arab region pilot rankings.

Target Audience

Emphasises international students, especially graduate students, and sponsors. “Mum and Dad in Mainland China”.

Scope

890 institutions listed in 2014.

Other Activities

Consulting, QS Stars rating system, Grad School Tours.

Accreditation

Accredited by IREG.

Volatility

Between 2013 and 2014 the average top 100 university moved up or down 3.94 places.

Bias

A paper by Christopher Claassen shows a strong bias to the UK, Australia and the Netherlands and against the US.

“UK universities are rated more highly, on average, by the QS and THE ranking systems than they are by the latent trait model,9 and the Webometrics ranking also shows a positive bias toward Spanish universities. These three systems favour their home country universities, on average, by at least .25 points on the latent variable scale: enough for a university outside the top 200 to increase its ranking by 20 to 50 places.”

(Claassen, 2015)

Highest Ranked Russian Institution in World University Rankings 2014

Lomonsov Moscow State University 114

Other Russian institutions ranked (5top100 universities highlighted)

St Petersburg State University 233

Bauman Moscow State Technical University 322

http://www.topuniversities.com/university-rankings/world-university-rankings/2014#sorting=rank+region=+country=193+faculty=+stars=false+search=

http://www.topuniversities.com/

Novosibersk State University 328

Moscow State Institute of International Relations 399

Moscow Institute of Physics and Technology State University 411-420

People’s Friendship University of Russia 471-480

National Research Nuclear University MEPhI (Moscow Engineering Physics Institute) 481-490

St Petersburg Polytechnic State University 481-490

Tomsk State University 491-500

Higher School of Economics 501-550

National Research Tomsk Polytechnic University 501-550

Kazan Federal University 551-600

Ural Federal University 551-600

National Research University Saratov State University 601-650

Southern Federal University 601-650

Far Eastern Federal University 701+

Lobachevsky State University of Nizhniy Novgorod 701+

National University of Science and Technology MISIS 701+

Plekhanov Russian University of Economics 701+

Voronezh State University 701+

Indicators 2014

See here for details of methodology.

Academic Survey

QS claim a total of 63,676 responses in 2014, including those carried over from the previous 2 years. Responses varied by faculty area: from Life Sciences and Medicine with 17.9% to Social Sciences and Management with 30.9%. Respondents are asked to nominate the best universities for research in their subject area.

Respondents are reached through several channels with response rates ranging from 2 to 8%:

Previous respondents 1,724 in 2014

http://www.iu.qs.com/academic-survey-responses/

http://www.topuniversities.com/university-rankings-articles/world-university-rankings/qs-world-university-rankings-methodology

The mailing lists of World Scientific publishing company were used until 2011 and may be used in future

Lists from Mardev, an online publishing company provided 200,000 names An Academic survey sign up facility yielded 25,000 names between 2010 and 2014 Institutional lists of up to 400 names per institution. In 2014 400 institutions supplied lists

with a total of 190,000 names.

QS state that respondents are screened and that no respondent can vote for their declared affiliation. For the subject areas there is a weighting for regions indicated by respondents based on completed responses. International and domestic responses are counted separately and then combined with an 85% weighting for international responses. The count of domestic respondents is adjusted against the number of institutions available for selection in that country and the total response from that country.

Each subject area has a maximum score of 100 and they are combined with an equal weighting.

In 2013, 10.0% of responses were from the US, 6.5% from the UK, 6.3% from Brazil and 1.7% from Russia.

Employer survey

Data comes from several sources: respondents, databases, partners and an employer survey sign up facility. There were 28,759 responses in 2014: 10.3% were from the USA, 6.6% UK, France 5.4% and 2.4% from Russia. Unchanged responses are carried over for 3 years and there is a regional weighting based on completed responses. There is a 70% weighting for international responses followed by square rooting.

Citations per faculty

The citations are from a five year period and are derived from Scopus. Faculty data are Full Time Equivalent and include research and teaching staff. Self-citations are excluded.

Faculty Student Ratio

Faculty and Student data are full time equivalent and are derived from a variety of sources. QS prefer to use data submitted from institutions but will use other sources if necessary.

International students and faculty

Data is derived from a variety of sources. International refers to citizenship.

QS Ranking by Faculty (Subject group)

QS now provides rankings of universities in five faculty or subject areas and in 30 subjects. They use two indicators from the world rankings and two additional indicators with different weightings, as indicated in Table 1 and Table 2. The top 400 universities in each subject area are listed.

http://www.iu.qs.com/2013/09/a-new-approach-to-faculty-areas/#comment-85741

http://www.iu.qs.com/links/

http://www.iu.qs.com/employer-survey-responses/

Table1. Weighting of indicators in QS Faculty/Subject area rankings.

Faculty area Academic survey

Employer survey

Citations per paper

H-index

Arts and humanities

60 20 10 10

Engineering and technology

40 30 15 15

Life sciences and medicine

40 10 25 25

Natural sciences 40 20 20 20Social sciences and management

50 30 10 10

Russian Universities in the QS Faculty/Subject Are Rankings

Arts and Humanities

92 Lomonosov Moscow State University

263 St Petersburg State University

Engineering and Technology



Life Sciences and Medicine


Natural Sciences



210 Novosibersk State University

248 National Research Nuclear University MEPhI

367 Moscow Institute of Physics and Technology State University

Social Science and Management



232 Higher School of Economics

QS Rankings by Subject

Thirty subjects are ranked according to varied weightings of the four indicators used in the Subject area rankings.

Table 2. Indicator weightings QS Subject Rankings

Subject Academic Employer Citations H

Biology 40% 10% 25% 25%

Earth Sciences 40% 10% 25% 25%

Environmental Sci 40% 10% 25% 25%

Medicine 40% 10% 25% 25%

Materials 40% 10% 25% 25%

Pharmacy 40% 10% 25% 25%

Chemistry 40% 20% 20% 20%

Mathematics 40% 20% 20% 20%

Physics 40% 20% 20% 20%

Computer Science 40% 30% 15% 15%

Chemical Eng 40% 30% 15% 15%

Electrical Eng 40% 30% 15% 15%

Mechanical Eng 40% 30% 15% 15%

Agriculture 50% 10% 20% 20%

Economics 50% 30% 10% 10%

Civil Eng 50% 30% 10% 10%

Psychology 50% 30% 10% 10%


Law 50% 30% 5% 15%

Acc & Finance 50% 40% 5% 5%

Politics 50% 40% 5% 5%

Communication 50% 20% 10% 20%

Education 60% 10% 15% 15%

Geography 60% 10% 15% 15%

Statistics 60% 10% 15% 15%

Languages 70% 30% 0% 0%

History 70% 10% 5% 15%

Sociology 70% 10% 5% 15%

Linguistics 80% 10% 5% 5%

Philosophy 80% 10% 5% 5%

English 90% 10% 0% 0%


Performance of Russian universities in QS subject rankings

Modern languages

51-100 Lomonosov Moscow State University

201-250 St Petersburg State University

Philosophy


Computer Science and Information Systems


Electrical and Electronic Engineering


Mechanical Aeronautical and Mechanical Engineering


Biological Sciences


Chemistry



Earth and Marine Sciences


Environmental Sciences


Materials Science

http://www.vsu.ru/russian/ratings/docs/qswu_ranking2014.pdf


Mathematics



201-250 Novosibersk State University

Physics and Astronomy



201-250 Novosibersk State University

211-300 National Research Nuclear University MEPhI

301-350 St Petersburg Polytechnical University

Statistics and Operational Research


The Times Higher Education Subject Rankings

History

In 2009 Times Higher Education (THE) announced that it was ending its partnership with QS and that it would use data supplied and analysed by Thomson Reuters. The THE-QS were “unfit for purpose”, especially the academic survey. The first edition of the new THE rankings came out in 2010 and was subject to much criticism partly because of the elevation of Alexandria University in Egypt to 4 th

place in the world for research impact.

There was a good deal of tweaking for the 2011 rankings and the status of the rankings appears to have recovered, especially among leading research-intensive universities and their supporters in Europe and the United Kingdom.

In November 2014, THE suddenly announced that it was ending its relationship with Thomson Reuters and would use data from Scopus. THE indicated that it was entering a new period of “transparency and accountability” and spoke of “reforms” to the ranking.

So far, based on a “master class” earlier this year in Moscow, it appears that THE will keep the structure and weighting of the current rankings. However, there has already been one possibly significant change. THE have taken over the reputation survey themselves and have apparently succeeded in ensuring a broader participation.

It is not possible to compare the current reputation survey with the previous editions since Thomson Reuters apparently did not allow the release of data on the nationality of respondents. However, THE seem to have got more respondent s from China (13.8% of total respondents), Japan (7.2%) , Russia (5.5%) and Germany (4.8%). The corresponding scores for the QS academic survey are China 1.1%, Japan 2.9%, Russia 1.7% and Germany 3.8%. It also seems to have got more respondents from the arts and humanities.

It is not clear exactly what the effect of this. Tsinghua University has risen from 36 th to 26th place in the THE reputation rankings between 2014 and 2015 and Peking University from 41 st to 32nd. Oddly enough, considering the small number of respondents from China, Peking still does better on the QS reputation survey where it is 19th and Tsinghua 27th.

Tokyo has fallen slightly to 12th place and does worse than it does in the QS survey where it is 7th.

Moscow State University does well in this year’s reputation survey where it is 25 th better than last year and much better than QS, who put it in 83rd place for academic reputation.

There are also signs that THE are considering some changes to the citations indicator. Phil Baty has suggested that something will be done about the problem of papers, many of them in particle physics, with thousands of citations and thousands of contributors . One solution is to just delate such papers, which is what Thomson Reuters did when compiling its list of influential scientific minds. Another is to use fractional counting of contributors or authors, which is an option in the Leiden Ranking.

Summary of the THE World Rankings

“Underpinning the World University Rankings 2014-2015 is a sophisticated exercise in information-gathering and analysis: here we detail the criteria used to assess the global academy's greatest universities

The Times Higher Education World University Rankings are the only global university performance tables to judge research-led universities across all their core missions - teaching, research, knowledge transfer and international outlook.

We employ 13 carefully calibrated performance indicators to provide the most comprehensive and balanced comparisons, which are trusted by students, academics, university leaders, industry and governments.

The methodology for the 2014-2015 World University Rankings is identical to that used since 2011-

2012, offering a year-on-year comparison based on true performance rather than methodological change.”

[my highlights]

(http://www.timeshighereducation.co.uk/world-university-rankings/2014-15/world-ranking/methodology)

Publisher

Times Higher Education, a weekly magazine published in the UK by TES Global. Formerly Times Higher Education Supplement (newspaper format) but never a section of the Times.

Frequency

Once a year in October

Publication Mode

Print, iPhone/iPad and online. www.timeshighereducation.co.uk

Ranking products

World University Rankings, Reputation rankings, Under 50 rankings, Asian rankings, BRICS plus emerging economies rankings, MENA rankings “snapshot.

Target Audience

Leading European research universities, administrators and policy makers in Europe, especially the UK. Currently interested in Asia and emerging economies.

In December 2009 Phil baty wrote:

“We want academics and university administrators to have confidence in a more rigorous and transparent ranking.”

Scope

Over 800 universities assessed. Data provided for 400 universities. Now talking about expansion to include about 1000 institutions and institutions outside a narrow elite.

Other activities

http://www.timeshighereducation.co.uk/

THE World Universities Summit, Africa Universities Summit, MENA Universities Summit, conferences, seminars, masterclasses. It seems that they are talking about integrating rankings, advertising and benchmarking.

Volatility

Between 2013 and 2014 the average place among top 100 universities was 4.34, more than the QS world rankings and more than the Shanghai ARWU in most years. The volatility could be due to the large weighting for the reputation survey, where universities get a handful of votes outside the top 50 or one hundred, the ranking of more universities which changes the averages against which raw scores are benchmarked, the scaling of many indicators, and/or the citation indicator where a single paper can produce a large number of citations.

Bias

The Claassen paper shows a strong bias to the UK and against Brazil, China and continental Europe and is neutral with regard to USA. Possible explanations include reliance until last year on the Thomson Reuters database which is biased towards English language publications, an emphasis on income which is measured by three indicators, a heavy weighting for reputation and international orientation citations.

Top Ranked Russian Institution in 2014

196 Lomonosov Moscow State University (well below QS)

Others

None in the top 400.

Indicators with current weighting

To date it appears that THE are keeping the structure and weighting of their rankings. Note that scores are standardised and converted into Z-scores.

Teaching: The learning environment (30%)

Teaching Reputation Survey 15%. Distributed according to UNESCO count of the number of researchers in each country.

Faculty student Ratio 4.5%. Full time equivalent numbers for students and faculty.

Doctorate to bachelor degrees awarded 2.25%

Doctoral degrees awarded per academic staff 6%

Institutional income per academic staff 2.25%.

Research: Volume Income Reputation (30%)

Research reputation 18%. Distributed according to UNESCO count of the number of researchers in each country.

Research income per staff and normalised for purchasing power parity 6%

Papers per academic and research staff 6%.

Industry income: Innovation (2.5%)

Research income from industry per academic staff.

International outlook (7.5%)

International students as a percentage of total students 2.5%

International faculty as a percentage of total faculty 2.5%

Percentage of publications with International Collaborators 2.5%

Citations: Research Impact (30%)

Citations over a six year period to papers published over a five year period. Normalised by field (250 according to Simon Pratt of Thomson Reuters, may be 300 this year) and year. The citations in a specific year in a specific field are compared with the world average. Until now the Web of Science database has been used. This year it will be the Scopus (Reed-Elsevier) database.

Regional modification (the score for the university is divided by the square root of the score for the whole country). In other words universities benefit from being in a country with low citations. The effect of this was seen in 2011 when Hong Kong universities suffered after being counted separately rather than as a part of China.

Comments on Methodology

THE originally hoped to reduce the reliance on reputation data but found that it was difficult to get reliable data from institutions.

THE expects universities to submit data on an annual basis and will not include those who fail to provide information in contrast to QS who will make up the gaps with old data or from third party sources.

Several universities do not supply data about research income from industry and there is formula for making up the gap.

Eleven of the indicators are grouped into clusters of five, three and three. A combined score is given to the public but detailed information is provided only to the institutions themselves. There is a serious reduction in transparency since it can be difficult to work out exactly what has contributed to a rising or falling score for teaching or research.

THE Rankings by Subject-Group

So far, THE has only published subject-group rankings, not subject rankings. It would not be a surprise if THE did start adding subject rankings to its menu in the near future.

THE have six subject groups compared to five for QS with medical sciences classified separately.

Methodology

See here for details. 100 ranked in each subject area, compared to 400 for the QS faculty/subject group rankings.

It is possible that specialised institutions may be ranked in the subject group rankings but not in the overall rankings. In the overall rankings universities are excluded if they teach only one subject, do not teach undergraduates or produce less than 200 papers per year. For the subject group rankings the threshold drops to 50 or 100 papers per year.

Indicators are the same as for the World University Rankings but with different weightings.

http://www.timeshighereducation.co.uk/world-university-rankings/2014-15/subject-ranking/subject/arts-and-humanities/methodology

For the arts and humanities, the indicators in the Teaching: the learning environment and the Research groups go up to a 37.5% weighting each and citations go down to 30 %. For the three science groups, citations go up to 35%. For Engineering and Technology, 2.5% is taken from citations and given to research income from industry. For the social sciences, 5% is taken from citations and given to teaching and research.

Table 3. From Times Higher Education.


Performance of Russian Universities

Engineering and Technology


Physical sciences


85 Novosibirsk State University

95 National Research Nuclear University MEPhI

Similarities and Differences

The QS and THE subject rankings both follow the basic principal of using the indicators of their world rankings but with significant modifications.

QS uses two world rankings indicators, the academic survey and the employer survey, and combines these with two others, h-index and citations per paper. The result is that the subject group and subject rankings are heavily biased towards research.

THE uses all the indicators in their world rankings but weights them differently for the various subject groups.

The QS rankings continue to emphasise the academic survey which has a weighting ranging from 40% for biology and medicine to 90% for English. A rankings strategy that targets the subject group or subject rankings should take account of the different indicators and the different weightings it has for various subject groups and subjects.

In addition, The QS surveys are also open to influence by institutions for example by emphasising certain disciplines when submitting survey lists.

In the THE rankings the citations indicator has a disproportionate influence, firstly because it has a large weighting in all subject groups and secondly because the regional modification further increases its impact. Its impact is greater for three subject groups, Life Sciences, Physical sciences, and Clinical, pre-Clinical and Health.

Since the QS rankings have not until now used field normalisation (there has been talk about normalisation for 5 subject groups) institutions with medical schools and research centres do very well. The THE citations indicator rewards institutions that contribute to a few multi author massively cited projects, mainly in particle physics but also in astronomy, genetics and medicine.

A subject-based strategy should take note of the prominence of the surveys in the QS rankings and prepare accordingly. For example, less than half of the ranked universities submit lists of names to QS. Others submit the names of outstanding researchers even though these will almost certainly have been submitted by someone else or submit names of their own researchers, who cannot nominate you in the survey.

The academic survey is less prominent in the THE rankings but still accounts for a substantial weighting in every subject group.

In addition, the submission of data for every indicator should be carefully reviewed and the small print in the submission instructions followed carefully. An example of the importance of this is an incident in 2009 when QS adjusted the results for Lomonosov Moscow State Publication after official publication, raising its position from 155th to 101st, after data were submitted that included research-only staff in the faculty totals.

“Note: On 7th September 2009 QS received complete and authorised student and faculty data from Lomonosov Moscow State University (MSU). The data was received too late to be included in the analysis for the 2009 results. The revised Full Time Equivalent (FTE) student number is 33,668 and the revised Faculty number is 10,418 FTE. With this new data, MSU has reached an overall ranking of 101st as published in the Top Universities Guide.”

For the THE rankings, the citations indicator can be of crucial importance. An example of this is the National Research Nuclear University MEPhI which is ranked 95th in the Physical Sciences despite having low scores for teaching and research. It has a perfect score of 100 for citations. This is because it has one contributor to the biennial Review of Particle Physics, which has thousands of authors and citations combined with a relatively small number of publications.

Reference

Claassen, C. (2015). Measuring university quality. Forthcoming: Scientometrics.

http://page2rss.com/1d6df54d95fa646384105f38c3c90bb5

http://page2rss.com/1d6df54d95fa646384105f38c3c90bb5

Appendix: Times Higher Education: Subject Group Methodology


Subject tables

The subject tables employ the same range of 13 performance indicators used in the overall World University Rankings, brought together with scores provided under five categories:

Teaching: the learning environment

Research: volume, income and reputation

Citations: research influence

International outlook: staff, students and research

Industry income: innovation.

Here, the overall methodology is carefully recalibrated for each subject, with the weightings changed to best suit the individual fields. In particular, those given to the research indicators have been altered to fit more closely the research culture in each subject, reflecting different publication habits: in the arts and humanities, for instance, where the range of outputs extends well beyond peer-reviewed journals, we give less weight to paper citations.

Accordingly, the weight given to “citations: research influence” is halved from 30 per cent in the overall rankings to just 15 per cent for the arts and humanities.

More weight is given to other research indicators, including the academic reputation survey.

For social sciences, where there is also less faith in the strength of citations alone as an indicator of research excellence, the measure’s weighting is reduced to 25 per cent.

By the same token, in those subjects where the vast majority of research outputs come through journal articles and where there are high levels of confidence in the strength of citations data, we have increased the weighting given to the research influence (up to 35 per cent for the physical and life sciences and for the clinical, pre-clinical and health tables).

A breakdown of the methodology for each subject is provided at the foot of the tables.

Criteria

No institution can be included in the overall World University Rankings unless it has published a minimum of 200 research papers a year over the five years we examine.

But for the six subject tables, the threshold drops to 100 papers a year for subjects that generate a high volume of publications and 50 a year in subjects such as social sciences where the volume tends to be lower.

Although we apply some editorial discretion, we generally expect an institution to have at least 10 per cent of its staff working in the relevant discipline in order to include it in the subject table.

The majority of institutions in Thomson Reuters’ Global Institutional Profiles database, which fuels the rankings, provide detailed subject-level information. In rare cases where such data are not supplied, institutions are either excluded or public sources are used to inform estimates.

Documents

Richard Holmes€¦ · Web viewRichard Holmes. The QS World University Rankings and THE World University Rankings by Subject Group and Subject. The QS World University Rankings