Beyond the RAE: New methods to assess research quality

July 2008

Purpose of presentation

• Outline the new approach to assess research quality – known as the Research Excellence Framework (the REF)

• Consider and discuss any outstanding questions

Context

Government funds research through the ‘dual support’ system» HEFCE funds research infrastructure, salaries, blue skies

research (known as ‘QR’)» Research Councils fund responsive mode projects

HEFCE policy objective for research funding is selectivity» Highest quality research receives the most funding

Method to identify the highest quality research has, until now, been the RAE

» Most recent of which = RAE 2008» Results of RAE2008 will be announced on 18 December 2008 » Will inform HEFCE research funding from 2009-10

Background to the Research Excellence Framework

Government has stated that RAE 2008 will be the last such exercise

• Perception that the RAE was a burden

New framework for research assessment and funding will make greater use of quantitative indicators - ‘metrics’

HEFCE consulted in November 2007 on the proposed Research Excellence Framework

• Consultation was about the high-level process only • HEFCE have announced two main changes as a result of the

consultation responses:» Will take place in 2010 (not 2009 as proposed) to inform HEFCE

research funding from 2011-12 (not 2010-11 as proposed)» Processes for STEM and non-STEM subjects will not be so distinct

New approach to assess research quality

For all subjects, the new approach will include some combination of metrics-based indicators, including bibliometrics where appropriate, as well as input from expert panels

• Exact details and relative weightings to be determined – informed by results of pilot exercise

Pilot exercise: commenced July 2008• College is one of 22 institutions selected for the pilot• HEFCE contracted Evidence to collect and analyse data• Initial databases of staff and research outputs submitted to HEFCE 8

August 2008• Next steps:

» validation of data by Evidence & Pilot HEIs (August 08)» Additional records located, added to database (Aug – Sep 08)» Final database produced (31 October 08)» Citation counts established & normalised (31 December 08)» Results of pilot analysed and disseminated (31 March 09)

Bibliometric Indicator

Bibliometrics will be the main quality measure for Science• Supported by e.g. research income, PhD student numbers

Citations will be the bibliometric indicator used• Not journal impact factors or e.g. (H)irsch index

Source of citation information is to be determined• Thomson ISI and SCOPUS will be investigated during the pilot

Outputs and citations will be fairly historical • 5 – 10 year period up to c.2006• Pilot will use RAE period 01 January 01 to 31 December 2007

Self citations will be removed from citation counts• Pilot will model inclusion and exclusion of self citations

Citation rates will be normalised by the field average for that type and year of output

Subject Groupings

Initial consultation had proposed 6 subject groupings (broadly as per RAE 2008 main panel groups)

» Clinical Medicine (RAE 2008 UoAs 1-5)

» Health Sciences (RAE 2008 UoAs 6-9)

» Subjects Allied to Health (RAE 2008 UoAs 10-13)

» Biological Sciences (RAE 2008 UoAs 14-16)

» Physical Sciences (RAE 2008 UoAs 17-19)

» Engineering and Computer Science (RAE 2008 UoAs 23-29)

Will now need to be extended to include ‘non-science’ subjects

Allocation of outputs to subject groups

Clinical Medicine

Health Sciences

Subjects Alliedto Health

Physical Sciences

Engineering andComputer Science

Behavioural SciencesBiologyEntomologyEvolutionary BiologyFisheriesHorticulture

Mathematical & Computational BiologyMycologyOrnithologyPlant SciencesZoology

Biological Sciences

Oceanogr. & Marine Biol.Adv. Marine BiologyEuropean J. PhycologyMicrobial EcologyJournal of PhycologyAquatic ToxicologyAquatic Microbial Ecol.Freshwater BiologyFish/Shellfish Immunol.Marine Environ. Res.Canadi. J. Fish. Aq. SciMarine Ecol.-Prog. Ser.J. N. Amer. Benth. Soc.Rev. Fish. Biol./FisheriesJ. Exp. Mar. Biol./Ecol.NZ J. Marine & FW. Res.J. Marine Biol Ass. UKDiatom Res.etc

Marine & Freshwater Biology

Subject Groups Fields Journals

• Further work needed where journals fall into more than one field

• Separate process for specific journals which cover multiple subjects (e.g. Nature)

Issues with using citations

• Inadequate coverage in publication databasesE.g. Thomson ISI

(Leiden University, Nov 2007)

• High citation may not indicate good quality• Need to ensure publication records and addresses are accurate

and current

Internal coverage percentage

80-100% 60-80% 40-60% <40%

Mol biol & biochem Appl phys & chem Mathematics

Other soc sci

Biol sci: humans Biol sci: anim & plants Economics Humanities & arts

Chemistry Psychol & psychiat Engineering

Clinical medicine Geosciences

Phys & astron Soc sci in medicine

Allocation of outputs to institutions (HEIs)

Many unanswered questions:

How to allocate outputs to HEIs?• Assign where author employed at time of publication? • OR where author employed on a given (census) date?

How to treat co-authors?• Each output count once OR multiple times?• How to handle outputs published by staff at different HEIs?• Should author contribution be measured? – How?

Should selection form part of the process?• Select staff OR submit all?• Select outputs OR include all (for selected staff?)

Possible behavioural implications

New process could lead to:

• Publication behaviours:» Preference to publish in particular journals» Impact on other publication types (conferences, internet) or

research which produces outcomes which aren’t able to be assessed under the new system

» Difficulties establishing new and emerging fields» Reduction in publication volume – desire to publish a few highly

cited articles » Increase in single-authored/reduction in co-authored articles

(due to exclusion of self-citations)» Formation of citation clubs

• Staffing behaviours:» Impact on staff who are establishing their research record