• These slides have been produced by the REF team, and were last updated on 30 January 2012

• They provide a summary of the assessment framework and guidance on submissions and the REF panel criteria

• They may be used freely for the purposes of briefing any interested parties on the REF

• Should anyone other than the REF team edit these slides, then any slides that you add or amend with your own interpretation of the REF guidelines, should not be presented with the REF logo and design. It should be clear to the audience that such slides have not been produced by the REF team.

The Research Excellence Framework

Assessment framework,

guidance on submissions

and panel criteria

Presentation outline

• Overview

• REF panels

• Staff

• Outputs

• Impact

• Environment

Overview

Purpose of the REF

The REF replaces the RAE as the UK-wide framework for assessing research in all disciplines. Its purpose is:

• To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year)

• Provide accountability for public funding of research and demonstrate its benefits

• To provide benchmarks and reputational yardsticks

Overview:

Principles of the REF

The REF is a process of expert review. The assessment is founded on the professional judgement of discipline-based expert panels.

The conduct of the REF is guided by the principle of:

• Equity: All types of research and forms of output in all disciplines shall be assessed on an equal basis

• Equality: HEIs are encouraged to submit the work of all their excellent researchers

• Transparency: The assessment criteria, procedures and outcomes to be published in full

Overview:

Key changes since 2008 RAE • Inclusion of assessment of impact

• Fewer UOAs/panels, operating more consistently

• Strengthened equality and diversity measures

• Revised eligibility criteria for staff

• Addition of (limited) use of citation data in some UOAs

• Removal of ‘esteem’ as a distinct element

• Revised approach to ‘environment’ and data collection

• Increased ‘user’ input on panels; and an integrated role for additional assessors

• Publication of overall quality profiles in 1% steps

Overview:

Guidance and criteria

Comprehensive information and guidance is set out in:

• Assessment framework and guidance on submissions (July 2011):

- Sets out the information required in submissions and the definitions used

• Panel criteria and working methods (Jan 2012):

- Sets out how panels will assess submissions

• Further supplementary guidance will be published during 2011-12 on technical and procedural matters

Overview:

The assessment framework

65% 20% 15%

Overview:

Submissions • Each submission in a UOA provides evidence about the

activity and achievements of a ‘submitted unit’ including:

- Staff details (REF1a/b/c)

- Research outputs (REF2)

- Impact template and case studies (REF3a/b)

- Environment data (REF4a/b/c)

- Environment template (REF5)

• A submitted unit may, but need not, comprise staff who work within a single ‘department’ or organisational unit

Overview:

Multiple and joint submissions • Institutions will normally make one submission in each

UOA they elect to submit in

• Joint submissions are encouraged where this is an appropriate way of describing collaborative research

• Multiple submissions only by exception and with permission from the REF manager:

- Where an HEI also makes a joint submission in that UOA

- Where HEIs have merged

- In Sub-panel 28 where one submission is in Celtic studies

- Where a sub-panel considers there is a case given the nature of the disciplines covered. These sub-panels are listed in the panel criteria statements.

Overview:

Timetable Overview:

Publication of results• The primary outcome of the REF is an ‘overall quality

profile’ to be awarded to each submission:

- Using the same scale as RAE2008, but in steps of 1%

- To be published in Dec 2014

• Further reports and feedback during 2015:

- An overview report by each main panel, including observations by each of their sub-panels

- Concise feedback on submissions, to the heads of HEIs

- The output, impact and environment sub-profile for each submission

- A report by the Equality and Diversity Advisory Panel

- Submissions will be published (except for confidential or sensitive information)

Overview:

Outputs Impact Environment

4* 3* 2* 1* U

20 45 35 0 0

4* 3* 2* 1* U

0 40 40 20 0

65%

Overall Quality Profile

12

4*

0104137

U1*2*3*

4* 3* 2* 1* U

12.8 32.8 43 11.4 0

20% 15%

The overall quality profile is comprised of the aggregate of the weighted sub-profiles produced for outputs, impact and environment.

Quality Level

% of Research Activity

Example of a quality profileOverview:

REF panels

Main and sub-panel roles REF panels:

There are 36 sub-panels working under the guidance of 4 main panels. Membership is published at www.ref.ac.uk

Main Panel A 1 Clinical Medicine

2 Public Health, Health Services and Primary Care

3Allied Health Professions, Dentistry, Nursing and Pharmacy

4 Psychology, Psychiatry and Neuroscience

5 Biological Sciences

6 Agriculture, Veterinary and Food Science

REF panels:

Main Panel B 7 Earth Systems and Environmental Sciences

8 Chemistry

9 Physics

10 Mathematical Sciences

11 Computer Sciences and Informatics

12Aeronautical, Mechanical, Chemical and Manufacturing Engineering

13Electrical and Electronic Engineering, Metallurgy and Materials

14 Civil and Construction Engineering

15 General Engineering

REF panels:

Main Panel C 16 Architecture, Built Environment and Planning

17 Geography, Environment Studies and Archaeology

18 Economics and Econometrics

19 Business and Management Studies

20 Law

21 Politics and International Studies

22 Social Work and Social Policy

23 Sociology

24 Anthropology and Development Studies

25 Education

26 Sports-related Studies

REF panels:

Main Panel D 27 Area Studies

28 Modern Languages

29 English Literature and Language

30 History

31 Classics

32 Philosophy

33 Theology and Religious Studies

34 Art and Design: History, Practice and Theory

35 Music, Drama, Dance and Performing Arts

36Communications, Cultural and Media Studies, Library and Information Management

REF panels:

Additional assessors

• Both ‘academic’ assessors (to assess outputs) and ‘user’ assessors (to assess impacts) will be appointed

• Assessors will play a full and equal role to panel members, in developing either the outputs or impact sub-profiles. They will be fully briefed, take part in calibration exercises and attend meetings:

- Some appointments in 2012, where the sub-panel has already identified a clear gap in expertise required to assess outputs or impact

- Further appointments in 2013, in the light of the survey of institutions’ submission intentions

REF panels:

Additional assessors will be appointed to extend the breadth and depth of panels’ expertise:

UOA descriptors and boundaries

• UOA boundaries are not rigidly defined and all panels

expect submissions to cross boundaries with other UOAs

• Sub-panel membership is broad and includes members

with interdisciplinary expertise

• Assessors will be appointed to extend their breadth and

depth of expertise. Where there is a significant overlap

between UOAs, ‘joint’ assessors may be appointed to work

with more than one sub-panel.

• The sub-panels prefer to assess submitted work within the

sub-panel but may, exceptionally, cross-refer parts of

submissions to other sub-panels for advice. The original

sub-panel remains responsible for recommending the

outcome

REF panels:

Main panel working methods

• Each main panel will work with its sub-panels to apply common assessment criteria

• Panels will undertake calibration exercises and reviews of emergent outcomes to make sure overall assessment standards are applied consistently

REF panels:

Sub-panel working methods

• The sub-panel chair will allocate work to members/assessors with appropriate expertise

• All outputs will be reviewed in sufficient detail to contribute to the outputs sub-profile

• Each case study will normally be assessed by at least one academic member and one user member/assessor

• Sub-panel members may raise audit queries with the REF team for investigation

• The panel secretariat will minute panel procedures

REF panels:

Staff

Category A and C staffHEIs select which staff to include in submissions:

• Category A staff: Academic staff with a contract of at least 0.2 FTE, on the payroll of the HEI on 31 Oct 2013, with a primary employment function of either ‘research only’ or ‘teaching and research’

• Category C staff: Staff employed by an organisation other than an HEI with a contract or job role including research, and whose research is primarily focused in the submitting unit on 31 Oct 2013

(Category C staff will contribute to the quality profile but not the volume measure for funding purposes)

• Research assistants are eligible only by exception

Staff:

Codes of practice on staff selection

• The funding bodies encourage HEIs to submit the work of all their excellent researchers, and HEIs have legal obligations affecting their staff selection procedures

• Each HEI is required to develop, document and apply a code of practice on the fair and transparent selection of staff for the REF:

- Must be signed off by the head of the institution and submitted to the REF team by 31 July 2012 at the latest

- The REF Equality and Diversity Advisory Panel will examine these for adherence to the guidance

- They will be published at the end of the assessment process

Staff:

Codes of practice on staff selection

• Codes should demonstrate fairness to staff by addressing the principles of:

- Transparency: clearly setting out the procedures for staff selection, and communicating these to all eligible staff

- Consistency: applying consistent procedures across the institution

- Accountability: clearly defining responsibilities for decisions, with appropriate training for those involved

- Inclusivity: promoting an inclusive environment, with robust procedures for staff to disclose individual circumstances

• An Equality Impact Assessment should inform the code and be kept under review at key stages of the selection process

Staff:

Individual staff circumstances

• Up to four outputs must be listed against each member of staff

• This can be reduced without penalty where circumstances have constrained an individual’s ability to work productively or produce four outputs during the assessment period:

- A wide range of circumstances will be taken into account

- With as much clarity as possible about how many outputs may be reduced without penalty

- To be treated consistently across the exercise

• The allowances for maternity, paternity and adoption leave have been revised following consultation

Staff:

Individual staff circumstancesStaff:

Clearly defined circumstances

• ‘Tariffs’ define the number of outputs that may be reduced without penalty

• These will be applied consistently by all REF sub-panels

• Circumstances may be combined up to a maximum reduction of three outputs

• Where an individual has a combination of clearly defined and complex circumstances, these should be submitted collectively as ‘complex’

Staff:

Clearly defined circumstances:Early career researchers

Date at which the individual first met the REF definition of an early career researcher:

Number of outputs may be reduced by up to:

On or before 31 Jul 2009 0

Between 1 Aug 2009 and 31 Jul 2010 inclusive

1

Between 1 Aug 2010 and 31 Jul 2011 inclusive

2

On or after 1 Aug 2011 3

Staff:

Staff eligible for selection who started their careers as independent researchers on or after 1 August 2009

Clearly defined circumstances:Part-time working, secondments and career breaks

Total months absent between 1 Jan 2008 and 31 Oct 2013:

Number of outputs may be reduced by up to:

0-11.99 0

12-27.99 1

28-45.99 2

46 or more 3

For part-time working, the equivalent ‘months absent’ should be calculated

Staff:

Clearly defined circumstances:Maternity, paternity and adoption leave

• Changes were made to these arrangements, following consultation

• Individuals may reduce the number of outputs by one, for each discrete period of:

- Statutory maternity leave or statutory adoption leave taken substantially during the period 1 January 2008 to 31 October 2013, regardless of the length of the leave. 

- Additional paternity or adoption leave lasting for four months or more, taken substantially during the period 1 January 2008 to 31 October 2013.

• Constraints relating to pregnancy, maternity, paternity, adoption or childcare that fall outside of – or justify the reduction of further outputs in addition to – these allowances may be submitted using the arrangements for complex circumstances.

Staff:

Clearly defined circumstances:Other circumstances in UOAs 01-06

• Outputs may be reduced by up to two for the following:

- Category A staff who are junior clinical academics (and have not gained a CCT or equivalent prior to 31 October 2013.

- Category C staff who are employed primarily as clinical, health or veterinary professionals

Staff:

Complex circumstances

• EDAP will consider all cases of complex circumstances

• EDAP will make recommendations to the Main Panel Chairs, who will decide

• Sub-panels will be informed of the decisions and will assess the remaining outputs without penalty

• Information will be kept confidential to EDAP, Main Panel Chairs and the REF team

• ECU will publish a range of worked examples – including EDAP’s rationale

Staff:

Outputs

Eligibility

• Outputs must be:

- a product of research (as defined for the REF)

- first brought into the public domain between 1 January 2008 and 31 December 2013

- Authored/co-authored by the member of staff against whom it is listed (regardless of where they were employed prior to the census date)

Outputs:

Outputs ‘pre-published’ before 2008

• A change in the guidance on pre-publication of outputs has been made, following consultation:

An output first published in its final form during the REF publication period, but ‘pre-published’ in 2007 – is eligible for submission to the REF, provided that the ‘pre-published’ output was not submitted to the 2008 RAE. Examples may include:

- An online first article or preprint

- A preliminary version or working paper

Outputs:

Range of output types

• Outputs may include but are not limited to: printed or electronic publications, materials, devices, images, artefacts, products, buildings, confidential or technical reports, patents, performances, exhibits or events

• All forms of outputs shall be assessed on a fair and equal basis

Outputs:

Co-authorship

• Co-authored outputs can be listed against more than one co-author when returned in different submissions

• An output can be listed no more than twice within the same submission and must be accompanied by a statement explaining the substantial and distinctive contribution of each of the submitting authors

Outputs:

Double-weighting

• Institutions may request ‘double-weighting’ for outputs of extended scale and scope

• Requests for double weighting should be accompanied by a supporting statement

• Sub-panels will assess the claim for double-weighting separately from assessing the quality of the output

• If a sub-panel accepts a request for double weighting, the output will count as two outputs in both a submission and in the calculation of the outputs sub-profile

Outputs:

Additional information on outputs• Panels may make use of additional information in the

form of citation data and details provided by the HEI

- Sub-panels have requested further information on the research process or content where this is not evident from the output

• Sub-panel will not make use of journal impact factors, rankings or lists, or the perceived standing of the publisher, in assessing the quality of research outputs

Outputs:

Citation data• The following sub-panels will make use of citation data:

- Main Panel A: Sub-panels 1-6

- Main Panel B: Sub-panels 7-11

- Main Panel C: Sub-panel 18

• These sub-panels will use only the citation data provided by the REF team, sourced from Scopus (except for Sub-panel 11 that will, in addition, use Google Scholar)

• All other sub-panels will neither receive nor make use of citation data

• None of the sub-panels will use journal impact factors, journal rankings or other forms of bibliometric analysis

Outputs:

Citation data• Where used, sub-panels will consider citation data as

follows:

- As one indicator of ‘academic significance’. Expert review remains the primary means of assessing outputs

- Panels will assess all outputs on an equal basis regardless of the availability of such data. They recognise the limitations of citation data – especially for recently published outputs – and will have due regard to potential equality implications

- In the relevant UOAs only, citation counts from Scopus will be provided to panels on a consistent basis. HEIs will be able to verify the outputs have been matched correctly, and view citation counts on the submission system

• The funding bodies do not sanction or recommend that HEIs rely on citation data to inform the selection of staff or outputs for their REF submissions

Outputs:

Assessment criteriaThe criteria for assessing the quality of outputs are

originality, significance and rigour*

Four star Quality that is world-leading in terms of originality, significance and rigour

Three starQuality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence

Two star Quality that is recognised internationally in terms of originality, significance and rigour

One star Quality that is recognised nationally in terms of originality, significance and rigour

UnclassifiedQuality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment

* Each main panel provides a descriptive account of the criteria

Outputs:

Impact

Definition of impact for the REF• An effect on, change or benefit to the economy,

society, culture, public policy or services, health, the environment or quality of life, beyond academia

• Impact includes an effect, change or benefit to:

- The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding

- Of an audience, beneficiary, community, constituency, organisation or individuals

- In any geographic location whether locally, regionally, nationally or internationally

• It excludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI

Impact:

Range of impacts

• Panels recognise that Impacts can manifest in a wide

variety of different ways, may take many forms and occur in

a wide range of spheres

• Examples of impact may include:

- Impacts on public policy and services,

- Impacts on society, culture and creativity,

- Impacts on practitioners and services,

- Impacts on the environment,

- Impacts on the economy

Impact:

Examples of impacts: Main Panel AImpact:

Impacts on: For example:

Health and welfare Patient outcomes have improved

Society, culture and creativity Public debate has been stimulated

The economy Costs of healthcare have changed

Commerce Business performance has improved

Public policy and services Policy decisions informed by research

Production Waste levels have reduced

Practitioners and services Professional standards influenced by research

Environment Changes to environmental risk management

International development Quality of life improvements

Range of impacts – Main Panel BImpact:

Impacts on: For example:

The economy Patient outcomes have improved

Public policy and services Public debate has been stimulated

Society, culture and creativity Public engagement in science has been stimulated

Health Business performance has improved

Practitioners and professional services

Policy decisions informed by research

Environment Waste levels have reduced

Examples of impacts: Main Panel CImpact:

Impacts on: Examples:

Creativity, culture and society

Enhancements to heritage preservation, conservation and presentationShaping or informing public attitudes and values

Economy, commerce or organisations

Improved products, processes or workplace practicesEnhanced corporate social responsibility policies

EnvironmentChanges in public awareness or behaviourBusiness operations have been changed to achieve environmental objectives

Health and welfareDevelopment of policy or practice with regard to health services or social care provision

Practitioners and professional services

Influence on professional standards, guidelines or trainingPractitioner debate has been stimulated by research findings

Public policy, law and services

Legislative change or effect on legal practiceInfluence on policy (by government, NGO or private organisation)Impact on democratic participation or access to justice

Examples of impacts: Main Panel DImpact:

Impacts on: Examples:

Civil society

Cultural life

Economic prosperity Design of new products or services

Education

Policy making

Public discourse

Public services

SubmissionsImpact:

Case studies (REF3b)

• In each case study, the impact described must:

- Meet the REF definition of impact

- Have occurred between 1 Jan 2008 and 31 July 2013 (can be at any stage of maturity)

- Be underpinned by excellent research (of at least 2* quality) produced by the submitting unit between 1 January 1993 to 31 December 2013

• Submitted case studies need not be representative of activity across the unit: pick the strongest examples

Impact:

Underpinning research• Case studies must be underpinned by research

produced by the submitted unit that has made a material and distinct contribution to the impact

• Case studies must provide evidence that the underpinning research meets the quality threshold of at least two star

• HEIs may submit case studies of a confidential nature, these may be redacted for publication or not published at all

• HEIs may request advance permission to submit case studies containing sensitive material that could only be assessed by individuals with national security clearance

Impact:

Types of evidence• Evidence of the nature and extent of the impact, in

terms of its reach and significance, should be provided and may include:

- Critical reviews in the media,

- Citation by international bodies,

- Visitor or audience numbers and feedback

- Business performance measures, sales, turnover,

- Documented evidence of change to public policy/legislation/regulations/guidelines

- Verifiable influence on particular projects or processes which bring benefits

Impact:

Case studies (REF3b)• Each case study is limited to 4 pages and must:

- Describe the underpinning research produced by the submitting unit

- Reference one or more key outputs and provide evidence of the quality of the research

- Explain how the research made a ‘material and distinct’ contribution to the impact (there are many ways in which this may have taken place)

- Explain and provide evidence of the nature and extent of the impact: Who/what was affected? How were they affected? When?

- Provide independent sources that could be used to verify claims about the impact (on a sample audit basis)

Impact:

Template (REF3a)• The unit’s approach to enabling impact from its research:

- Context for the approach

- The unit’s approach during 2008-2013

- Strategy and plans for supporting impact

- Relationship to the submitted case studies

• Provides additional information and context for the case studies, and can take account of particular circumstances that may have constrained a unit’s selection of case studies

• To be assessed in terms of the extent to which the unit’s approach is conducive to achieving impact of ‘reach and significance’

Impact:

Assessment criteria

The criteria for assessing impacts are reach and significance*

Four star Outstanding impacts in terms of their reach and significance

Three star Very considerable impacts in terms of their reach and significance

Two star Considerable impacts in terms of their reach and significance

One star Recognised but modest impacts in terms of their reach and significance

UnclassifiedThe impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit

* Each main panel provides a descriptive account of the criteria

Impact:

Environment

Data (REF4) • All submissions to include data on:

- Research doctoral degrees awarded (REF4a)

- Research income (REF4b)

- Research income in-kind (REF4c)

• Definitions are aligned with HESA returns. We will provide HESA data to institutions to assist in preparing submissions, and will use it for verification purposes

• Sub-panels 8, 9, 19, 25 and 26 request specific additional data, to be included within the environment template (REF5)

• Data will be considered by panels alongside the qualitative information provided in REF5

Environment:

Environment template (REF5)• Each submission to include a completed template (with

page limits depending on the number of staff submitted):

- Overview

- Research strategy

- People (including staffing strategy and staff development; and research students)

- Income, infrastructure, and facilities

- Collaboration and contribution to the discipline

• Panel criteria indicate appropriate forms of evidence, and the relative importance of each section

• No expectation that the environment relates to a single department or organisational unit

Environment:

Assessment criteriaThe criteria for assessing the environment are

vitality and sustainability*

Four star An environment that is conducive to producing research of world-leading quality, in terms of its vitality and sustainability

Three starAn environment that is conducive to producing research of internationally excellent quality, in terms of its vitality and sustainability

Two starAn environment that is conducive to producing research of internationally recognised quality, in terms of its vitality and sustainability

One starAn environment that is conducive to producing research of nationally recognised quality, in terms of its vitality and sustainability

Unclassified An environment that is not conducive to producing research of nationally recognised quality

* Each main panel provides a descriptive account of the criteria

Environment:

Further information

www.ref.ac.uk (includes all relevant documents)

Enquiries from staff at HEIs should be directed to their nominated institutional contact

(see www.ref.ac.uk for a list)

Other enquiries to info@ref.ac.uk