Demonstrating research impact in the REF

Graeme Rosenberg

REF Manager

• What is the REF?

• What is impact?

• How should it be demonstrated?

• How will it be assessed?

Outline:

What is the REF?

• The REF is the new framework for assessing research in all disciplines in all UK higher education institutions

• It replaces the RAE

• Its purpose is:

- To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year)

- Provide accountability for public funding of research and demonstrate its benefits

- To provide benchmarks and reputational yardsticks

The assessment framework

Overall quality

Outputs

Maximum of 4 outputs per researcher

Impact

Impact template and case studies

Environment

Environment data and template

65% 20% 15%

Submissions • There are 36 discipline-based units of assessment

(UOAs)

• Each institution decides which UOAs to submit in

• Each submission in a UOA must contain a standard set of information:

- Staff details (REF1a/b/c)

- Research outputs (REF2)

- Impact template and case studies (REF3a/b)

- Environment data (REF4a/b/c)

- Environment template (REF5)

Assessment

Sub-panel responsibilities

• Contributing to the panel criteria and working methods

• Assessing submissions and recommending the outcomes

Main panel responsibilities

• Developing the panel criteria and working methods

• Ensuring adherence to the criteria/procedures and consistent application of the overall assessment standards

• Signing off the outcomes

• Submissions will be assessed by 36 expert sub-panels working under the guidance of 4 main panels

• Panels will carry out the assessment according to the published criteria and working methods

Outputs Impact Environment

4* 3* 2* 1* U

20 45 35 0 0

4* 3* 2* 1* U

0 40 40 20 0

65%

Overall Quality Profile

12

4*

0104137

U1*2*3*

4* 3* 2* 1* U

12.8 32.8 43 11.4 0

20% 15%

The overall quality profile is comprised of the aggregate of the weighted sub-profiles produced for outputs, impact and environment.

Quality Level

% of Research Activity

Example of a quality profile

Timetable

Criteria phase

2011

• Develop and publish Guidance on submissions (Jul)

• Develop, consult on and publish Panel criteria (Jan 2012)

Submissions phase

2012-13•HEIs submit Codes of practice

•Launch the REF submissions system

•Submission deadline 29 Nov 2013

Assessment phase

2014•Panels assess submissions

•Publish outcomes Dec 2014

What is impact?

• Impact is defined broadly for the REF:

an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia

• The REF definition of impact is retrospective – benefits or changes that have already occurred, not future potential

• Impacts can be manifest in a wide variety of ways, may take many forms and occur in a wide range of spheres, in any geographic location

Examples of types of impact

Public debate has been shaped or informed by research

A social enterprise initiative has been created

Policy debate or decisions have been influenced or shaped by research

A new product has been commercialised

Enhanced professional standards, ethics, guidelines or training

Jobs have been created or protected

Improved business performance

Changes to the design or delivery of the school curriculum

The policies or activities of NGOs or charities have been informed by research

Improved management or conservation of natural resources

Improved forensic methods or expert systems

Production costs have reduced

Levels of waste have reduced

Improved quality, accessibility or efficiency of a public service

Enhanced preservation, conservation or presentation of cultural heritage

Organisations have adapted to changing cultural values

New forms of artistic expression or changes to creative practice

More effective management or workplace practices

Changes to legislation or regulations

Enhanced corporate social responsibility policies

Research has informed public understanding, values, attitudes or behaviours

Improved access to justice, employment or education

Enhanced technical standards or protocols

Improved risk management

Improved health or welfare outcomes

Research has enabled stakeholders to challenge conventional wisdom

Changes in professional practice

Examples from the pilot exercise

Improved treatment and quality of life for patients with type 1 diabetes

Detector technology used to screen liquids at airports

Conservation of bumblebees

Influencing policy on child maintenance and better service for separated parents

Enriched public appreciation of heritage at Hampton Court Palace

Pilot examples are published at: www.ref.ac.uk/background/pilot

an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia

• Impacts can be manifest in a wide variety of ways, may take many forms and occur in a wide range of spheres, in any geographic location

• Panels provide examples of impact relevant to their disciplines, intended to be helpful - not as exhaustive or prescriptive lists

Demonstrating impact

•Sets out the submitted unit’s approach to impact

Impact template

•Specific examples of impacts already achieved

Case studies

20% of the impact sub-

profile

80% of the impact sub-

profile

Case studies • One case study must be submitted per 10 FTE staff

(with a minimum of two cases in a submission)

• Impacts that took place during 2008 to 2013; underpinned by research since 1993

• Submitted case studies need not be representative of activity across the unit: pick the strongest examples

• Case studies must be completed on a template - maximum 4 pages

The case study template 1. Summary of the impact

2. Underpinning research:

- What was it? Who undertook it and when?

- What were the key insights or findings?

3. References to the research

4. Details of the impact:

- How did the research lead/contribute to the impact?

- Who were the beneficiaries?

- How did they benefit?

5. Sources of corroboration

Lessons from the pilot exercise • All the material required to make a judgement should be

included

• The narrative should be coherent and clearly explain the ‘links in the chain’

• Clear definition of who benefitted, and what had changed

• Indicators should be meaningful and contextualised

• Key claims should be capable of verification

• Where the impact arises from public engagement:

- How was the engagement activity based on the research?

- Dissemination in itself is not impact – what was the benefit?

Preparing impact submissions • HEIs seem to be at varying stages of:

- Briefing staff

- Identifying potential cases

- Drafting case studies

• Some issues seem to be:

- How to write compelling narrative, and who should be involved

- Evidence gathering – retrospective and from many sources

- Institutional memory – what if the staff have left?

- Second guessing the panels to decide on the strongest cases

How will impacts be assessed?• Sub-panels will assess impacts against the criteria of

reach and significance:

- A holistic judgement about each case study

- ‘Reach’ is not a geographic scale.

• Each case study will be examined by at least one academic and one ‘user’ on the panel

• Panels will make judgements on the merits of the evidence provided – not further investigations

• A sample of case studies will be audited to corroborate or verify information

Underpinning research

• Panels will assess the impact, not the underpinning research

• But each case study must be underpinned by research that satisfies the following criteria:

- produced by staff while working in the submitting HEI

- generated outputs published between 1 Jan 1993 and 31 Dec 2013

- meets the quality threshold of at least equivalent to 2*

- made a material and distinct contribution to the impact

Assessment criteria

The criteria for assessing impacts are reach and significance*

Four star Outstanding impacts in terms of their reach and significance

Three star Very considerable impacts in terms of their reach and significance

Two star Considerable impacts in terms of their reach and significance

One star Recognised but modest impacts in terms of their reach and significance

UnclassifiedThe impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit

* Each main panel provides further description of the criteria

Further information

www.ref.ac.uk (includes all relevant documents)

Enquiries from staff at HEIs should be directed to their nominated institutional contact

(see www.ref.ac.uk for a list)

Other enquiries to info@ref.ac.uk