Real world approaches to assessing the impact of research on policy

Annette Boaz

King’s College London

23rd September 2010

2

Outline

• Models of the relationship between research

policy and practice

• Rationales for assessing impact

• What can we learn from the literature?

• 10 research impact assessment case

studies

• Consider the implications for SSH research

3

Models

Linear – relay race

Relationship – two communities

Systems – co-production

Diffusion

Engineering

4

Activities designed to promote research

impactKnowledge

brokers

Research briefings

Participation in Networks

Co-locating researchers and

policy makersJoint

research projects

Advisory groups

Seminars/ presentations

5

Models

Linear – relay race

Relationship – two communities

Systems – co-production

Research briefings

Joint research projects

Advisory groups

Effective knowledge transferdepends on

• the nature of the research and its utility

• the personal characteristics of both researchers and users

• the links between researchers and users

• the context for the application of the research

See Nutley, Walter and Davies, 2007, pp 81-83,

See also Overseas Development Institute

6

7

Why assess the impact of our research?

Learning

Performance

In its quest for excellence, the European Commission must and will encourage, promoteand support every effort to understand and monitor the quality of research at universities. 1

Accountability

Promoting organizational achievements

Moderating between stakeholders

1Commissioner Janez Potočnik, Assessing Europe’s University-Based Research, 2010

8

Review question: How do you (best) evaluate the impact of research on

policy?• Explored the websites of 30 organizations

• Hand search of selected journals and reference lists

• Search of SPRU and IDS library catalogues

• Search of 10 research databases

• 1987-2007

• UK and international sources

Research programmes

Research

Initiatives to increase research use

Research Users

Systematic reviews

Focusing the review on policy use

Single study

Policy makers

Country of OriginAustralia, 8

Canada , 20

Finland, 4

Switzerland, 3

The Netherlands, 9

UK, 61

USA, 38

Other,13

Other includes: Brazil, Denmark, Spain and Norway

11

Methods for measuring research impact

Figure 5: Methodologies

0

10

20

30

40

50

60

To

tal n

um

ber

of

pap

ers

Discussed

Used

How do people evaluate the impact of research

programmes? Wide range of methods and approaches (often in combination):

· Qualitative methods: semi-structured interviews, documentary analysis, field visits and observations

· Quantitative methods: surveys, bibliometrics and patent/new technology tracking

· Panels and peer review

· Workshops and focus groups

· Process tracking

Figure 6: Frameworks for structuring and interpreting data

02468

101214161820

Tota

l nu

mb

er

of

pa

pe

rs

Discussed

Used

Ten impact assessment case studies

Stockholm Environment Institute

14

The Foundation for Research, Science and Technology (New Zealand)

Finnish Environment Institute

The Irish Environmental Protection Agency

CGIAR

Environment Agency (England and Wales)Defra

Land and Water Australia

French Ministry of Ecology, Energy, Sustainable Development and the Sea

Swedish Environmental Protection Agency

15

The Irish Environmental Protection Agency

Finnish Environmental Institute

Embedded approach: Ten year independent evaluation, four year strategic evaluations, internal annual evaluations, annual researcher self evaluation

Goal: how to increase impact

Methods: multi-method16

Key themes:

• Diverse of objectives, scope and scale

• Types of approach

• Generic challenges

• Challenges specific to environmental research

17

What are the implications for SSH

research in Latin America and Europe?

18

8 key questions (when designing an impact

evaluation)1. What is your conceptual framework?2. What are the outcomes of interest?3. What methods will best explore the outcomes of interest?4. How do you address attribution?5. What is the direction of travel for the evaluation?6. Is this a mixed method approach, providing scope for triangulation?7. Will the methods selected capture context and the complexity?8. When might be the best time to conduct the evaluation?

These need to be considered in the light of the time, skills and resources available for the evaluation.See Boaz et al 2009

My contact details: Annette.boaz@kcl.ac.uk

Some useful references and websites:

-Boaz, A, Fitzpatrick, S and Shaw, B (2009) Assessing the impact of research on policy: a literature review. Science and Public Policy, 36(4): 255-270

-Nutley S, Walter I, and Davies H (2007) Using evidence: How research can inform public services. Bristol: The Policy Press.

-O’Leary, G, Boyle, R, Donlon, B and Sheils, L (2008) Providing a framework for accountability and learning in environmental research. Administration. 5 (4): 159-170

-The Overseas Development Institute RAPID programme: http://www.odi.org.uk/programmes/rapid/

- Wooding S, Nason E, Klautzer L, Rubin J, Hanney S and Grant J (2007) Policy and practice impacts of research funded by the ESRC. A case study of the Future of Work programme, approach and analysis. Cambridge: RAND Europe. http://www.rand.org/pubs/technical_reports/2007/RAND_TR435.pdf

20

21

Managing Editors: Annette Boaz, Division of Health and Social Care Research, King's College London, UK and David Gough, Institute of Education, University of London, UK

Volume 6, 2010, 4 issues: January, May, August and November http://www.policypress.co.uk/journals_eap.asp