Tracking the external impacts of academic research
in long-term ways and for HEFCE purposes
Patrick Dunleavy
London School of Economics and Political Science
See our ‘Impact of Social Sciences’ blog at:
http://blogs.lse.ac.uk/impactofsocialsciences/
Structure of this talk
1. Defining external research impacts and exploring how they operate
2. Capturing useful evidence of external impacts (starting now for the longer term)
3. HEFCE’s maximalist view of external “impacts”, and its case study method
Defining research impacts in an evidence-based way
A research impact is: “a recorded or otherwise auditable occasion of influence from university research upon another actor or organization”.a. Academic impacts from research are influences upon
other actors in academia or universities, e.g. as measured by citations.
b. External impacts are influences on actors outside higher education, that is, in business, government or civil society, e.g. as measured by references in the trade press or in government documents, or by coverage in mass media.
Primary impacts – observable
occasions of influence
Academic
work
Academic impacts
e.g. citations
External impacts
e.g. influencecounts
Academic/universityoutputs
Economic/public policy/civil society
outputs
Academic work and external impacts: a simple view
Primary impacts – observable
occasions of influence
Academic
work
Academic impacts
e.g. citations
External impacts
e.g. influencecounts
Dynamicknowledgeinventory
Academic/universityoutputs
Economic/public policy/civil society
outputs
Academic work and external impacts: allowing for time lags and cumulation/ delay effects
Media, cultural and civil society systems
Economicand
businesssystems
Publicpolicy
systems
Discovery
Application
Single discipline processesWidersociety
A naïve (science-based) view of how an academic discipline achieves external impacts
Media, cultural and civil society systems
Economicand
businesssystems
Publicpolicy
systems
Discovery
Integration
Application
Renewal
Single discipline processesImpactsinterface
Widersociety
How key forms of scholarship within each academic discipline begin to achieve external
impacts
Media, cultural and civil society systems
Economicand
businesssystems
Publicpolicy
systems
Discovery
Integration
Application
Renewal
Single discipline processesJoined-up
scholarshipImpactsinterface
Widersociety
AcademicService
Bridging
University- local
integration
How cross-disciplinary influences add to and mediate external impacts
Media, cultural and civil society systems
Economicand
businesssystems
Publicpolicy
systems
Discovery
Integration
Application
Renewal
Think tanks
SpecialistMedia
Professions
Entrepreneurs
Consultants
Policycommunities
NGOs
Single discipline processesJoined-up
scholarshipImpactsinterface
Widersociety
AcademicService
Bridging
University- local
integration
Media
Corporations
How the impacts interface shapes external impacts
2. Capturing useful evidence of external impacts (starting now for the longer term)
Six metrics of academic/university involvements with external impact
• External funding linkages, especially business-to-sciences/ technology departments
• Media and specialist media presence
• Government website presence
• Academic service in government
• Reported ‘knowledge transfer’ activities
• Academics’ perceptions of impacts
2007-08 2008-09 Change % Change
Collaborative research (£000s) 697,030 731,734 34,704 5
Contract research
Total number of contracts 27,051 28,111 1,060 4
Total value of contracts (£000s) 834,627 937,373 102,746 11
Consultancy
Total number of contracts 64,292 64,025 -267 0
Total income (£000s) 334,768 331,541 -3,227 -1
of which, number with SMEs 22,802 20,596 -2,206 -11
number with large business 10,499 10,360 -139 -1
Patents
Number of new patent applications 1,898 2,097 199 9
Number of patents granted in year 590 653 63 10
Intellectual property income
Total revenues (£000s) 66,271 124,368 58,097 47
Total costs (£000s) 21,003 27,794 6,791 24
Spin-off companies
Number created 2,223 2,289 66 3
Estimated external investment received (£000s) 89,497 154,451 64,954 42
Universities’ key interactions with business in the UK
Proportion of research found in UK national pressfrom different disciplines (2008)
0 10 20 30 40 50 60
Geography
Languages
Anthropology
Computer science
Humanities*
Arts
Education
Architecture
Communication and Media
English
Philosophy
Sociology
History
Law
Psychology
Business and Finance
Economics
Politics
Science and Technology
Health and Medicine
Number of items
Creative and arts
Science and engineering
Medical sciences
Mixed
HSS
0 5 10 15 20 25
Archeology
Design
Geography
Economics
Technology
Management and media studies
Politics and democracy
Transport policy
Public policy and policy studies
Education
Development
Psychology and psychiatry
Employment and work
Criminology and criminal policy
Health policy and public health
Law and legal
Medicine
Social policy and anthropology
Percentage of all research projects found
Proportion of research found on UK government websites from different disciplines
University funding bodies
Wider government
bodies
All bodies
Top academics (managing universities) 28 21 24 Heads of department 15 12 13 Professors 58 57 57 Lecturers 0 11 7 Total 101% 100% 100% Number of academics 80 131 232
Academic Service: University staff involved in UK central government quasi-government agencies
Source: Griffiths, 2010
Knowledge transfer practice % academics involved 2009
% academics involved 2008
type of academic
activity
Attending conferences 87 56 general
Informal advice to business
57 35 application
External lectures 65 34 application
Networks 67 32 integration
Joint publication 46 26 application
Advisory boards 38 22 service
Student projects/placements
33 20 renewal
External visits 19 application
Formed/run consultancy 14 18 application
Contract research 37 18 application
Undertaken consultancy 43 17 application
Academics’ reporting of ‘knowledgetransfer’ activities - top 11 activities
Knowledge transfer practice
% academics involved 2009
% academics involved 2008
type of academic activity
Been involved in consortia
35 17 application
Joint research 49 17 application
Post-course placements na 14 renewal
Prototyping and testing 10 na application
Patenting 7 12 application
Licensed research 5 10 application
Standards forum 31 10 application
Spin out companies 4 7 discoveryapplication
Enterprise education 6 4 renewal/ application
External secondment 10 3 application
Academics’ reporting of ‘knowledgetransfer’ activities – next 10 activities
Area of external impacts Level of impact
Social scientists only
Academics spanning social sciences and humanities
All respondents (including academics across all social science and humanities)
Public policy Actual 4.6 3.6 3.4Potential 6.0 5.5 5.1
Civil society Actual 4.5 4.1 4.1Potential 5.6 5.3 5.1
Public debates and culture
Actual 3.9 5.0 4.6Potential 5.4 5.9 5.6Actual 3.6 3.1 3.0
Potential 4.7 4.0 3.8Science and technology
Actual 3.1 3.4 2.9Potential 4.5 4.1 3.9
Social science and humanities academics’ perceptions of external impacts (2008)
Eight longer-term steps towards universities boosting their external impacts
• Clean up ‘multiple branding’ problems• Reappraise events programmes• Develop an impacts file for all individual academics,
and then for departments• Pull together databases and services to give improved
‘customer relationship management’ of research clients – e.g. bring in CRM systems
• Start (and fill up) an online depository• For all research on the closed-web, publish a useful
open-web version (see session D)• Improve professional communication – starting with
multi-author blogs (see session D)• Work better in networks (see Panel 2)
3. HEFCE’s expansive view of external impacts, and case study method
Primary impacts – observable
occasions of influence
Academic
work
Academic impacts
e.g. citations
External impacts
e.g. influencecounts
Dynamicknowledgeinventory
Academic/universityoutputs
Economic/public policy/civil society
outputs
PRIMARY EXTERNAL IMPACTS OF RESEARCH
Primary impacts – observable
occasions of influence
Changes in organizational
activities or outputs
Academic
work
Academic impacts
e.g. citations
External impacts
e.g. influencecounts
Dynamicknowledgeinventory
Economic/public policy/civil society
outputs
Multiple otherfactors
WHAT MORE DOES HEFCE NEED? – 1. EVIDENCE OF CHANGED OUTPUTS
DEMONSTRATED CETERIS PARIBUS,
CONTROLLING FOR ALL OTHER
INFLUENCES
Primary impacts – observable
occasions of influence
Changes in organizational
activities or outputs
Changes in societal
outcomes
Academic
work
Academic impacts
e.g. citations
External impacts
e.g. influencecounts
Dynamicknowledgeinventory
Economic/public policy/civil society
outputs
Economic/public policy/
civil societyoutputs
Multiple otherfactors
WHAT MORE DOES HEFCE NEED? – 2. EVIDENCE OF CHANGED OUTCOMES
AGAIN CETERIS
PARIBUS, CONTROLLING
FOR ALL OTHER
INFLUENCES
Primary impacts – observable
occasions of influence
Changes in organizational
activities or outputs
Evaluations of outcome changes
– against societal values
Changes in societal
outcomes
Academic
work
Academic impacts
e.g. citations
External impacts
e.g. influencecounts
Dynamicknowledgeinventory
Economic/public policy/civil society
outputs
Economic/public policy/civil
societyoutputs
Multiple otherfactors
+ ? -
WHAT MORE DOES HEFCE NEED? – 3. PROOF OF POSITIVE SOCIAL BENEFITS
Four star Exceptional: Ground-breaking or transformative impacts of major value or significance with wide-ranging relevance have been demonstrated
Three star Excellent: Highly significant or innovative (but not quite ground-breaking) impacts relevant to several situations have been demonstrated
Two star Very good: Substantial impacts of more than incremental significance or incremental improvements that are wide-ranging have been demonstrated
One star Good: Impacts in the form of incremental improvements or process innovation of modest range have been demonstrated
Unclassified The impacts are of little or no significance or reach; or the underpinning research was not of high quality; or research-based activity within the submitted unit did not make a significant contribution to the impact.
HEFCE’S SCORING BANDS FOR IMPACT CASE STUDIES’ ‘SIGNIFICANCE’/ VALUE
AND ‘REACH’/RELEVANCE
low high medium
low
m
ediu
mhi
gh
Reach (or relevance)
Sig
nif
ica
nc
e (o
r v
alu
e)
CHARTING ALL HEFCE’S CRITERIA
low high medium
low
m
ediu
mhi
gh 4*
exceptional
3*excellent
Reach (or relevance)
Sig
nif
ica
nc
e (o
r v
alu
e)
CHARTING ALL HEFCE’S CRITERIA
low high medium
low
m
ediu
mhi
gh 4*
exceptional
3*excellent
2*very good
good1*
Reach (or relevance)
Sig
nif
ica
nc
e (o
r v
alu
e)
CHARTING ALL HEFCE’S CRITERIA
low high medium
low
m
ediu
mhi
gh
Uunclassed
4*exceptional
3*excellent
2*very good
good1*
Reach (or relevance)
Sig
nif
ica
nc
e (o
r v
alu
e)
CHARTING ALL HEFCE’S CRITERIA
Additional criteria:
▪ Impact occurring in the
last 15 years
▪ Quality of Underlying
Research
▪ Distinctiveness of Research
Short-run steps to prepare for HEFCE case studies
• Develop a large long list of possible case studies• Collate grants, publications, contacts and
testimonials evidence from (senior) academics on all potential cases, especially for 2005-11
• For ‘probable’ cases - firm up outside evidence of outputs, activity
and outcome changes; - push back pre-history to 1997 if needed• Enlarge single-person cases wherever feasible• Strengthen ‘sub-group’ case studies, by adding
members (if feasible, without diluting ‘quality’), and re-branding for greater coherence by 2013
• Monitor potential cases closely from now to 2013, and discuss with uninvolved senior case-writers
Thank you for listening
See our ‘Impact of Social Sciences’ blog at: http://blogs.lse.ac.uk/impactofsocialsciences/
From there, download the free Handbook: Maximizing the Impacts of Social Science
Research