Upload
irishhumanitiesalliance
View
60
Download
1
Tags:
Embed Size (px)
Citation preview
Impact: how should we
capture it?Impact and the Humanities
June 2015
Liam Cleere, University College Dublin
2
Agenda
• Context
• Beyond Publications
• Measures of Impact for Arts & Humanities?
• Implementing Impact
3
About UCD
• Ireland’s largest university– over 1,300 faculty and 30,000 students
• €430m annual turnover
• National leader in research funding– €100m annual research income
• 139th in QS World University Ranking 2014/15
• Ireland’s Global University– home to over 6,000 international students from 124 countries and delivers degrees to over
5,000 students on overseas campuses
4
International research excellence
Top 50 in world - 2 Veterinary Science (40)Development Studies (45)
Top in Ireland - 12
Veterinary Science Development StudiesArchitecture/Built Environment Business & ManagementLaw Engineering -ChemicalEngineering – Civil & Structural Earth & Marine ScienceAgriculture & Forestry Materials ScienceStatistics & Operational Research Mathematics
Top 100 in world – 9English Language & Literature HistoryModern Languages Agriculture & ForestryVeterinary Science Geography & Area StudiesDevelopment Studies LawPolitics & International Studies
Source: QS Subject Rankings 2015
5
Irish Context
• 2006: Strategy for Science Technology and Innovation (SSTI)
– Double the number of PhDs
• 2009: Research impact through bibliometric analysis of research publications in peer reviewed journals
– Indicators of economic impact and societal contribution absent
• 2011: Research Prioritisation process: Increased focus on economic and societal impacts
– Government and funding agency led
– Focus on jobs, FDI and applied research
– SFI & Horizon 2020 require impact plans
• No National Research Excellence Framework
Beyond Publications Project
6
1. Current State Assessment 2. Future Options
3. Final Report Delivery &
Recommendations
2013 2014
Phase 1 Phase 2 Phase 3
Report:
• “Furthering the research impact of University
College Dublin”, May 2014
Steering Committee
Meeting
Steering Committee
Meeting
Steering Committee
Meeting
7
Why capture impact?
1. Higher Education Institutions overview – To enable research organisations to monitor and manage their performance and understand
the contribution that they are making to communities
2. Accountability– To demonstrate the value of research to government, stakeholders, and the wider public
3. Inform funding– To understand the socio-economic value of research and subsequently inform funding
decisions
4. Impact Journey– To understand the method and routes by which research leads to impacts, optimising the
potential of research findings and developing better ways of delivering impact
8
Point of view from ‘Furthering Impact’
• Broader role for university– Building a more just, inclusive and wiser society
• No standardised approach for addressing the
broader picture of research impact– provides an opportunity for UCD to clarify its position on research
impact, and
– implement an effective system for capturing research outputs and
communicating their value and relevance in social, cultural and
economic ways
9
CompositionDesign
PerformanceSoftware
ExhibitionArtefact
Digital or visual mediaDevice and product
*OtherScholarly edition
Patent/ published patent applicationConfidential report for external body
Website contentResearch datasets
BookWorking paper
Edited bookResearch report for external body
Book ChapterConference contribution
Journal article
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0%1%
5%9%
11%11%11%12%
14%16%
18%21%
26%31%
42%46%
52%74%
77%96%
100%
Which of the following outputs result from your research activities?
Key outputs currently captured
CompositionDesign
PerformanceSoftware
ExhibitionArtefact
Digital or visual mediaDevice and product
*OtherScholarly edition
Patent/ published patent applicationConfidential report for external body
Website contentResearch datasets
BookWorking paper
Edited bookResearch report for external body
Book ChapterConference contribution
Journal article
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0%1%
5%9%
11%11%11%12%
14%16%
18%21%
26%31%
42%46%
52%74%
77%96%
100%
10
Key outputs are being lost
Outputs beyond publications
CompositionDesign
PerformanceSoftware
ExhibitionArtefact
Digital or visual mediaDevice and product
*OtherScholarly edition
Patent/ published patent applicationConfidential report for external body
Website contentResearch datasets
BookWorking paper
Edited bookResearch report for external body
Book ChapterConference contribution
Journal article
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0%1%
5%9%
11%11%11%12%
14%16%
18%21%
26%31%
42%46%
52%74%
77%96%
100%
11
Key outputs are being lost
Outputs beyond publications• Impact Case Studies• Research sample repositories• Written commentary and advice to agencies
and NGOs• Health education materials• Booklets, DVDs, Video• Public outreach (through seminars)• Public lectures / speech• Radio talk• Guidelines• Materials for museums• Tours• Advice• Field schools• Podcasts• Television• Popular magazines
12
Why UCD need to do something
• UCD bias towards publications as indicators of research excellence continues Bibliometric analysis remains the yardstick
• UCD Research Reputation suffers By not communicating research impact as a key building
block of reputation
• UCD cannot fully communicate the value & relevance of research to funders & other key stakeholders
• UCD falls further behind in telling its research impact story Compared to leading international institutions
13
What is impact?
• “the consequences of an action that affects people’s lives in areas that matter to them”1
1 ESF (2012) The Challenges of Impact Assessment (Pubd online) Available at: <http://www.dfg.de/download/pdf/dfg_im_profil/evaluation_statistik/programm_evaluation/impact_assessment_wg2.pdf> [Accessed 30 Mar 2014]
14
Distinction between Academic and Economic & Societal impacts
15
UCD’s Academic Impact
16%Above world average in 2010-2014?
Source: Elsevier SciVal
16
CollegeNo. Publication in
RMSNo. Publications
in Scopus
Approx. % Coverage in
ScopusScience 5,052 3,733 73.9%Health Sciences 3,714 3,257 87.7%Engineering & Architecture 3,068 1,951 63.6%Agriculture, Food Science & Veterinary Medicine 2,831 1,903 67.2%Human Sciences 2,430 883 36.3%Arts & Celtic Studies 1,367 199 14.6%Business & Law 1,309 564 43.1%Grand Total 19,771 12,490 63.2%
Coverage Issues
Source: Elsevier SciVal & UCD Research Management System 2010 to 2014
17
What gets measured gets done?‘The REF Effect’
Name Overall 2008 2009 2010 2011 2012 2013
% Increase 2008 to
2013University of Leicester 445 34 29 40 96 108 138 305.9%University of Liverpool 384 29 30 31 67 117 110 279.3%Queen's University Belfast 343 28 31 31 51 101 101 260.7%Trinity College Dublin 298 26 32 22 65 74 79 203.8%University College Dublin 348 36 35 50 60 83 84 133.3%
Increasing volume of journal articles in Arts and Humanities per year
Source: Elsevier SciVal
18
Difficulty of measuring book to book citation
• Prestige of the publisher of the book?
– e.g. Danish BFI list of publishers
• Paperback publication of what was originally a
hardback monograph?
• Coverage of books in Scopus continues to grow
—with a plan to reach 120,000 titles by the end
of the year
19
Altmetric for institutionsA measure of societal impact?
• UCD’s highest scoring publication is from the College of Arts & Celtic
CollegeNumber of articles mentioned
in social mediaNumber of articles
in Altmetric for Institutions
By total number of
authorsCollege of Science 1,110 4,619 415College of Health Sciences 1,006 3,901 345College of Agriculture, Food Science and Veterinary Medicine 365 1,761 162College of Engineering and Architecture 124 1,559 139College of Human Sciences 227 1,175 188College of Business & Law 52 586 88College of Arts & Celtic Studies 43 222 66
coverage in Altmetric for Institutions is problematic due to its bias toward counting references with Digitial Object Identifiers
20
What about Economic & Societal
impact?
21
Many ways to measure impact
• Input Measures
– Include research funding, human resources, existing knowledge, equipment and facilities.
• Output Measures
– refer to the measurement of the products of the research activity. The most obvious output measure is publications, but processes or tools used to
disseminate research can also be considered as a type of output indicator
• Expert reviews
– assesses research impact by obtaining information from groups of experts, for example the UK REF uses expert panels to rate the submissions from
universities
• Surveys
– structured surveys can be used as an impact assessment measure. An example of this approach is the Impact Finder Tool, created by the RAND
Corporation.
• Case Studies
– contain a narrative describing research impact with supporting metrics and references to detail the nature and scale of the impact.
• Hindsight studies
– attempt to retrospectively trace an observed research impact to the research inputs, activities, output and outcomes that led to the impact.
• Economic models
– are often used to assess value for money. Econometric analysis is used to assess research impact at a macro-level while cost-benefit analysis is often
used to determine impact from research projects or programmes.
22
Impacts linked to inputs, activities, outputs, outcomes and time
23
IMPLEMENTING IMPACT
24
Context for implementation
• UCD’s Strategy 2015-2020
– Increase the quality, quantity and impact of our research, scholarship and innovation
– Provide an educational experience that defines international best practice
– Consolidate and strengthen our disciplines
• Schedule• Resourcing plan• Governance structure
Draft Impact Implementation Roadmap
Optimise discoverability of publications
• Prepare for publication• Dissemination
approaches• Keeping Track of Your
Research
2. Enhance Academic Impact
3. Capture & Convey Socio-Economic Impact
2015 2016 2017 2018
Capture & convey impact & outputs beyond publications
• Implement Impact System (RMS II)
• Impact collection (using Seed funding round)
• Capture case studies• Classify• Assess• Reward
• Publish impacts
Phase 1 Phase 2
4. Evaluate impact
Creating impact from knowledge
• Academic impact assessment
• Economic Value added (EVA)
• Societal impact assessment
Phase 3
1. Excellence & Impact Framework
Definition & development of the impact & excellence framework
• Requirements• Key Performance Indicators
Planning
25
26
Research Quality Assessment Pilot
• Assessment of the quality and level of research activity in
the School of Archaeology benchmarked internationally
• Provide an overview of how research relates to T&L and
the student experience within the discipline
• Make recommendations to strengthen the discipline
• Inform strategic planning
27
The Process – Outline
• School Research Statement Report
• Individual Staff submissions (4 research outputs)
– Jan 2010-Dec 2014
• Impact Case Study/Studies
• Peer Panel Assessment – site visit 14-15 October
• Panel Report - December
28
Forming the overall quality profile
In forming the overall profile, the panel will assess three
sub-elements:
• Environment (15%)
• Outputs (65%)
• Impact (20%)
29
Research Statement Report
• Environment
– Staff FTE & Headcount
– Research Income €
– Contextual narrative on strategy, organisation, infrastructure, funding, collaboration
• Outputs
– 4 publications per staff
– Benchmarking outputs against peer institutions (incl. citation analysis)
– Research PhD numbers
• Impacts
– Narrative on context for impact
– Impact case studies
30
Collection of Outputs
• 4 per person (2010 -2014) using template
• Outputs can be:– Publications: [including books, book chapters, special issues; journals, monographs,
conference papers, policy evaluation reports, critical review articles etc.]
– Products: [this may include - physical artefacts, digital artefacts, research datasets, software
etc.]
– Patents
• Make outputs available online (where possible)– Institutional Repository / Research Information System
– Online Journal etc.
31
Output Collection Template
32
Capturing Impact
• Title of case study
• Images
• Summary
• Research description
• Details of the impact
• References
33
Timeline
- Collect 4 outputs per academic staff- Collect Impact Case studies
2. Collect Outputs
3. Finalise Documentation
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
4. Peer Review & Final Report1. Preparation
- Planning- Document process- Draft Research Statement Report- Explore indicators
- Finalise Research Statement Report- Finalise Outputs
- Review panel site visit - Final Report
1st April:Staff Briefing
30th June:Deadline for collection of outputs & case studies
1st Sept:Deadline for Finalised Research Statement Report
34
APPENDICESExamples of impact
35
History and Policy papers bring past issues to current
policy debatesHistory Hub – Connecting Past and Present
History does not repeat, but it does rhyme – Mark Twain
What is the impact of pay-TV on sport? Why did the Irish health service develop in the way
that it did? How did small, local hospitals become so important? And was the policy focus misplaced on hospitals rather than on people’s health?
Would the 1950s style of bureaucratic diversity have prevented the ‘groupthink’ that became characteristic of the Central Bank and Department of Finance before the economic crash of 2008?
Did British policy responses to parades disputes during the 1990s peace process militate against a resolution of disputes which continue today?
Image: Postcard of St. Patrick’s Ward, St. Vincent’s
Hospital by paddykilty.wordpress.com
History Hub.ie is based at the School of History & Archives in UCD. It
hosts and podcasts, digitised archival documents and papers written by
historians on current affairs.
Access to the archival material gives people an understanding of the
process of history writing. HistoryHub.ie goes further, delivering open
access, peer reviewed research by leading historians in a format that is
robust, topical, and accessible, thus bringing history to the modern day.
Access to such a wide source of materials (archival documents, policy
papers, podcasts, original documentary videos) means history is no
longer a ‘closed book’. The site offers a unique opportunity to analyse
the past and, most importantly, to learn from it. Active citizenship must
be informed citizenship. HistoryHub.ie remains central to discourse in
Ireland’s modern democracy.
36
Global crisis At the forefront of environmental humanities
What is the relationship between mud-slides, poverty, and economic policy in the Philippines?
What impact has the de-regularisation and
privatisation of ‘natural’ commodities such as water and oil had on our world and its environment?
In Ireland, how has the financial crisis and the
housing collapse changed our living spaces and surroundings?
Image: Pyrite Hell by Paul Reynolds
UCD is at the forefront of interdisciplinary research examining the intersections between environmental and economic crisis, and in particular, capitalism’s transformations of cultural, environmental, and social landscapes.
Led by Dr Sharae Deckard from the School of English, Drama, and Film, UCD hosted a major academic symposium on ‘World-Ecology, World-Economy, World-Literature’ in 2013, which considered issues such as food and water security, energy regimes, disasters, and extreme weather.
In partnership with Dublin City Council, Dublin UNESCO City of Literature, and Cultúr Lab, the symposium was accompanied by a number of events open to the general public, including a photographic exhibition entitled ‘Landscapes of Crisis’, poetry readings, and discussion evenings.
37
War: The Facts Ireland’s first Centre for War Studies in UCD
According to the Peace Pledge Union, in recorded history since 3600 BC, over 14,500 major wars have killed close to four billion people – two-thirds of the current world population.
Since 1495, no 25-year period has been without war.
One out of every two casualties in war is a civilian caught in the crossfire.
When it comes to war studies, we learn about past atrocities to prevent history repeating itself. Founded in 2008 and led by Professor Robert Gerwarth, the Centre for War Studies in UCD promotes a wide range of international research activities focused on the origins, nature, and consequences of all war-related violence, from ancient times to the present day.
The Centre attracted the first ERC Grant ever in the Humanities in Ireland. It employs research staff from eight different countries and its output is translated into 25 languages.
38
Culture and Heritage at UCD
Source: The Economic, Cultural and Social Impact of UCD