Upload
hathu
View
215
Download
1
Embed Size (px)
Citation preview
BEYOND BIBLIOMETRIC
INDICATORS: A 360°
EVALUATION OF RESEARCH
IMPACT
W. Nicol Keith, Glasgow University
The professional development
framework to realise the
potential of researchers
Major new approach to researcher
development
Builds the UK research base
Develops world-class
researchers
Enhances the personal, professional
and career development of
researchers
We train & expect our
researchers to be well
rounded
Should we consider research
assessment & impact in the same way?
Why do we need research assessment?
Who do we assess?
Institute: a group of researchers with an articulated shared mission, operating within one or
more research programmes/units under the same management
Defining the context: What do we assess?
Examples of complex impact assessments in practice
How do we assess? Evidence for the REF
REF Examples of Impact & evidence
Examples of complex impact assessments in practice:
Training future leaders in field of cancer control
Fellowship Portfolio • The UICC has a range of Fellowship schemes & these can be viewed as
a portfolio.
• An impact evaluation of the UICC Fellowship portfolio was conducted The survey spanned years 1997- 2010 and over 600 Fellows reported on the perceived impact of the Fellowship on their research capacity, skills/knowledge & professional influence levels.
• Evaluation to date highlights the power of the portfolio as each has unique features which together deliver training relevant to gaps in skills and knowledge.
Who do we assess?
Background to the review
Summary of Grantees
– ICRETT
• Survey spans years 1997- 2010
• 1120 number of grantees contacted
• 447 grantees responded
• Map shows geographic distribution of respondents
YY
• Survey spans years 1998-2010
• 139 number of grantees contacted
• 95 grantees responded
• Map shows geographic distribution of respondents
ACSBI
• Survey spans years 1998-2010
• 103 number of grantees contacted
• 57 grantees responded
• Map shows geographic distribution of respondents
Overview of Fellowship Satisfaction & Performance
Success is measured across several areas (Research Capacity & Output; Skills & Knowledge; Professional Influence).
YY grants establish scientific projects and as such generate high numbers of joint publications.
ICRETTs are short term grants and for technology/skills transfer which lead to publication, increased knowledge or implementation of skills in for example a clinical setting.
ACSBI fellowships are advanced training grants which establish young investigators
Evidence of activity: Joint publication
Case study: YY Evaluation & Key Findings
• The objectives of the YY Fellowship are – To initiate, set up or pursue bilateral cancer research projects with collaborating investigators abroad
– To exchange and exploit complementary skills and material
– To receive training in advanced experimental research methods and techniques
• It is aimed at individuals with scientific or medical qualifications & a minimum of two years postdoctoral
experience who are actively engaged in cancer research
• Survey spans years 1998-
2010
• 139 number of grantees
contacted
• 95 grantees responded • number of invalid emails
unknown
• Map shows geographic
distribution of respondents
Professional Development
• Making an impact on Project Management & Leadership skills
• Focusing the future Research Direction
• Building the Research Programme
Increasing Research Capacity & Output
Building Skills & Knowledge
Increasing Professional Influence at Home & in the wider
International Community
Reasons for Success
• Reasons for Success – The survey indicates that YY Fellowships have a positive impact on all major key
performance indicators associated with a well rounded, successful international cancer researcher.
– In order to gain additional insight into the possible underlying factors driving success, the associations between variables was analysed using cross tabulation analysis. This analysis complements the basic survey output.
– Cross tabulation analysis is useful for showing a side by side comparison of survey questions to determine how they are interrelated. In statistical terms, it is a joint distribution between two (or more) variables.
• Summary of Findings – An Effective HOST/MENTOR Increases Fellowship Success – Publishing with the MENTOR increases chances of gaining Funding – The increased Knowledge & Skills resulting from the Fellowship help define future
Research Direction & Build the Research Programme – A successful Fellow has increased Project Management & Leadership skills – A successful Fellow is Internationally competitive – A supportive HOME environment is important to the implementation of training
An Effective MENTOR / GRANTEE Relationship Increases
Fellowship Success
Fellowship Success
• An effective Mentor/Grantee relationship
results in:
• Achieving Objectives
• An increase in Research Skills
• Increased Knowledge
• Publications
• This allows for:
• increased Funding
• Increased Professional Influence
• A positive impact on defining Research
Direction & Building a Research
Programme
• Together, these result in Improved
Management & Leadership Skills & an ability to
Compete at an International Level
A supportive HOME environment allows successful
implementation of training
For the benefits of
the Fellowship
experience to be
fully exploited, the
Home Supervisor &
Institution needs to
be fully supportive
of the returning
Grantee
Benefits to the Home Institute
Skills/tech transfer
Publications grantee/host supervisor: publications are an
internationally recognized indicator of success and
reputation. Fellows publications are contributing to the world
knowledge of cancer.
Returning fellows have an increased ability to attract
funding. The flow of funds into home institutes increases the
research capacity and knowledge generation.
The home benefits from the networks established by the
fellows as these extend influence internationally.
Impact Case Study: Leadership, influence &
capacity building
Testimony of Dr Rajesh Dikshit of ICRETT and ACSBI Fellowship opportunities: YouTube Fellowship impact
presentation at WCC
Impact Evaluation of the
American Cancer Society international
fellowships for beginning investigators
(ACSBI)
Aligning Sponsor strategy and training
opportunities
Geographic distribution of grantees
After 2007
Between 2002 -2007
Before 2002
ACSBI
• Survey spans years 1998-2010
• 103 number of grantees contacted
• 57 grantees responded
• Map shows geographic distribution of respondents
Area of work
Area of work & date
Job title
Job title & date
UICC Experience: Who or what do we assess?
• Impact assessment is context dependant – Low-middle income
countries
– Lab training / service development /implementation of technology
• Use it to fulfil current needs & identify future needs
• Complex research situations require more than impact factors, H-index & funding income
A few thoughts on evidence of influence &
impact……Social media & open access
• Some Advantages
– Stay current on new research
– Find resources
– Share & discuss ideas
– Develop local / international hubs
– Career advice
– Job opportunities
– Evidence of reputation, influence & impact
Researcher ID: visualising influence
• Geographic
locations for
publications
that have
cited our
work
• Geographic
locations of
co-authors
• Endorsements for Skills
& Experience
• Recommendations
• Publications
Example of Science Networking Site: ResearchGate
• Assess impact of
your publications
– Downloads
– views
Example of Science
Networking Site:
ResearchGate
• Assess impact of
your institute
– Downloads
– Views
– collaborations
Example of Science Networking
Site: ResearchGate
In conclusion: Why, Who & What
Choose the right tools to measure impact
Look at the appropriate evidence