Research evaluation: is it our business? Librarians in the brave new world of research evaluation
Andria McGrathSenior Information Specialist, Research Support, King’s College London
What is research evaluation?
• External evaluation – in the UK the RAE now REF3 major elements of the RAE: – publication outputs, – grant income awarded, – research students
• Vital importance – link to income• Benchmarking – Rankings• Internal evaluation
The players in institutions
• Research Office• PVC Research• Research group/dept heads• Administrators• Librarians – publications expertise• Institutional repository managers
3
Evolving interest
• REF events at King’s 2008 -2010• 1st event – Beyond the RAE – representatives
from 75 universities – 1/3 librarians• Survey –
– biggest challenges – verifying publication lists– Biggest concerns for REF – bibliometrics – data
sources, subject differences, skilling up to cope
• http://www.kcl.ac.uk/iss/support/ref
4
Events Nov 08 and June 09
• Focus on bibliometrics – pilots• Pre-event survey before Nov event:
– Only about half of institutions had a centralized publications data collection system
– One or two mentioned Symplectic– One mentioned something called Pure
5
Data collection systems
• Some – only concerned with collecting RAE pubs• Others – aimed at collecting all publication info• Wider uses – profile pages / internal evaluation• Some creating internal ‘home grown’ systems
– Separate systems for 3 RAE data elements– Integrated system, taking feeds from other systems
(eg Research Gateway at King’s)
REF focus changed
• Bibliometrics element toned down• Impact (4th King’s REF event in June 2010)• Not publication impact – practical impacts of
research• Research leaders had seen the light on
– data collection systems – bibliometrics
7
CRIS Current Research Information SystemsAlready common in mainland Europe
8
CERIF
• Common European Research Information Format
• A standard – uses XML• Objects or entities with attributes
eg project, person, organizational unit; • Relationships – 'linking relations’ • Rich semantics – roles and time
9
R4R project - Ready for REF
• JISC project – King’s and Southampton• CERIF4REF XML schema created• Plugins for Eprints, Dspace and Fedora• Transfer data to REF collection system• Research admins, IR managers, systems staff
and librarians involved• http://r4r.cerch.kcl.ac.uk/
10
Librarians and CRIS systems
• St Andrews experience• If research office led – get in on the ground
floor• Librarians have plenty of expertise to offer
11
Which librarians?
• Senior managers• Subject/liaison librarians• Research support librarians• Instutional repository managers• Information resources depts. – subs• Enquiries staff
12
Strategic directions for libraries
• RLUK strategic themes• 1st – redefining the research library model• New services / new roles for staff• Value of libraries for research and researchers
(RIN/RLUK report)• RLUK – Reskilling for Research (forthcoming)• Research support librarians
13
Some new areas for librariansassociated with research evaluation• CRIS systems & CERIF• New ways of interacting with bibliographic /
citation databases• APIs – for publication or citation data• Discovering other institutional systems
eg grants DBs / student systems that feed into CRISs
• Bibliometrics principles and new products14
New opportunities
• Showing our value– To administrators / research office– To academics and senior research leaders– To Graduate School
• Opportunities to advocate Open access
15
Bibliometrics – the new buzz word
Awareness growing
•By librarians
•By research leaders in institutions
•By academics
16
Librarians – natural synergy
• Citation databases• Citation reports• Training offered – eg by Thomson Reuters• Training points up limitations of basic
measures• Normalizing of measures crucial but difficult
without specialised products
17
By research leaders
• A few initiates who understand the issues leading the way
• Percentiles tables in Essential Science Indicators used for evaluations
• H-index being used by some• At least one institution has employed a
bibliometrician
18
Academics
Some beginning to take an interest
•Discovering:– ResearcherID– Citation reports in citation databases– Google Scholar
•Graduates – training opportunity – “Making an impact”
19
Holy grail
• Research information system• Dashboard for research managers – eg for
grant information• Captures data with minimum effort• Integrated with bibliometrics• For internal & external purposes – evaluation
and promotional
20
Is it our business?
21