Citations count! Bibliometrics in the

Repository and for REF at Cardiff University

information

Kate BradburySenior Consultant, Research Support

BradburyK1@cardiff.ac.uk

Outline

1. Bibliometrics – introduction

2. Bibliometrics in ORCA (Cardiff’s repository)

3. Bibliometrics for REF

4. New developments

1. Bibliometrics - introduction

1.1 What is bibliometric data?

Data resulting from quantitatively analysing academic publications.

http://www.wordle.net/

1.2 Some popular bibliometric measures:

• Citations: The number of times a journal article has been cited by other authors.

• Impact Factor (Thomson): The journal impact factor is the average number of times articles from the journal published in the past two years have been cited in the Journal Citation Reports year.

• SJR (SCImago Journal Rank)/SNIP (Source Normalized Impact per Paper): used by Scopus Analytics for journal rankings.

• H-score or H-index: a scholar with an index of h has published h papers each of which has been cited by others at least h times. Variations include the g-score (more weight given to highly cited publications) and m-score (length of time publishing taken into account).

1.3 New measures - altmetrics

• Alternative metrics.

• Measure of attention not quality.

• Measures “data trail” or “digital exhaust”.

• Counts may include downloads, page views, mentions in

news, social media and blog mentions, readers in

reference manager sites.

• Not an alternative to using academic citations but an

alternative to only using academic citations.

1.4 Some uses for bibliometric data

• Publicising work/profile & in CVs.

• Finding journals to publish in.

• Finding partners for collaborative and interdisciplinary working.

• Determining wider impact from Altmetrics.

• REF - Citation data & contextual data is being used for all of panel A sub-panels and some panel B sub-panels.

• Requests from grant funders for citation data.

• Major component of League Tables eg QS Top Universities (Scopus), Times World University rankings (Web of Science), Shanghai Index (Web of Science).

1.5 Some limitations of bibliometric data• Comparing different disciplines – need benchmarks and

normalisation.• Publication types not all evenly covered.• May be working in a very small, specialised field not well

covered by citation data.• Bias in favour of English language publications.• Early-career researchers – takes time for citations to

accumulate.• Analysis of new areas of research may be limited.• Negative citations.• Self-citations.

Therefore should be used cautiously & should be complementary to other analysis of research quality & impact.

2. Bibliometrics in ORCA

2.1 Why add bibliometrics to the repository?

• Information for readers – showcasing Cardiff’s research.

• Information for authors – indicating to authors the reach of their research, both in social media and academic citations.

• Information for analysis – download citation data to analyse it, for example for a school or department.

• Easy linking to citing works.

• Livens up the text!

ORCA Institutional Repository: citation data

ORCA Institutional Repository – article altmetrics

3. Bibliometrics for REF

3.1 Background• 2008: HEFCE proposed basing the next REF mainly on bibliometric data & ran

a consultation exercise 2007-2008.

• Commissioned a report from Leiden University to analyse the 2001 RAE exercise data.

• Commissioned a pilot exercise & expert advisory groups.

• “The pilot exercise concluded that citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality in the REF; but there is scope for such data to inform and enhance the process of expert review..”

• Used citation data for outputs plus contextual data for Panel A and some Panel B Units of Assessment.

• Now running an independent review of the role of metrics in research assessment commissioned by David Willetts & consulting again.

• New Universities minister - Greg Clark - new policy?

3.2 REF2014 panels using citation data

3.3 Contextual data for REF2014

• Provides context for the REF panels to judge the citation count for an individual article, in relation to the field of research and a publication’s age.

• Each journal in the database is assigned to one or more subject classifications, using their ‘All Science Journal Classification’ (ASJC) codes. 

• The citation count for the article can be judged alongside the contextual data for the subject where the journal is classified. ie how many citations would place it in the top 1%, 5%, 10%, 25% of other articles for the same subject published in the same year.

3.4 REF 2014 support for bibliometrics at Cardiff

• Training & support for use of bibliometric data.

• Checked all REF journal outputs for the relevant panels against Scopus to ensure that it was returning a citation count.

• If not, checked why. Reasons include:

– Incorrect data

– Missing issue

– Missing articles

– Journal not indexed on Scopus

– Article too recent

• Reported to Scopus if required & corrected.

4. New developments

4.1 Cardiff - SciVal subscription for 6 months trial

• Assessment of research performance based on publication counts and citation data measured alongside benchmarked data for the subject.

• All research universities.

• Groups – country, Russell group, self-created groups (eg. Research groups).

• Subjects – provided by SciVal or self-created subject/journal groups.

• Collaboration data.

4.2 Cardiff – new CRIS system

• We are just finalising our selection of a new Current Research Information System (CRIS).

• Currently exploring potential with bibliometrics & altmetrics.

• Issues:

– Do not want to lose ORCA functionality – citations and altmetric data.

– Importing data from Scopus/potentially SciVal as well as Thomson data.

4.3 Some recent developments & trends

• HEFCE review to “consider the robustness of metrics across different disciplines, and assess their potential contribution to the development of research excellence and impact.”

• Increase in the number of UK university subscriptions to Scival (Elsevier) and InCites (Thomson) and corresponding increase in library jobs for bibliometricians/research data/research information analysts. Increased opportunities for collaboration with colleagues from planning, research office, business intelligence sections.

• Intensified university interest in League Tables & how the position of individual universities can be improved.