What is bibliometrics and why we should care about it? Definition Why do we need bibliometrics? What to measure and how? Usage – Peer review – Citations

What is bibliometrics and why we should care about it?

• Definition• Why do we need bibliometrics?• What to measure and how?• Usage – Peer review – Citations – Alternative

metrics• Conclusion

Tullio Basaglia

CERN Library, Geneva

21/10/2014, AIS-Grid School - T. Basaglia

Definition

• Bibliometry = biblion (book) and metron (measure)

• It’s the discipline aiming at determining the impact of scholarly journals, journal articles, authors, and research institutions.

• By impact we mean: how the work produced by scholars is received, used, assessed, and critiqued, how their contributions are recognized, and finally how influential they are on different scientific communities.

• It is worth noting that, in the era of the web and of the social networks, the amount and the diversity of the “objects” (videos, tweets,...) produced by scholars has made the scholarly communication landscape more complex; this allows a much better “visibility” of scientific communication, also outside the scientific community. We talk about “webometrics” when we indicate the measurement of impact of scientific publications on the Web (=mainly social networks).


Definition – cont’d

• The impact of an article, a journal, an author, an institution in terms of their contribution to the advancement of research in any research domain was always considered very important. Such impact was (and still is) obviously correlated with prestige and recognition within a community of scholars.

• Today, in a time of fast and steadily growing volume of publications and of strong competition for career and funding, in order to certify the quality of research in the most ”objective” manner, we need to perform measurements. That’s when bibliometrics comes into play. It should define what to measure and how.


Why do we need bibliometrics?

• We need to measure impact for essentially three reasons.

oResearch assessment (a related term is “Scientometrics”)oFaculty evaluationoResource allocation to finance research activities


What to measure and how?


Yes No Yes ProblematicMeasurable?

Societal impact

• We should not forget that there is a long-term impact of scientific production, and it’s the one on the society at large.• There is a need for evaluative and decision-making tools for

assessing the contribution of the public sector investments in Science and Technology to economic growth and social well-being.• However, this ‘societal’ impact should not be interpreted

reductively as “the contribution of Science to GNP”. Such investments can have wide-ranging effects on the general level of scientific education in the society.


Usage

• COUNTER: Counting Online Usage of NeTworked Electronic Resources. C. is an international initiative to improve the reliability of online usage statistics. It is supported by the publisher, vendor, and librarian communities.

• Number of downloads: “User requests include viewing, downloading, emailing and printing of items, where this activity can be recorded and controlled by the server rather than the browser. Turnaways will also be counted.” (COUNTER code of practice).

• There is a demonstrated correlation between downloads and citations. • Articles potentially citeable, but not accessible (=barriers imposed by

the publisher), influence this correlation. • The (potential) contribution of Open Access publishing (today, ~20%

of the total) to the usage of scientific literature needs to be taken into account. 21/10/2014, AIS-Grid School - T. Basaglia

Peer review: a definition and a (quite) radical opinion

• Peer review is the evaluation of work by one or more people of similar competence to the producers of the work (Wikipedia)

• “It is ordinarily claimed that journals play two intellectual roles: a) to communicate research information, and b) to validate this information for the purpose of job and grant allocation.

• […] the role of journals as communicators of information has long since been supplanted in certain fields of physics, so let's consider their other role. Having queried a number of colleagues concerning the criteria they use in evaluating job applicants and grant proposals, it turns out that the otherwise unqualified number of published papers is too coarse a criterion and plays essentially no role. […] "hot preprints" on a CV can be as important as any publication.

• So many of us have long been aware that certain physics journals currently play NO role whatsoever for physicists. “Winners and Losers in the Global Research Village, Paul Ginsparg, 1996 [founder of arXiv.org]


Citations – the Impact Factor


•Impact Factor of a journal (Thomson-Reuters):• A = the number of times that articles published in a journal in 2006 and 2007,

were cited by articles in indexed journals during 2008. • B = the total number of "citable items" published by that journal in 2006 and

2007. ("Citable items" are usually articles, reviews, proceedings, or notes; not editorials or letters to the editor.)

• 2008 impact factor = A/B.

300 cit. in 2008 to 2006-2007 articles = IF of journal X for the year 2008 is

3.100 citable articles published in 2006-7

• Originally it was created (E. Garfield, 1955) as a tool to compare journals’ impact in order to decide which one(s) was/were worth subscribing to.

Criticism on the IF•“The journal impact factor was developed as a means to measure the impact of scientific journals. Over time, its use has been extended to measuring the quality of scientific journals, the quality of individual articles and the productivity of individual researchers.”

•“Universities in Germany, for instance, regularly plug the impact factor of journals in which scientists publish into formulae to help them determine departmental funding. The Italian Association for Cancer Research requires grant applicants to complete worksheets calculating the average impact factor of the journals in which their publications appear. [...] [In Finland] government funding for university hospitals is partly based on publications points, with a sliding scale corresponding to the impact factor of the journals in which researchers publish their work.”

• “The European Association of Science Editors recommends that journal impact factors are used only – and cautiously – for measuring and comparing the influence of entire journals , but not for the assessment of single papers, and certainly not for the assessment of researchers or research programmes either directly or as a surrogate.”

Source: EASE statement of inappropriate use of impact factors, 2012

Tullio Basaglia - GS-SIS

Citations: the h-index


H-index: The h-index is an index that attempts to measure both the productivity and impact of the published work of a scientist or scholar. The index is based on the set of the scientist's most cited papers and the number of citations that they have received in other publications.A scholar with an index of h has published h papers each of which has been cited in other papers at least h times. Thus, the h-index reflects both the number of publications and the number of citations per publication.(Source: Wikipedia)

Criticism on the h-index

• The h-index does not account for the number of authors of a paper. In the original paper, Hirsch suggested partitioning citations among co-authors.

• The h-index is bounded by the total number of publications. This means that scientists with a short career are at an inherent disadvantage, regardless of the importance of their discoveries. Had Albert Einstein died after publishing his four groundbreaking Annus Mirabilis papers in 1905, his h-index would be stuck at 4 or 5. However, as Hirsch indicated in the original paper, the index is intended as a tool to evaluate researchers in the same stage of their careers.

(Source: Wikipedia)


Problems linked to citations as a measure of impact

• The necessity of persistent identification of authors becomes even more important. The ORCID (Open Research and Contributors ID) project aims at providing a solution.

• Citation to data sets are underrepresented in this landscape. Efforts in unique identification of datasets (and software) might help these objects emerge from the citation metrics landscape. Attribution of DOIs (Digital Object Identifiers) will help.


San Francisco Declaration on Research Assessment


•There is a pressing need to improve the ways in which the output of scientific research is evaluated by funding agencies, academic institutions, and other parties. ... The Journal Impact Factor is frequently used as the primary parameter with which to compare the scientific output of individuals and institutions. • The Journal Impact Factor, as calculated by Thomson Reuters, was originally created as a tool to help librarians identify journals to purchase, not as a measure of the scientific quality of research in an article. With that in mind, it is critical to understand that the Journal Impact Factor has a number of well-documented deficiencies as a tool for research assessment. • These limitations include: A) citation distributions within journals are highly skewed; B) the properties of the Journal Impact Factor are field-specific: it is a composite of multiple, highly diverse article types, including primary research papers and reviews; C) Journal Impact Factors can be manipulated (or "gamed") by editorial policy; and D) data used to calculate the Journal Impact Factors are neither transparent nor openly available to the public.

San Francisco Declaration on Research Assessment (2013, by a group of scientific publishers)

DORA


• We make a number of recommendations for improving the way in which the quality of research output is evaluated. A number of themes run through these recommendations: • the need to eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations;

• the need to assess research on its own merits rather than on the basis of the journal in which the research is published; and

• […] exploring new indicators of significance and impact.

In summary…


• Linking reward (carrier, funds) to invalid outcome measures leads to predictable and undesirable results

• An example: when company X rewarded mechanics for car repairs, more “car repairs” were authorized by customers

• The problem lies in the mechanical link between metrics and incentives

Alternative metrics (alt-metrics)• It’s still in its infancy

• Problematic validation of citation metrics. Web citations suffer obviously from a problem of quality control

• We assume that Twitter and blogs are mainly used by non-scholars. Altmetric.com uses an algorithm to decide if a tweet comes from a layman or not. How effective is this algorithm? Can we be sure that a certain content is authoritative?

• Social news (Reddit, Slashdot): no research has emerged aiming at tracking scholarly metrics on those recommendation sites

• Wikipedia: studies are available on the impact of Wikipedia articles in scholarly literature


(Kind of a) conclusion

• In the absence of any sensible performance metrics for the transmission of knowledge, bibliometric measurements have been adopted, that serve that need. New technology is needed, that should capture the complexities of scientific interaction.

• Probably, a combination of usage, citation, and other data (=multidimensional indicator) should be used to develop metrics of scholarly impact that go beyond the purely quantitative approaches in use today


Questions?


[email protected]

Documents

What is bibliometrics and why we should care about it? Definition Why do we need bibliometrics? What to measure and how? Usage – Peer review – Citations