Upload
stefanie-haustein
View
938
Download
0
Tags:
Embed Size (px)
Citation preview
Scientific interactions and research evaluation: from bibliometrics to altmetrics
Stefanie Haustein [email protected] @stefhaustein crc.ebsi.umontreal.ca/sloan
Invisible Colleges
Père Marin Mersenne (1588-1648)
Henry Oldenburg (1619-1677)
http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpg http://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg
Invisible Colleges
Père Marin Mersenne (1588-1648)
Henry Oldenburg (1619-1677)
http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpg http://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg
Information Hubs
Scientific Societies
L’Académie royale des sciences 22 December 1666
The Royal Society 28 November 1660
http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-Louis_XIV_und_Colbert_in_der_Akademie.jpg https://www.rcseng.ac.uk/museums/hunterian/images/lost-museums-2011/the-royal-society-repository/Image%201%20-%20GreshamCollege.jpg/image_preview http://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg
Scientific Societies
L’Académie royale des sciences 22 December 1666
The Royal Society 28 November 1660
http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-Louis_XIV_und_Colbert_in_der_Akademie.jpg https://www.rcseng.ac.uk/museums/hunterian/images/lost-museums-2011/the-royal-society-repository/Image%201%20-%20GreshamCollege.jpg/image_preview http://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg
Institutionalization of Science
Scientific Journals
Le journal des sçavans 5 January 1665
Philosophical Transactions 6 March 1665
http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpg http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg
Scientific Journals
Le Journal des sçavans 5 January 1665
Philosophical Transactions 6 March 1665
http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpg http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg
Registration Certification
Dissemination Archiving
Scientific Articles: 17th to Early 19th Century • Experiments and descriptions of the natural world • Avoiding “fine speaking” • Various styles of arguing • Qualitative and personal judgements
Harmon, J.E. & Gross, A.G. (2007). The Scientific Literature. A Guided Tour. Chicago: University of Chicago Press.
Modern Scientific Articles • Professionalized and highly specialized • Increased focus on data, graphs, tables and theory • Impersonal, technical and codified • Style guides and gatekeeping • Citations • Introduction, Methods, Results and Discussion
Larivière, V., Archambault, É. & Gingras, Y. (2008). Long-term variations in the aging of scientific literature: From exponential growth to steady-state science (1900-2004). Journal of the American Society for Information Science and Technology, 59(2), 288-296. Sollaci, L.B. & Pereira, M.G. (2004). The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey. Journal of the Medical Library Association, 92(3), 364-371
Sollaci & Pereira (2004, p. 365)
Proportion of IMRaD adoption in medical journals Number of references 1900 to 2004
Larivière, Archambaul & Gingras (2008, p. 293)
arXiv submission statistics from http://arxiv.org/stats/monthly_submissions Larivière, V., Lozano, G.A. & Gingras, Y. (2014). Are elite journals declining? Journal of the Association for Information Science and Technology, 65(4), 649-655.
• Improved access • Acceleration
• Collaboration • Peer review • Distribution of preprints
• Decreasing importance of scientific journals • Journal functions • Diversification of publication
venues • Appearance and symbolic
capital of journals unchanged
Digital Revolution Submissions to arXiv
Share of top 1% most cited papers
Larivière, Lozano & Gingras (2014, p. 652)
Academic Publishing Market
Larivière, V., Haustein, S. & Mongeon, P. (in press). The oligopoly of academic publishers in the digital era. PLoS ONE.
Larivière, Haustein & Mongeon (in press)
• Aggravation of serials crisis
• Elsevier: €3,400 subscription price, 30% increase
• Profit margins of commercial publishers up to 40%
• Decline of scientific societies as publishers
• >50% of papers owned by five major publishers
Budapest Open Access Initiative
Open Access
“immediate, free availability on the public internet, permitting any users to read, download, copy, distribute, print, search or link to the full text of these articles”
Budapest Open Access Initiative (2002)
Archambault et al. (2013, p. 13)
Freely available journal papers 2004 to 2011 • Gold and Green • Libre and Gratis • Hybrid journals
• Elsevier: $500 to 5,000 • Springer: $3,000 • Wiley: $3,000
Archambault, É., Amyot, D., Deschamps, P., Nicol, A., Rebout, L. & Roberge, G. (2013). Proportion of Open Access Peer-Reviewed Papers at the European and World Levels 2004-2011. Report for the European Commission. http://www.science-metrix.com/pdf/SM_EC_OA_Availability_2004-2011.pdf
Open Science “opening up the research process by making all of its outcomes, and the way in which these outcomes were achieved, publicly available on the World Wide Web”
Kraker et al. (2011, p. 645)
Kraker, P., Leony, D., Reinhardt, W. & Beham, G. (2011). The case for an open science in technology enhanced learning. International Journal of Technology Enhanced Learning, 3(6), 643-654.
• Open Data • Open Source • Open Methodology • Open Access • Open Peer Review
Björneborn, L. & Ingwersen, P. (2004), Toward a basic framework for webometrics. Journal of the American Society for Information Science and Technology, 55(14), 1216–1227.
Definition of Scholarly Metrics
adapted from: Björneborn & Ingwersen (2004, p. 1217)
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics altmetrics
Otlet, P. (1934). Traité de documentation: le livre sur le livre, théorie et pratique. Pritchard, P. (1927). Statistical bibliography or bibliometrics? Journal of Documentation, 25, 348-349..
Definition of Scholarly Metrics
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics altmetrics
“La «Bibliometrie» sera la partie définie de la Bibliologie qui s'occupe de la mesure ou quantité appliquée aux livres.” “the application of mathematics and statistical methods to books and other media of communication”
Pritchard (1969, p. 348)
Otlet (1934, p. 14)
Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multidimensional indicators of performance (pp. 263–287). Cambridge, MA: MIT Press. Rousseau, R. & Ye, F. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 10–12. doi:10.1007/s11434-013-5939-3
informetrics
scientometrics cybermetrics
webometrics altmetrics
“study and use of scholarly impact measures based on activity in online tools and environments” “a good idea but a bad name”
Rousseau & Ye (2013, p. 2)
Priem (2014, p. 266)
bibliometrics
Definition of Scholarly Metrics
Definition of Scholarly Metrics
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics altmetrics
scholarly metrics
Scholarly metrics are metrics based on acts and events (e.g., viewing, reading, saving, diffusing, mentioning, citing, reusing, modifying) related to scholarly documents (e.g., papers, blog posts, datasets, code, notes) or scholarly agents (e.g., researchers, universities, journals).
Gross, P.L.K. & Gross, E.M. (1927). College libraries and chemical education. Science, 66(1713), 385-389..
Citation analysis for objective collection management
Bibliometrics for Library Mangement
Gross & Gross, 1927, p. 386
Journals cited in the Journal of the American Chemical Society 1926
Garfield, E. (1955). Citation indexes for science. A new dimension in documentation through association of ideas. Science, 122, 108-111.
Bibliometrics for Information Retrieval
Retrieval tool to handle information overload
• Institute for Scientific Information • Science Citation Index
• Source Author Index • Citation Index
“It would not be excessive to demand that the thorough scholar check all papers that have cited or criticized such papers, if they could be located quickly. The citation index makes this check practicable.”
Garfield, 1955, p. 108
Derek J. de Solla Price Science since Babylon (1961) Little Science – Big Science (1963)
Bibliometrics for Sociology of Science
Price, D. J. d. S. (1961). Science Since Babylon. New Haven / London: Yale University Press, Price, D. J. d. S. (1963). Little Science, Big Science. New York: Columbia University Press.
Merton, R. K. (1988). The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. Isis, 79, 606–623.
Robert K. Merton • Social norms of science
• Communalism • Universalism • Disinterestedness • Organized skepticism
• Matthew effect
Bibliometrics for Sociology of Science
“symbolically, [the reference] registers in the enduring archives the intellectual property of the acknowledged source by providing a pellet of peer recognition of the knowledge claim” Merton (1988, p. 621)
• Performance measurement and policy instrument
• Commercialization
Bibliometrics for Research Evaluation
“When used properly, bibliometric indicators can provide a ‘monitoring device’ for university research-management and science policy. They enable research policy-makers to ask relevant questions of researchers on their scientific performance, in order to find explanations of the bibliometric results in terms of factors relevant to policy.”
Moed et al. (1985, p. 131)
Moed, H.F., Burger, W.J.M., Frankfort, J.G, van Raan, A.F.J. (1985). The use of bibliometric data for the measurement of university research performance. Research Policy, 14(3), 131-149.
• Oversimplification publications = productivity citations = impact
• Uninformed use and misuse Impact Factor h-index
• Adverse effects “salami” publishing honorary authorship self-citations citation cartels
Bibliometrics for Research Evaluation
Hvistendahl, M. (2013). China’s publication bazaar. Science, 342(6162), 1035-1039. van Noorden, R. (2013). Brazilian citation scheme outed: Thomson Reuters suspends journals from its rankings for ‘citation stacking’, Nature, 500(7464), 510-511.
Criticism against current form of research evaluation Ø Alternative forms of research ouput
Ø Alternative use and visibility of publications
Altmetrics
“We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem.”
…
Priem et al. (2010)
Information overload
…
Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Alt-metrics: a manifesto. October. Retrieved from http://altmetrics.org/manifesto/ Piwowar, H. A. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159.
Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215. (for Mendeley reader counts only: 2010-2012 PubMed/WoS papers) Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. (for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI)
Altmetrics
What do we know?
%
Altmetrics
What do we know?
Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215. (for Mendeley reader counts only: 2010-2012 PubMed/WoS papers) Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. (for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI)
%
Altmetrics
What do we know?
Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215. (for Mendeley reader counts only: 2010-2012 PubMed/WoS papers) Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. (for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI)
%
Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830.
Altmetrics
What do we know? Biomedical & Health Sciences
Social Sciences & Humanities Life & Earth Sciences
Natural Sciences & Engineering Mathematics & Computer Science Tw
itter c
over
age
Docu
ment
type
Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830.
Altmetrics
What do we know? Biomedical & Health Sciences
Social Sciences & Humanities Life & Earth Sciences
Natural Sciences & Engineering Mathematics & Computer Science Tw
itter c
over
age
Meaning of metrics Opportunities • Heterogenous users • Diverse motivations
Challenges • Understanding underlying processes • Determining the meaning of metrics
Meaning of metrics
Bertin, M., Atanassova, I., Gingras, Y., & Larivière, V. (in press). The invariant distribution of references in scientific articles. Journal of the Association for Information Science and Technology.
Bertin, Atanassova, Gingras & Larivière (in press, p. 13)
Distribution of references along the IMRaD structure
Citing in a journal article
Meaning of Metrics
Haustein, S., Bowman, T.D. & Costas, R. (in press). Interpreting “altmetrics”: viewing acts on social media through the lens of citation and social theories. In Sugimoto , C.R. (Ed.). Theories of Informetrics: A Festschrift in Honor of Blaise Cronin.
Acts leading to (online) events used for metrics
Haustein, Bowman & Costas (in press, p. 13)
RESEARCH OBJECT
Opportunities • Detailed life cycle of scientific output • Fine-grained indicators and adequate benchmarks
Challenges • Versions of research output • Publication dates
Time Analysis
Time Analysis Journal article
• Submitted manuscript • Revised manuscript • Accepted manuscript • Version of Record
• Online publication • Journal issue
• Online date • Issue month
Ø Adjusting indicators
3 March 2014 15 July 2014
21 January 2015 February 2015
Time Analysis
Haustein, S., Bowman, T.D. & Costas, R. (in press). When is an article actually published? An analysis of online availability, publication, and indexation dates. Proceedings of the 15th International Conference on Scientometrics and Informetrics, Istanbul, Turkey.
Tweets before publication?
Communities of Attention Opportunity • Differentiating between types of use • Measuring social impact
Challenges • Determining engagement • Identifying users
Level of engagement on Twitter
Communities of Attention
!Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R. & Larivière, V. (in press), Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology. doi: 10.1002/asi.23456
Communities of Attention
Automated Twitter bots
Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R. & Larivière, V. (in press), Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology. doi: 10.1002/asi.23456
tweeting preprints from arXiv subject area
tweeting topic-relevant papers
selection of interesting papers
Communities of Attention
exposure
enga
gem
ent
median dissimilarity with paper title
med
ian
num
ber o
f fol
low
ers
influencers / brokers
orators / discussing
disseminators / mumblers broadcasters
tweet text differs from paper title
tweet text is identical to paper title
few followers many followers Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
Communities of Attention
708 of 125,083 users (0.6%) tweeting WoS papers published in 2012 (>100) Node size represents number of papers
Network of users tweeting the same papers
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
Communities of Attention
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
Communities of Attention
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
Communities of Attention
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
Communities of Attention
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
Stefanie Haustein
Thank you for your attention! Questions?
[email protected] @stefhaustein crc.ebsi.umontreal.ca/sloan
Thank you for your attention!