Disciplinary differences

and other biases Exploring social media metrics in scholarly context

stefanie.haustein@umontreal.ca

@stefhaustein Stefanie Haustein

Overview

• Altmetrics: definitions

• Bibliometrics: in retrospect

• Altmetrics: present

• correlations

• publication age biases

• disciplinary biases

• subject biases

• Altmetrics: future

• References

Altmetrics: definitions

• term coined by Jason Priem

• introduced as a better filter

than and alternative to

citations and peer-review

http://altmetrics.org/manifesto/

• “…altmetrics is a good idea,

but a bad name”

“…we would like to propose

the term influmetrics”

Rousseau & Ye (2013)

• rather complementary than

alternative to citations

• social media metrics

Altmetrics: definitions

• ultimate goals

• similar to but more timely than citations

predicting scientific impact

• different, broader impact than captured by citations

measuring societal impact

• impact of various outputs

“value all research products”

Piwowar (2013)

Altmetrics: definitions

• Altmetrics are “representing very different things” (Lin & Fenner, 2013)

• unclear what exactly they measure:

• scientific impact

• social impact

• “buzz”

• all of the above?

Altmetrics: definitions

ad-hoc

classifications

need to be

supported

by research

Altmetrics: definitions

scientist on

Twitter tweeting

scientific paper

in non-scholarly

manner:

• scientific impact?

• social impact?

• buzz?

Altmetrics: definitions

• complex to define and classify tools and motivations

• scientific and non-scientific audiences cannot be

determined on the platform used

• level of engagement differs not only between

platforms but also within:

saving paper to Mendeley library vs. tweeting about it

saving vs. reading

retweeting link vs. discussing content

differentiation between audiences and engagement

needed to determine meaning of metrics

Bibliometrics: in retrospect

• when Garfield created SCI, sociologists of science

analyzed meaning of publications and citations

(Merton, Zuckerman, Cole & Cole, etc.)

• sociological research • What is it to publish a paper?

• What are the reasons to cite?

• empirical bibliometric research • disciplinary differences in publication

and citation behavior

• delay and obsolescence patterns

Bibliometrics: in retrospect

• empirical studies helped sociologists to understand

structure and norms of science

• for bibliometricians, studies provided a theoretical

framework and legitimation to use citation analysis

in research evaluation

• knowledge about disciplinary differences and

obsolescence patterns helped to normalize statistics

and create more appropriate indicators

Bibliometrics: in retrospect

• similar to development of SCI in the 1960s, social

media metrics have to be analyzed:

• qualitative studies to analyze who, how and why

people use various social media platforms

• large-scale quantitative studies to determine

differences and biases in terms of disciplines, topics,

document types, publications years, publication types

and sources, author age and affiliation, etc.

to find out what various social media metrics mean

and what they can be used for

Altmetrics: correlations

e.g., Mendeley

• 793 Nature papers: ρ=0.559

820 Science papers: ρ=0.540

• 1,651 JASIST papers: ρ=0.458

• 5,596 PLoS ONE papers: ρ=0.3

• 1,136 bibliometrics papers: ρ=0.448

• 1,389 F1000 papers: ρ=0.686

• 62,647 social science papers: ρ=0.516

14,640 humanities papers: ρ=0.428

• random sample of

200,000 WoS papers: ρ=0.35

• 586,600 PubMed papers: ρ=0.386

Bar-Ilan (2012)

Priem, Piwowar, & Hemminger (2012)

Bar-Ilan et al. (2011)

Li, & Thelwall (2012)

Mohammadi & Thelwall (in press)

Zahedi, Costas, & Wouters (2013)

Haustein, et al.(submitted)

Li, Thelwall, & Giustini (2012)

Altmetrics: age biases

Current biases influencing correlation coefficients

Altmetrics: age biases

Current biases influencing correlation coefficients

Altmetrics: age biases

Current biases influencing correlation coefficients

Altmetrics: age biases

Current biases influencing correlation coefficients

compare documents of similar age

normalize for age differences

Altmetrics: disciplinary biases PubMed papers covered by Web of Science 2010-2012

Altmetrics: disciplinary biases PubMed papers covered by Web of Science 2010-2012

Altmetrics: disciplinary biases x-axis:

coverage of

specialty on

platform

y-axis:

correlation

between social

media counts

and citations

bubble size:

intensity of use

based on mean

social media

count rate

Altmetrics: subject bias

Scatterplot of number of citations and number of tweets (A, ρ=0.181**) and Mendeley readers (B, ρ=0.677**),

bubble size represents number of Mendeley readers (A) and tweets (B). The respective three most tweeted (A)

and read (B) papers are labeled showing the first author.

General Biomedical Research papers 2011

Altmetrics: subject bias

Article Journal C T

Hess et al. (2011). Gain of chromosome band 7q11 in papillary thyroid carcinomas of young patients

is associated with exposure to low-dose irradiation PNAS 9 963

Yasunari et al. (2011). Cesium-137 deposition and contamination of Japanese soils due to the

Fukushima nuclear accident PNAS 30 639

Sparrow et al. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at

Our Fingertips Science 11 558

Onuma et al. (2011). Rebirth of a Dead Belousov–Zhabotinsky Oscillator Journal of Physical

Chemistry A -- 549

Silverberg (2012). Whey protein precipitating moderate to severe acne flares in 5 teenaged athletes Cutis -- 477

Wen et al. (2011). Minimum amount of physical activity for reduced mortality and extended life

expectancy: a prospective cohort study Lancet 51 419

Kramer (2011). Penile Fracture Seems More Likely During Sex Under Stressful Situations Journal of Sexual

Medicine -- 392

Newman & Feldman (2011). Copyright and Open Access at the Bedside New England

Journal of Medicine 3 332

Reaves et al. (2012). Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells Science 5 323

Bravo et al. (2011). Ingestion of Lactobacillus strain regulates emotional behavior and central GABA

receptor expression in a mouse via the vagus nerve PNAS 31 297

Top 10 tweeted documents: catastrophe & topical / web & social media / curious story

scientific discovery / health implication / scholarly community

Altmetrics: future

• before applying social media counts in information

retrieval and research evaluation, we need:

to understand and define meaning of various

social media metrics

to identify different biases

to differentiate between audiences and

level of engagement

more transparency and reliability in data aggregation

References Bar-Ilan, J. (2011). Articles tagged by 'bibliometrics' on Mendeley and CiteULike. Paper presented at the Metrics 2011 Symposium on

Informetric and Scientometric Research, New Orleans, Louisiana.

Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars' visibility on the social web.

In Proceedings of the 17th International Conference on Science and Technology Indicators, (pp. 98-109).

Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting biomedicine: an analysis of tweets and citations

in the biomedical literature. Journal of the American Society for Information Science and Technology.

Haustein, S., Bowman, T.D., Holmberg, K., Larivière, V., & Peters, I., (submitted). Astrophysicists on Twitter: An in-depth analysis of

tweeting and scientific publication behavior. Aslib Proceedings.

jasonpriem (2010, September 28). I like the term #articlelevelmetrics, but it fails to imply *diversity* of measures. Lately, I'm liking

#altmetrics. [Twitter post].

Li, X. & Thelwall, M. (2012). F1000, Mendeley and Traditional Bibliometric Indicators. In Proceedings of the 17th International Conference

on Science and Technology Indicators, (pp. 541-551).

Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2),

461-471.

Lin, J. & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information Standards

Quarterly, 25(2), 20-26.

Mohammadi, E., & Thelwall, M. (in press). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation

and knowledge flows. Journal of the American Society for Information Science and Technology.

Piwowar, H. (2013). Value all research products. Nature, 493(7431), 159.

Priem, J., Piwowar, H., & Hemminger, B.M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv.

Priem, J., Taraborelli, D., Groth, P. & Neylon, C. (2010). Alt-Metrics: A Manifesto.

Rousseau, R., & Ye, F.Y. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 58(26), 3288-3290.

Zahedi, Z., Costas, R., & Wouters, P. (2013). What is the impact of the publications read by the different Mendeley users? Could they help

to identify alternative types of impact? Presentation held at the PLoS ALM Workshop 2013 in San Francisco.

Stefanie Haustein

Thank you for your attention!

Questions?

stefanie.haustein@umontreal.ca @stefhaustein