No Evidence without Stories, No Stories Without Evidence Responsible use of metrics in research assessment

1

Cameron Neylon - 4 August 2015 - University of Sydney @cameronneylon - http://orcid.org/0000-0002-0068-716X

A QUESTION OF VALUES

2Nichole Burrows: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY

3https://www.flickr.com/photos/schnappi/6116266037 CC BY

4

MissionMissionValues

Mission

5

AssessmentAssessment

Mission

6

AssessmentAssessment

7http://www.flickr.com/photos/drachmann/327122302 CC-BY-SA

More data is the answer!

8https://www.flickr.com/photos/blprnt/3261815454 CC BY

9

DATA AND MEASUREMENT

Scott Akerman: http://www.flickr.com/photos/sterlic/4299631538 CC BY

10http://www.flickr.com/photos/verzo/8020565592/ CC BY

11http://www.flickr.com/photos/andrein/372192048/ CC BY-SA

More data is the answer?

12https://www.flickr.com/photos/blprnt/3261815454 CC BY

Types of Data•Citations

•Usage

•Bookmarks

•Conversation

Citations•Traditional and valued

•Less consistent than most think – WoS, Scopus, CrossRef,

(Europe)PMC •Track a very specific kind of usage •Only track a very specific kind of

Usage•Page views, downloads

•Measure usage by a wider community

•Measure greater range of use • Inconsistent between publishers •Rarely get demographics of viewers

Bookmarks•Academic and “public” facing sites

– People collecting literature – Building a library

•Measure greater range of use vs cites •Focus on use by researchers

(Mendeley)

Conversations/Social Media•Twitter, Facebook, Blogs, News sites

– Sharing literature within communities

– Conversations about literature

•Open to the public (but often actually driven by/involving researchers)

A story…

18

19altmetric.com

20

21

22

0 20 40 60 800

5000

10000

15000

PLOS papers authored by University of Cape Town staffUsage data from the PLoS journal website (combined HTML views and PDF downloads). Circle size correlates with CrossRefcitation counts. Red circles represent papers with "HIV" in title.

Months

Total Views

Not an obvious outlier...

23

...but interesting interactions...

24

25

…driven by this person...

26

...who I can talk to.

27

28

29

30See http://bit.ly/NJOyUn for the whole conversation

This. Is Impact.

31

32

DISSECTING IMPACT

Steve Evans: https://www.flickr.com/photos/babasteve/3140688308 CC BY

Impact

33

Impact

34

Research

Economic

Cultural

Education

Health

Environment

Impact

35

Research

Economic

Cultural

Education

Health

Environment

36

Research

Economic

Cultural

Education

Health

Environment

37

Research

Economic

Cultural

Education

Health

Environment

Research

Outputs

38

Research

Economic

Cultural

Education

Health

Environment

Research

Outputs

39

Research

Economic

Cultural

Education

Health

Environment

Research

Outputs

40

Research

Research

Outputs

41

Research

Research

Outputs

42

Research

43

Research

Citation

44

Research

Bookmark

Citation

45

Research

Bookmark

Citation

Previous co-authorship

46

Research

Bookmark

Citation

Previous co-authorship

Conference co-attendance

Social media conversation

47

48

49

50

51

52

53

54

55

56

57https://www.flickr.com/photos/see-through-the-eye-of-g/5392290809 CC BY

58https://www.flickr.com/photos/jenny-pics/3239638494 CC BY

59https://www.flickr.com/photos/garycycles2/2575676610 CC BY

60https://www.flickr.com/photos/53487196@N08/5439253136 CC BY

61https://www.flickr.com/photos/jenny-pics/3239638494 CC BY

62

“Stories that persuade with data” Anita de Waard

63

“Stories that persuade with data” Anita de Waard

64

evidence

TELLING A GOOD STORY

65Magenta Rose: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY

66

67m01229: https://www.flickr.com/photos/39908901@N06/8538482488 CC BY

68Martin Howard: https://www.flickr.com/photos/martinhoward/2709364519 CC BY

69

70

http://ivory.idyll.org/blog/tag/skepticism.html

12 13 14 15 16

0

5000

10000

15000

Usage vs. Citations for Harvard Authors 01−04/12220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlateswith number of CrossRef citations, and color with PLOS journal. Data collected April 11, 2013.

Age in Months

Total Views

71

12 13 14 15 16

0

5000

10000

15000

Usage vs. Mendeley for Harvard Authors 01−04/12220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlateswith number of Mendeley bookmarks, and color with PLOS journal. Data collected April 11, 2013.

Age in Months

Total Views

A Collaboratively-Derived Science-Policy Research Agenda

72

12 13 14 15 16

0

5000

10000

15000

Usage vs. Facebook for Harvard Authors 01−04/12220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlateswith number of Facebook activity, and color with PLOS journal. Data collected April 11, 2013.

Age in Months

Total Views

73

12 13 14 15 16

0

5000

10000

15000

Usage vs. Wikipedia for Harvard Authors 01−04/12220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlateswith number of Wikipedia pages, and color with PLOS journal. Data collected April 11, 2013.

Age in Months

Total Views

74

12 13 14 15 16

0

5000

10000

15000

Usage vs. Wikipedia for Harvard Authors 01−04/12220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlateswith number of Wikipedia pages, and color with PLOS journal. Data collected April 11, 2013.

Age in Months

Total Views

Afghanistan's Ethnic Groups Share a Y-Chromosomal Heritage Structured by Historical Events

75

76Philip Choi: https://www.flickr.com/photos/superturtle/121564208 CC BY

77Philip Choi: https://www.flickr.com/photos/superturtle/121564208 CC BY

There are four pillars to our research vision: •an unstinting commitment to research

excellence •a willingness to harness that excellence to

address some of the vital national and global challenges of our time

•a desire to engage with the communities for whom our research has real meaning and consequences and from whom we can learn – whether in the private or public sector, local or global

•new initiatives to develop, nurture and support our researchers; that is, to develop not only brilliant minds, but also research leaders.

“Qualities” not “Quality”

78

79

Academic quality is highly context - specific, and it is sensible to think in terms of research qualities, rather than striving for a single definition or measure of quality.

The central problem identified here is that academic quality is a complex notion that cannot easily be reduced to quantification – the use of proxy variables runs the risk of misrepresenting the qualities of research contributions and may lead to unintended consequences.

As PLOS noted in its submission to this review, “it is unclear whether any unique quality of research influence or impact is sufficiently general to be measured”.

Wilsdon et al. (2015) - The Metric Tide

80

MissionMissionValues

RESPONSIBILITY USE OF METRICS

81Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY

RESPONSIBILITY USE OF METRICS

82Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY

indicators

Develop the stories…

83https://www.flickr.com/photos/jblmmwr/8027451065 CC BY

…into a shared mission

84Daniel Spiess: https://www.flickr.com/photos/deegephotos/5984387881 CC BY-SA

Get informed…

85

Sign DORA86

…and mean it

87

Above all…

88

89Joe Flintham: https://www.flickr.com/photos/joeflintham/5631590491 CC BY-SA

It’s just good scholarship…

90

91

92https://www.flickr.com/photos/ell-r-brown/4655401891 CC BY

@cameronneylon cn@cameronneylon.net

http://orcid.org/0000-0002-0068-716X

93