Grey Literature & Social Media as Open Review

social media

grey literature

social media- applications that enable users to

create and share content or to

participate in social networking

grey literature- report or

manuscript circulated or published by

unconventional routes

how big are these two entities and how are they

used?

reddit: “the front page of the internet” 2,200,000 Unique Pageviews per month 4,900,000 Total Pageviews per month ~100,000-125,000 comments per month.

In r/science alone }

Twitter: “yours to discover” 320 million monthly active users worldwide 500 million tweets (140 character-long messages) per day

Van Noorden, Nature 2014

Facebook: “Connect with friends and the world around you.” 1.65 billion monthly active users worldwide

Van Noorden, Nature 2014

Academia.edu 33M members 6M active users

ResearchGate 9M Members

Pubpeer

Journal clubs

Journal clubs

DOI: 10.1111/bcp.12992

Blogs

Blogs

“after 14 months of informal post-publication

discussion, the hypothesis was refuted.” -Yeo et. al, 2016

How good is classical pre-publication peer review?

Editors at the British Medical Journal found on average that only 2 out of 9 major artificial errors

were detected (Schroter et al., 2008)

At JAMA only 2 out of 8 artificial errors were noticed

(Godlee et al., 1998)

At the Annals of Emergency Medicine, 68% of reviewers did not realize the conclusions were not

supported by the evidence (Baxt et al., 1998)

Journals rejected previously accepted/published work 8 out of 9 times when names and institutions of authors were changed from prestigious universities (Harvard,

Stanford) to fake names like “Tri-valley Institute.” (Ceci & Peters, 1982)

footnote: one author was denied after performing this study

Agreement between reviewers as to whether papers should be accepted, revised or rejected is

no better than chance. (i.e it is random) (Rothwell & Martyn, 2000)

Sentiments on around classical peer review

Nicholson & Alperin, The Winnower 2016

Would making reviews open change the content?

Nicholson & Alperin, The Winnower 2016

0%10%20%30%40%50%60%70%80%90%100%

Yes,verydifferent

Yes,somew

hatdifferent

Perhapsdifferentin

tone,but

notin

content

No,notsignificantlydiff

erent

itwaspu

blishedforothersto

see

(%)o

frespo

nden

ts

Would making reviews open require a lot of work?

Nicholson & Alperin, The Winnower 2016

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Agreatdeal

Alot

Amoderatea

mount

AliIle

Noneatall

(%)o

frespo

nden

ts

“I'm an editor at two medical journals: The BMJ and BMJ Open, both of which use open peer review. The BMJ conducted two randomised controlled trials showing that a) signed review and b) open peer review, with prepublication histories published next to each paper did not lower the quality or depth of content in peer reviewers' reports. The tone was, however, slightly more constructive”

“I think that if people have to sign their name to a document, they take more care and effort to review carefully and thoughtfully.”

“If it had been published the remarks that are exceptionally rude and/or stupid would probably have been avoided by the reviewers.”

Reasons that would incentivize scholars to make their peer reviews publicly available.

Nicholson & Alperin, The Winnower 2016

Myperformancereview/tenure

commiIeeexplicitlyrecognizepublished

reviews27%

Mypeerspublishedtheirreviews26%

Iwaspaidasmallhonorarium(cash)

7%

Iwaspaidin-kind(e.g.,freeaccesstojournal,

waivedarUcleprocessingfee,etc.)17%

ThejournaleditorgavemeposiUvefeedbackon

myreview16%

Nothingcouldmakemedoit3%

Somethingelse4%

RECOMMEND: MAJOR REVISIONS

“Scientific publishing + Reddit = The Winnower

and hopefully a more open world of science!!”

-Alexis Ohanian Founder of Reddit

the trajectory

our authors come from all around the world

rapid growth

but still lots to be done…

how do we incentivize people to participate

in open post-pub review?

can we automate identification,

validation, invitation of reviewers?

what else can we do to make peer review

more robust?

thanks!

josh@thewinnower.com

@thewinnower

thewinnower.com