5/14/2018

1

Trish Groves, MBBS, MRCPsych

• Director of Academic Outreach, BMJ

Richard Sever, PhD

• Assistant Director, Cold Spring Harbor Press and Co-Founder, bioRxiv

Heather Staines, MA, PhD

• Director, Partnerships, Hypothes.is

Moderator: Angela Cochran

• Associate Publisher, ASCE

Concurrent 3.1: Article Commenting—Future of Scholarly

Publishing or Fad Fizzling Swiftly? #CSE2018

History of Article Commenting

• The scholarly publishing process includes discussion of published results– Letters to the Editor, Discussions/Closures, etc.

– PROS: They are peer reviewed and should add something to the literature. They can be cited.

– CONS: “Open” discussion periods are short and submission process can be cumbersome

• Digital publishing opened door to online comments.– Many journals tried…and failed.

– PROS: Faster and available to many more people. Discussion period open longer/forever.

– CONS: Engagement is low and journals needed to moderate (labor intensive).

5/14/2018

2

Early Analysis

Adie, E. 2008. Commenting on Scientific Articles (PLOS Edition).

http://blogs.nature.com/nascent/2009/02/commenting_on_scientific_artic.html

Where Comments are Collected

• Organized– Journal platforms with the article (BMJ, PLoS One,

F1000 Research, etc.)

– Third-party platforms (ScienceOpen, PubPeer, PubMed Commons)

– Preprint platforms with the article (bioRxiv)

– Overlay “journals”

• Not organized– Blogs

– Twitter

– Discussion boards

Problems with Commenting?

News: Popular Science Turns Off

Commenting

“The editors argued that Internet

comments, particularly

anonymous ones, undermine the

integrity of science and lead to a

culture of aggression and

mockery that hinders substantive

discourse.” (http://www.newyorker.com/tech/elements/the-

psychology-of-online-comments)

5/14/2018

3

Why Do We Keep Trying to Make

Commenting Work?• Problems

– Anonymous Commenting: easier for personal ax grinding

– Loose verification of commenter

– COIs not typically disclosed

• Promise– Provide individuals with a forum to discuss scholarly content

– Encourage rich additions to the discussion

– Collaboratively take research to the next level

– Provide a window into what the authors were thinking

– Engage readers (sticky pages)

– Ensure that this forum is safe…for authors and users!

– Increase engagement and promote membership

Today’s Agenda

• What works at BMJ?

• Engagement at the Preprint level?

• Should commenting be replaced by

annotations?

• Is there a future for commenting?

• Is commenting the future?

What works at BMJ?

Dr Trish Groves tgroves@bmj.com twitter/@trished

Director of academic outreach, BMJ

Editor-in-chief BMJ Open, Honorary deputy editor The BMJ

5/14/2018

4

I’m editor in chief of BMJ Open and deputy editor of BMJ (The BMJ

Publishing Company Ltd), a wholly owned subsidiary of the BMA (British

Medical Association).

BMJ (the company) receives 8.7% of revenues from drug & device

companies through advertising, reprint sales, & sponsorship. For The BMJ

it’s 12%. The BMJ and BMJ Open are open access journals that charge

article publishing fees for research.

I’m director of academic outreach and advocacy for BMJ; this includes work

for the subscription-based BMJ Research to Publication eLearning

programme http://rtop.bmj.com/

Annual bonus scheme is based on performance of both BMJ and The BMJ.

Competing interests

“Medical journals should provide readers with a mechanism for submitting

comments, questions, or criticisms about published articles…authors of

articles discussed have a responsibility to respond to substantial criticisms

using same mechanisms and should be asked by editors to respond” (ICMJE)

Concerns may be raised by editors or readers through:

• letters to the editor

• complaints to the editor, publisher, or via COPE

• media or social media

• other forums eg PubPeer and, up to April 2018, PubMed Commons

Post publication peer review

http://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/correspondence.html

BMJ Rapid Responses

~110K Rapid Responses at The BMJ

since 1998 (and >700 at BMJ Open

since 2011)

Some Rapid Responses at The BMJ are

selected as readers' letters (indexed in

PubMed), but all count as published

online articles, citable by URL

5/14/2018

5

“We are grateful to Doidge and Dearden for their comprehensive

articulation of why our paper might appear invalid. We will respond to

the three critiques in order of their proposal, of which the first might

be the most important. Please see the figure here:

https://twitter.com/TaaviTillmann/status/978305036466966529

This figure illustrates three competing interpretations of our data and

the wider literature….”

The BMJ has resourced this well:

• premoderation by experienced editors; clear terms & conditions

• substantial contribution to the debate with clear English <1000 words

• from anyone as long as we have name, occupation, affiliation, functioning

email address, twitter handle, competing interests

• not libellous, obscene, offensive, breaching IP rights

• with signed consent from patient or other potentially identifiable individual

And?

• general medical journal with broad international reach

• virtuous circle – success encourages continued engagement

Why have Rapid Responses worked?

bioRxiv - a hub

Journals

Comments

Blogs

Confirmatoryresults

Contradictoryresults

Discussion

Re

pro

du

cib

ility C

ertific

atio

n

Discovery

Crossref Google Meta

New tools

Twitter

Preprintdiscussion

sites

Annotation

5/14/2018

6

5/14/2018

7

Feedback & Discussion

• Email(private)

• Social media(free-for-all)

• Comments(moderated)

• 3rd-party sites TBD

(external)

(1OOK/year)

(1O%, total 4200 )

Comments

Maiato: “I am convinced that sharing our points of convergence and divergence with the community is the correct way to move

forward and widen the discussion”

Liu (2018) bioRxiv 10.1101/263392

What’s next

5/14/2018

8

Does stepwise review change things?

“…the addition of badges to papers which signal that additional services

have been performed by PLOS and potentially other organizations…”

The Annotation

Ecosystem:

Alive and Thriving

Heather Staines

Director of Partnerships, Hypothesis

CSE, New Orleans

May 7, 2018

5/14/2018

9

Case Study: PubMed

● As of today: 60,653 annotations on

PubMed (8X more than

PubMedCommons even subtracting

the 7K archived PMC comments)

● But we aren’t natively embedded

on the site.

● And we’ve been live over a shorter

period of time○ PubMedCommons launched 2013

○ Hypothesis since 2015

Why is Hypothesis succeeding at PMC?

● It works everywhere, not just where it’s embedded.

● You can annotate the actual article, not just the abstract page.

● With DOI equivalence, users can also annotate the publisher article of

record

● Annotations are yours. You can search across your annotations, wherever

they are. Use tags to organize documents and notes.

● You can annotate publicly, in groups, or personally for yourself. Annotation

does not have to be public to be useful.

● All content types are accessible: html pages, PDFs, EPUBs, and data.

● Based on W3C standards, with a developing interoperable ecosystem, so

it can thrive beyond just a single vendor.

● Based on open source software, with APIs, that developers can build on

and extend for different purposes.

Publishers can feature their own content/use cases

5/14/2018

10

3.1 Million Annotations

Approx. 20% of annotations are public.

50% take place in groups.

By over 145k users, on pace to triple in the next 12 months.

Approx ~200k avg monthly annotations

Publishers and Platforms are adopting at Scale

● American Diabetes Association

● Cambridge University Press and Syracuse University-QDR

● Johns Hopkins University Press: Modernism / modernity

● University of Michigan Publishing

● Pensoft’s ARPHA Platform

● Elsevier

● Center for Open Science

● eLife

● PubFactory, HighWire, Silverchair, Ingenta, and more

● Open Therapeutics

● AGU and eJournalPress (Peer Review)

● MIT Press Cognet

● eJournalPress

Trish Groves, BMJ

@trished

Richard Sever, bioRxiv

@cshperspectives

Heather Staines, Hypothes.is

@heatherstaines

Angela Cochran, ASCE

@acochran12733