Upload
alex-holcombe
View
1.915
Download
0
Embed Size (px)
DESCRIPTION
The broadest problem in science: Our publishing system. Slides from a talk by Alex O. Holcombe
Citation preview
[email protected] of Psychology
@ceptional
The broadest problem in science:Our publishing system
http://www.slideshare.net/holcombea/
1
Opacity, Sluggishness, Redundancy, Redundancy, & Redundancy!
from Kravitz &
Baker (2011)
Inefficient
2
The File-Drawer Problem
unpublished results
files
•Little career incentive to publish a non-replication or a replication
•Very difficult to publish a non-replication or replication
•Most journals only publish papers that “make a novel contribution”
•Reviewers/editors tend to hold non-replicating manuscript to higher standard than original.
•Bem example
http://www.flickr.com/photos/nickperez/2569423078 t. magnum
Incomplete
3
The File-Drawer Problem
Tower of unpublished
resultsfiles
Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes ina scientific field, the less likely the research findings are to be true. Flexibility increases the potential for transforming what would be “negative” results into “positive” results.
Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.
http://www.flickr.com/photos/nickperez/2569423078 t. magnum
“In summary, while we agree with Ioannidis that most
research findings are false...”
Incomplete
4
Barriers to publishing replications and failed-replications
• No glory in publishing a replication
• Few journals publish replications
• usually uphill battle even with those that do
• The wrath of the original researcher
5
File-drawer fixes
• Journals that don’t reject replications for being uninteresting or unimportant
• Pre-registration of study designs and analysis methods
• Brief reporting of replications
✔•◦
✔
•◦ ◦
◦◦
6
• problems: incentives
http://psychfiledrawer.org/view_article_list.php
7
• problems: incentives
Pashler, Spellman,
Holcombe& Kang (2011)
DETAILS page: http://psychfiledrawer.org/replication.php?attempt=MTU%3D
8
• problems: incentives
http://psychfiledrawer.org/view_article_list.php
9
File-drawer fixes
• Journals that don’t reject replications for being uninteresting or unimportant
• Pre-registration of study designs and analysis methods
• Brief reporting of replications
✔•◦
✔
•◦ ◦
◦◦
10
ROIM- peer Review of Intro & Methods1. Authors plan a replication study
2. They submit an introduction and methods section
3. It is sent to reviewers, including the targeted author
4. The editor decides whether to accept/reject, based on:
1. Reviewer comments regarding the proposed protocol
2. Importance of the study, judged by argument in the introduction, number of citations of original, reviewer comments
5. The Intro, Method and analysis plan, and reviewer comments are posted on the journal website
6. When the results come in, the authors write a conventional results and discussion section and that together with the raw data are posted, yielding the complete publication
1. some sort of minimal peer review needed for that. What exactly?
✔••
More efficient, more complete
11
ROIM- peer Review of Intro & Methods
• Original author sort-of signed off on it, so can’t complain / hate the replication authors as much.
• Good way to start for a new PhD project, anyone planning to build on some already-published results
• Reduce the incentive to publish flashy, headline-grabbing but unreliable studies?
✔••
12
When a new paperappears, readers often spot logical flaws,experimental weaknesses, questionableassumptions or alternative interpretations.Yet individual criticisms may not beconsidered important enough to warrantpublication. Even major criticisms areunlikely to appear until months or yearslater, and are often overlooked in thehaystacks of the literature.
post-publication peer review
Incomplete
13
I did a spot check of fifty of those citing articles to see if any had noted problems with the paper: only one of them did so. The others repeated the authors’ conclusions
The article in question was published in the Proceedings of the National Academy of Sciences, and at the time of writing has had 270 citations.
http://deevybee.blogspot.com.au/2012/03/time-for-neuroimaging-to-clean-up-its.html
post-publication peer review
The people who DON’T notice the problems with a paper cite it, the people who DO, don’t. So people not in the area never find out how flawed a paper is.
14
post-publication peer review
Absence of it means we don’t have many indicators of quality of individual articles.
Reinforces reliance on poor measures like journal impact factor.
15
I have a dream.. our papers will be judged not by the impact factor of their journal, but by the quality of their content
16
I have a dream that one day,
our papers will be judged not by the impact factor of their journal, but by the quality of their content
Impact Factor announced (4.3)
from Peter Binfield’s talk
17
ROIM- peer Review of Intro & Methods
• Original author sort-of signed off on it, so can’t complain / hate the replication authors as much.
• Good way to start for a new PhD project, anyone planning to build on some already-published results
• Reduce the incentive to publish flashy, headline-grabbing but unreliable studies?
• How to incorporate post-publication commentary?
✔••
18
ROIM- peer Review of Intro & Methods
• What publisher to publish it?
19
Academic knowledge is boxed in by expensive journals.
Scientist meets publisher: the video
http://www.youtube.com/watch?v=GMIY_4t-DR0
20
•Academia marginalising itself
•More impact!
•We do our research to benefit anyone interested, not some exclusive club
•Many scholars, doctors, patients, engineers, policymakers (and esp. in poor countries/small universities) can’t get access
The academic community
is only hurting itself, and
its long term public
support, by keeping its
knowledge behind high
subscription walls -
Andrew Carr
The countries we work with can’t afford journals; they’re already paying an arm and a leg for textbooks -Sir John Daniel
Why Open Access?• Free• Full-text• Online access
21
Open Access “Hulk”
OA HULK WANTS TO KNOW WHO TO OCCUPY!
ELSEVIER!? ACS!? HARPERCOLLINS!?
YOU NAME IT, OA HULK WILL OCCUPY AND SMASH!
22
Experimental Brain Research $13,670Journal of Radioanalytic and Nuclear Chemistry $19,826
Journal of Mathematical Sciences $17,880Journal of Materials Science $16,699
Institutional Subscription Cost, 2012
Claudio Aspesi at http://poynder.blogspot.com.au/2012/03/open-access-brick-by-brick.html
$3983 USD per article for Elsevier $1350 USD per article for PLoS ONE
23
Claudio Aspesi at http://poynder.blogspot.com.au/2012/03/open-access-brick-by-brick.html
$3983 USD per article for Elsevier $1350 USD per article for PLoS ONE
24
•Not all closed-access publishers profiteer
•The real enemy is us!
Ryan Orr http://www.flickr.com/photos/ryan_orr/615470501
25
•Deposit your manuscripts in the university repository (http://ses.library.usyd.edu.au/
•Even with closed journals, you often have the right to deposit your final version (e.g. Word document before typeset by publisher)
•Best for university and its funders if research outputs open access; indeed it’s mandated by:
•NIH, Wellcome Trust
•Princeton, Harvard
•Queensland University of Technology
•Support open-access publishing models
•PLoS, BioMed Central, eLife
•I don’t recommend paying for open-access “choice” in closed corporate-published journals
•Support innovations that address problems of inefficiency and incompleteness
Let me know if you want to help open Sydney Uni
How to help
26
Comprehensive solution? Open Science
•As data comes in, uploaded automatically to web
•Electronic lab notebook
•Papers written via open collaborative documents on the web
27
Wikipedia(vs.(Academic(Papers(12/21
Mat Todd
28
Mat Todd
This stimulated 20 comments (from 11 different people) and four private e-mails (via the website).
Industry suffers less from such metrics, but it is nevertheless surprising that industry were so heavily involved in this project. For example, of the roughly 100 comments since January 2010 on The Synaptic Leap website, around 60 came from readers not involved in the kernel project at Sydney, and of those approximately 42 came from industry, 16 from academia. Besides the input described above to the resolution experiments, a different company contributed samples of PZQ enantiomers isolated by chromatography for analytical purposes, and another company is currently determining the phase diagram of PZQamine. Why would companies choose to be involved, particularly in a project in neglected tropical diseases where there is little profit margin and no new intellectual property available? One can appeal to human nature — we see a problem we can help solve, and we find it impossible to resist stepping in, p
29