Embed Size (px)
Citation preview
Paul Groth Elsevier Labs@pgroth | pgroth.com
“Don’t Publish, Release” Revisited
July 19 – 24 , 2015, Dagstuhl Perspectives Workshop 15302Digital Scholarship and Open Science in Psychology and the Behavioral Scienceshttp://www.dagstuhl.de/15302
“All studies would be adequately powered; exploratory and confirmatory research would be clearly distinguished, and the latter would be subject to pre-registration; study materials and raw data would be archived for examination and use by others; and replications would be highly valued and common- place. In reality, time and money are limited.”
“Unfortunately, there has been a recent “crisis in confidence” among psychologists about the quality of psychological research (Pashler & Wagenmakers, 2012).”
• All empirical papers must archive their data upon acceptance in order to be published unless the authors provide a compelling reason why they cannot (e.g., expense, confidentiality). The action editor will be the final arbiter of whether the reason is sufficiently compelling.
• “Data” refers to an electronic file containing nonidentified responses that are potentially already coded. Normally, the data would
represent an early stage of electronic processing, before individual responses have been aggregated. The data must be in a form that allows all reported statistical analyses to be reproduced while retaining the confidentiality of individual participants. This entails that the data are formatted and documented in a way that makes the structure of the data set readily apparent.
• Archiving consists either of submitting the data to the journal (to be displayed as supplementary material at the end of the article), sending it to some other archive that is accessible to established researchers and maintained by a substantial established institution, or authors making the data available on their own website, assuming that they can assure us the site will be maintained by a recognized institution for a reasonable period of time. Again, action editors will be the final arbiters of the appropriateness of an archive.
• Any publication that reports analyses of or refers to archived data will be expected to cite the original publication in which the data were reported.
• This policy is new and therefore open to modification. Our aim is to implement a policy that maximizes transparency while minimizing the burden on authors.
http://saravanerp.com
https://www.force11.org/meetings/beyond-pdf-2
Don’t Publish. Release!
Professor Carole Goble FREng FBCSUniversity of Manchester, UK
Research
Components: data, codes, interpretation
ForksChanges
Versions
Builds on prior work
Don't publish. Release!
Treat ALL Components and ALL Research Like Software
So where are we at?
Registered Reports
• “conducting the peer review prior to data collection and analysis”
http://ivory.idyll.org/blog/replication-i.html
PDF DOWNLOADS ONLY?
Is anything still hard?
20Yolanda GilUSC Information Sciences Institute
Measuring Time Savings with “Reproducibility Maps” [Garijo et al PLOS CB12]
2 months of effort in reproducing published method (in PLoS’10)
Authors expertise was requiredComparison of ligand binding sites
Comparison of dissimilar protein structures
Graph network generation
Molecular Docking
Work with D. Garijo of UPM and P. Bourne of UCSD
21Yolanda GilUSC Information Sciences Institute
“The Reproducibility Project: Cancer Biology has been funded through a $1.3mm grant from the Laura and John Arnold Foundation,”
http://validation.scienceexchange.com/#/cancer-biology
Cost of documentation
http://www.indoition.com/en/services/costs-prices-software-documentation.htm
Two norms sum it up
• Be transparent* • Embrace the iteration inherent in science
* If we encourage transparency, well then we should be constructive
Constructive criticism
“I like to think of the Braintrust as Pixar’s version of peer review, a forum that ensures we raise our game – not by being perspective but by offering candor and deep analysis” -- Ed Catmull p. 93
