Reproducible Research Workshop

SICSA PhD Conference

June 27, 2016

Ian Gent

University of St Andrews

http://ian.gent

http://www.slideshare.net/turingfan/sicsaphd2016

Image: © Columbia Pictures

Joint Work with Graham Mcdonald

• http://www.gla.ac.uk/schools/computing/researchstudents/grahammcdonald/

• Graham has put far more work into this session than me!

Two things from the title slide I want to highlight

Yes this really is my URL

SICSA PhD Conference

June 27, 2016

Ian Gent

University of St Andrews

http://ian.gent

http://www.slideshare.net/turingfan/sicsa-phd2016

Image: © Columbia Pictures

And it’s all about doing the same thing over

and over again

SICSA PhD Conference

June 27, 2016

Ian Gent

University of St Andrews

http://ian.gent

http://www.slideshare.net/turingfan/sicsa-phd2016

Image: © Columbia Pictures

Outline of Talk

• Part I: Reproducibility in (Computer) Science

• Part II: What we are doing for the rest of the session

Reproducibility in Science

• Officially, reproducibility is key to science

• If you reproduce my experiment it’s win-win-win• You win because you have done a good thing for Science• I win because my experiment is validated• Science wins because it knows that my conclusions are valid

• Computer Science is at a huge advantage• Many of our experiments are software runs• With no human intervention or automatable intervention• We can then rerun them very easily anytime we wish

• Even if our experiments need major human intervention• We can automate the parts that don’t• Plus data analysis etc

7

Reproducibility in Science

• Officially, reproducibility is key to science

• If you reproduce my experiment it’s win-win-win• You win because you have done a good thing for Science

• I win because my experiment is validated

• Science wins because it knows that my conclusions are valid

• Computer Science is at a huge advantage

• Unofficially?

8

#overlyhonestmethods

https://twitter.com/ianholmes/status/288689712636493824

#overlyhonestmethods (still going strong!)

https://twitter.com/DRG_physics/status/745927096850087936

xkcd/PhD/Dilbert Compliance

https://www.phdcomics.com/comics/archive.php?comicid=1569

Galileo’s Telescopes

• Imagine if we could look through Galileo’s telescopes

• And we hadn’t bothered to keep them • Or threw away the only postdoc …

• This has happened in computer science• Many many times

Galileo’s Telescopes

• SHRDLU is a famous early AI program

• We have the source code

• But we can’t run it!

Image AI Lab MIT

Interlude: Can you do experiments in CS?

Real Experiments that helped me get my first AI Journal paper

Yes it was 22 years ago!

What has this got to do with reproducibility?

How I almost didn’t get an AI Journal paper

• I ran experiments overnight

• Since “all” instances were easy they should just take a few hours

• When I came in the next morning the output was stuck

• No file writes for several hours

• I assumed that some file output had got lost somehow

• Killed the job and deleted all the relevant files• And the files had random seeds in them • If I had just not deleted them I could have reproduced the experiments

• If this effect was a one in a million event I might never have seen it again

• I am extremely lucky that it occurred once or twice in a thousand events

This is a fun story but with a bigger message

• Reproducibility is not about the worthy principle of science

• Reproducibility is about being able to reproduce your own work• Or else you can lose journal papers!

• And make the next paper much harder to write

• And it’s about building on the great work of others• The easier it is to reproduce somebody else’s work

• The easier it is to improve on their work and contribute more to science

• “It's not really for the benefit of other people. Experience shows the principal beneficiary of reproducible research is you the author yourself.”

Jon Claerbout

Part II: The rest of this session

• 11.55 Group Discussions

• 12.30 Report back session

• 12.55 Interactive Polling

• 13.05 End (but something will slip so it will be 13.10)

Groups and papers

• Graham has pre-allocated you to groups• Mailed you about your allocation on Friday

• Graham allocated a paper to each group• Papers are (very broadly) related to your research field

• You should now be sitting at the table allocated to your group • there are printed copies of the papers at your table.

• If you are unsure of your group raise your hand NOW • Somebody from the committee will help you

The Papers

• Algorithms• Learning Expressive Linkage Rules using GeneticProgramming• Robert Isele, Christian Bizer

• Computer/Complex Systems• Experimental demonstration of associative memory with memristive neuralnetworks• YV Pershin, M Di Ventra

• Human Computer Interaction [2 groups]• I did that! Measuring users' experience of agency in their own actions• D Coyle, J Moore, PO Kristensson, P Fletcher, A Blackwell

• Machine Learning [2 groups]• A Model for Learning the Semantics of Pictures• V. Lavrenko, R. Manmatha, J. Jeon

• Networks • Modeling and performance analysis of Bit Torrent like peer to peer networks• D Qiu, R Srikant

Some Issues to Think about

• We have flagged up 5 issues for you to think about

• With a number of suggested questions in each one

• Some of these issues may not be relevant to your paper• So don’t feel you have to discuss each one to the same depth• Please use some of your time to discuss Issue 5: General Issues

• And you may have more issues • Which would be great

• And you may have more questions within each issue• Which would also be great• Some of these issues may not be relevant to your paper -

1. Accuracy of Reproduction

• How accurately would you expect to be able to reproduce the results of the paper?

• If you were to try reproduce this paper, what parts of the experiment would you expect there to be variations or differences between the original experiment and your reproduction?

2. Failure of Reproduction

• If your reproduction had more variations than you expected, what variants in experimental design or results would result in you concluding that the reproduction failed?

• What would make you decide that the experiments are flawed enough to invalidate the conclusions?

3. Difficulty and Costs

• What aspects of the reported experiment would you expect to be most challenging to reproduce?

• What specialist or specific resources do you need access to for reproducing the experiment?

• Are these resources publicly available?

• What costs are attached to doing the reproduction?

• Would this be a good use of this money/time?

4. Legal and Ethical

• Are there legal or ethical issues around reproducing this experiment?

• If so, what are they?

• If not, why aren’t there any?

5. General Issues

• What are the main difficulties and costs, or legal and ethical issues relating to reproducibility in your field of computing science?

• What practical steps are generally taken in your field of computing science to achieve reproducibility?

• Why should we be striving for reproducibility in computing science?

• How much should paper reviewers focus on reproducibility?

• What about PhD examiners?

I’ve talked too long, Go!

11.55 Group Discussions12.30 Report back session12.55 Interactive Polling13.05 End (but something will slip so it will be 13.10)