1

Unreliable researchTrouble at the lab

Patrik Ahlberg Magnus Åhberg

Syeda Rabab NaqviThawatachart Chulapakorn

Group Members:

3

Outline:Introduction

Understanding Insignificance

Not Even Wrong

Blame the Ref

Harder to Clone than you wish

Making the Paymaster´s Care

Personal Reflections

4

Introduction

• Irreproducibility is on rampage• Self corrections are not working IRL

“There is no cost to getting things wrong, The cost is not getting them published.”

Brian Nosek, University of Virginia

5

Understanding Insignificance

• False Positives + False Negatives = Problem

• We are not using statistics

• Not possible with power of 0.8

6

Not even wrong

• Research is commonly not thought through or not well executed

• Examples:– The ”pentaquark” saga (experiment not properly

blinded)– Risks with computer model ”tuning”– Longevity associated genetic variations due to

inproper handling of research samples

7

Blame the ref

• Peer reviewers are not detecting errors– John Bohannon´s made-up paper was published in

157 out of 304 journals– Fiona Godlee yielded similar results when sending

an 8-error paper to the 200 reviewers of BMJ• Errors are explained by incompetence rather

than fraud• The replication mechanism for scientific self-

correction is not functioning well

8

Harder to Clone than you wish :

Replication is hard to meet our standard

To find errors in the publications , we need to replicate the work of others, but process is not according to our wish.

Replication

Lack of interest by Journals and Academic Researchers

9

Lack of Interest

Journals thirsty of novelty, show little interest in it.

Academic researchers mostly spend time on work which is

more likely to enhance their careers.

Replication is Hard

Original methods and data required

Unpublished Research

Clinical trials are very costly to rerun

Software may be different during the replication

Failure of replication due to tacit knowledge

Expermenter’s regress

10Finally, we would say that…

Making the paymasters careSituation Paymasters Researcher

Problem - Prefer pioneer- Deny failed result

- Paper machine- Bias discussion- Blame the

previous errorPurpose - Emphasize

replication as same as others

- Don’t serious the failed result (Hard)

- Develop technical and statistical skill

- Honest result- Enforce standard

(Journal)

Quality is more important than Quantity

11

Patriks work

• Unethical publishing cancels this problem– No reproducibility possible– Small feasibility tests

• The system has given up on the system– Seniors warning juniors– Reliability comes from people not publications

12

Own reflections

• The possibility to ”tune” computer models until the desired results appear relates to my own research

• It is easy to percieve patterns and get results that are expected and might not even exist

• Responsibility as a reviewer:– When do I accept a reviewer invitation?– Do I expect to find errors? What am I looking for as

a reviewer?

13

Own ReflectionsIf some one tries to repeat my experiment, he must have Access to original methods and data Awareness of simulation procedure Similar samples for study

14

Related to my (future) research

• Replication– Same or not? Why?

• Scrutiny– Procedure– Measurement

• Bias discussion• Raw data• Log book!!!

15