View
1.653
Download
0
Category
Preview:
DESCRIPTION
Citation preview
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Why most published research findings are falseArticle by John P. A. Ioannidis (2005)
Aurelien Madouasse
November 4, 2011
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The author: John P.A. Ioannidis
• C.F. Rehnborg Chair in Disease Prevention at StanfordUniversity (US)
• Professor of Medicine and Director of the StanfordPrevention Research Center (US)
• Chaired the Department of Hygiene and Epidemiology atthe University of Ioannina School of Medicine (Greece)
• Has a 51 page CV
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The author: John P.A. Ioannidis• C.F. Rehnborg Chair in Disease Prevention at Stanford
University (US)• Professor of Medicine and Director of the Stanford
Prevention Research Center (US)• Chaired the Department of Hygiene and Epidemiology at
the University of Ioannina School of Medicine (Greece)• Has a 51 page CV
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The journal: PLoS Medicine
• Public Library of Science• Peer reviewed• Open Access• Publication fee: US$2900
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The journal: PLoS Medicine• Public Library of Science• Peer reviewed• Open Access• Publication fee: US$2900
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The article (Checked 2011-10-22)
• Views: 410,087• Citations:
• CrossRef: 312• PubMed Central: 118• Scopus: 579• Web of Science: 585
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Context
• The article (Checked 2011-10-22)• Views: 410,087• Citations:
• CrossRef: 312• PubMed Central: 118• Scopus: 579• Web of Science: 585
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Introduction
• Published research findings sometimes refuted bysubsequent evidence
• Increasing concern false findings may be the majority
• This should no be surprising
• Here is why . . .
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis testing
• Consider a parameter measured in a population ofindividuals with a disease:
• Before treatment
• After treatment (Here assuming the treatment has an effect)
Some Parameter
Fre
quen
cy
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis testing
• Consider a parameter measured in a population ofindividuals with a disease:
• Before treatment• After treatment (Here assuming the treatment has an effect)
Some Parameter
Fre
quen
cy
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis
• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis
• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result
• If H0 were true, the probability of observing our datawould be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .
• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion
• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect
• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• We want to know whether the treatment has an effect
• We make a hypothesis• H0: The treatment has no effect
• We test our hypothesis
• We get a result• If H0 were true, the probability of observing our data
would be . . .• p(data|H0) = p − value
• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
• This framework assumes that we accept to be wrong . . .
sometimes
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• α = probability of declaring a relationship when there isnone - Type I error
• β = probability of finding no relationship when there isone - Type II error
• 1− β = probability of finding a relationship when there isone - Power
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Hypothesis Testing
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
• For a given hypothesis, whether we get it wrong dependson:
• Whether the hypothesis is true• The magnitude of the effect• The values we choose for α and β
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper
• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses
• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True
• Hypothesis testing can be seen as testing for a disease inEpidemiology
• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology
• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity
• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity
• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in
Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
PPV =p(1− β)
p(1− β) + (1− p)α
• Ioannidis uses R = p1−p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R
1+R × (1− β)R
1+R × (1− β) + 11+R × α
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Modelling the Framework for FalsePositive Findings
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R(1− β)
R(1− β) + α
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
• Among the studies that should have been reported asnegative
• A proportion u are reported as positive because of bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β αNo relationship β 1 − α
Total p 1 − p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
• Among the studies that should have been reported asnegative
• A proportion u are reported as positive because of bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)
Total p 1 − p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)
Total p 1 − p
• Positive predictive value
PPV =p(1− β + uβ)
p(1− β + uβ) + (1− p)(α + u(1− α))
• Ioannidis uses R = p1−p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
TruthTrue relationship No relationship
Trial
Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R(1− β) + uβR
R + α− βR + u − uα + uβR
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
Power = 0.8
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
Power = 0.5
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Bias
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
u = 0.05u = 0.2u = 0.5u = 0.8
Power = 0.2
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published
• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
• Increases the probability of a positive finding . . . by chance
• Positive findings more likely to be published• Association with publication bias?
• Positive findings more likely to receive attention
• Probability of at least one positive finding:
1 - probability of negative findings only
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Total p 1 − p
• Positive predictive value
PPV =p(1− βn)
p(1− βn) + (1− p)(1− (1− α)n)
• Ioannidis uses R = p1−p
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
TruthTrue relationship No relationship
Trial
Relationship 1 − βn 1 − (1 − α)n
No relationship βn (1 − α)n
Total p 1 − p
• Positive predictive value
• Ioannidis uses R = p1−p
PPV =R(1− βn)
R + 1− ((1− α)n + Rβn)
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
Power = 0.8
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
Power = 0.5
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Testing by Several IndependentTeams
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pre−study odds
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
Pre−study probability
Pos
t−st
udy
prob
abili
ty (
PP
V)
n = 1n = 5n = 10n = 50
Power = 0.2
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds
• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds
• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5
• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???
• Is it possible to design a study to estimate these???• Problem: Gold Standard
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???
• Problem: Gold Standard
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The use of odds instead of probabilities makes the articlehard to follow
• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?
• It would be great if the framework could be formallyassessed for various scientific fields!
• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• Link between magnitude of the effect, α, β and samplesize
• Trade off between α and β• Smaller effects require bigger samples
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• Link between magnitude of the effect, α, β and samplesize
• Trade off between α and β
• Smaller effects require bigger samples
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• Link between magnitude of the effect, α, β and samplesize
• Trade off between α and β• Smaller effects require bigger samples
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Some Parameter
Fre
quen
cy
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Comments on the framework
• The corollaries follow from the proposed model
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 1
The smaller the studies conducted in a scientific field, theless likely the research findings are to be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 2
The smaller the effect sizes in a scientific field, the lesslikely the research findings are to be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 3
The greater the number and the lesser the selection oftested relationships in a scientific field, the less likely theresearch findings are to be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 4
The greater the flexibility in designs, definitions, outcomesand analytical modes in a scientific field, the less likely theresearch findings are to be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 5
The greater the financial and other interests and prejudicesin a scientific field, the less likely the research findings areto be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Corollary 6
The hotter a scientific field (with more scientific teamsinvolved), the less likely the research findings are to be true
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
Plan
1 Context
2 Introduction
3 Modelling FrameworkHypothesis testingBiasMultiple testingComments
4 Corollaries
5 Conclusion
Why mostpublishedresearch
findings arefalse
AurelienMadouasse
Context
Introduction
ModellingFramework
Hypothesistesting
Bias
Multiple testing
Comments
Corollaries
Conclusion
How can we improve the situation?
• Cannot draw firm conclusions based on a single positiveresult
• It is possible to test for something until we find what wewant!
• And this is more likely to receive attention
• Selecting research questions• Avoid marketing driven questions• Importance of pre study odds
• Increase power• Larger samples
• For research questions with high pre-study odds• To test major concepts rather than narrow specific
questions
• Research standards
Recommended