Embed Size (px)
Citation preview
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 1/35
How to make Good andValid Conclusion of
Medical Research
Aria KekalihCommunity Medicine Department
Epidemiology and Biostatistic Unit
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 2/35
Acknowledgement
L.G. Portney and M.P.Watkins
Bikash Bhatttacharya
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 3/35
Discussion and Conclusion
How does the author interpret result
Did the author clarify if hypothesis were
rejected or accepted
What alternative explanations does the
author consider for the obtained findings
How are the findings related to prior reportWhat limitations are described? Are there
limitations that are not addressed?
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 4/35
Discussion and Conclusion
If the result are not significant, does theauthor consider the possibility of type II
error?
Regardless of the statistical outcome, arethe result clinically important?
Does the author discuss how the result
apply to practice
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 5/35
Discussion and Conclusion
Does the author present suggestions for
further study
Do the stated conclusion flow logically
from the obtained result?
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 6/35
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 7/35
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 8/35
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 9/35
Conclusion ValidityConclusion validity is the degree to which conclusions we
reach about relationships in our data are reasonable
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 10/35
Conclusion Validity
Conclusion validity is the degree to which the conclusion we reach is credible or believable
OR
Statistically labeled - Often Misunderstood, Least Considered
Relevance in Qualitative Research as well as Quantitative Research
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 11/35
Conclusion Validity
Quantitative Example
An inventor has developed a new, energy-efficient lawn mower engine. The inventor claims that the engine will run
continuously for 5 hours (300 minutes) on a single gallon of regular gasoline
Qualitative Example
Accountability as practices in our primary health care system creates an undesirable atmosphere of anxiety among
nurses
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 12/35
Validity- How they differ?
Program Cause-Effect Relation
What you do What you See
How Credible?
Accept / Reject
Internal Validity Conclusion Validity
External Validity Construct Validity
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 13/35
Threats to Conclusion Validity
The primary threat to conclusion validity is the possibility of making an error in the inference
process concerning the relationship between the program and the outcome(s) of the program
Error Types
Type I error: Conclude there is a relationship when in fact there is not (concluding/seeing an effect
that in reality is not there)
Type II error: Conclude there is no relationship when in fact there is a relationship (“miss” a true
effect)
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 14/35
Fishing & the error rate problem
Looking for a specific result by analyzing the data repeatedly under slightly differing conditions
or assumptions
Conducting multiple analyses and treating each one as though it was independent without error
rate adjustment
Likely to see a relationship when there isn't one when you keep reanalyzing your data and
don't take that fishing into account when drawing your conclusions
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 15/35
Signal-to-noise ratio problem
Is the ratio of
Signal
Noise
"noise" consists of factors that make it hard to see the relationship
„low reliability of measures
„poor reliability of treatment implementation
„random irrelevancies in the setting
„random heterogeneity of respondents
“signal" amount of information collected and the amount of
risk taken for decision
•low statistical power
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 16/35
Problems that can lead to either
conclusion error Assumptions behind analysis when violated - likely to draw erroneous conclusions about
relationships
Quantitative Research Example
Assumption that data is normally distributed is violated
Quantitative Research Example
Assumption that the respondent is free to say anything ‟ but under pressure from supervisors
respond in a particular to
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 17/35
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 18/35
Improving Conclusion Validity
Sample size : Collect more information -- use a larger sample sizeEffect size : Improve the impact of the program relative to the noise
Alpha Level : Increase your risk of making a Type I error
Power : Ability to see effect that’s there
Consistency and Repeatability for measures, reducing situational distractions in the measurement context
Training program operators and standardizing the protocols for administering the program
C
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 19/35
The Four Components to a
Statistical Conclusion
Amount of information
Impact of program
Willingness to risk being wrong in
finding an effect (rejecting the null
hypothesis)
Ability to see effect that’s there
Sample size
Effect size
Alpha level
Power
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 20/35
Given Values for Any Three,
Possible to Compute the Fourth
n = f(effect size, α, power)
effect size = f(n, α, power)
α = f(n, effect size, power)
power = f(n, effect size, α)
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 21/35
Statistical Conclusions
Statistical conclusions involve constructing two mutually
exclusive hypotheses, termed the null (labeled H0) and
alternative (labeled H1)
H0: Program Effect = 0
H1: Program Effect <> 0
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 22/35
The Decision MatrixIn reality
Whatwe conclude
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 23/35
The Decision MatrixIn reality
Whatwe conclude
H0 (null hypothesis) true
Alternative H1 false
In reality...• There is no real program effect• There is no difference, gain• Our theory is wrong
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 24/35
The Decision MatrixIn reality
Whatwe conclude
H0 (null hypothesis) true
Alternative H1 false
In reality...
Accept null
Reject alternative
We say...
• There is no realprogram effect
• There is no difference,gain
• Our theory is wrong
• There is no real program effect• There is no difference, gain• Our theory is wrong
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 25/35
The Decision MatrixIn reality
Whatwe conclude
H0 (null hypothesis) true Alternative H1 false
In reality...
Reject null
Accept alternative
We say...
• There is a real programeffect
• There is a difference,gain
• Our theory is correct
• There is no real program effect• There is no difference, gain• Our theory is wrong
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 26/35
The Decision MatrixIn reality
Whatwe conclude
H0 (null hypothesis) true
Alternative H1 false
In reality...
Reject null
Accept alternative
We say...
• There is a real programeffect
• There is a difference,gain
• Our theory is correct
• There is no real program effect• There is no difference, gain• Our theory is wrong
TYPE I ERROR
The odds of saying there isan effect or gain when in
fact there is none
# of times out of 100 when
there is no effect, we’ll say
there is one
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 27/35
The Decision MatrixIn reality
Whatwe conclude
H0 (null hypothesis) false
Alternative H1 true
In reality...• There is a real program effect• There is a difference, gain• Our theory is correct
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 28/35
The Decision MatrixIn reality
Whatwe conclude
Alternative H1 true
In reality...
Accept null
Reject alternative
We say...
• There is no realprogram effect
• There is no difference,gain
• Our theory is wrong
• There is a real program effect• There is a difference, gain• Our theory is correct
H0 (null hypothesis) false
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 29/35
The Decision MatrixIn reality
Whatwe conclude
Alternative H1 true
In reality...
Accept null
Reject alternative
We say...
• There is no realprogram effect
• There is no difference,gain
• Our theory is wrong
• There is a real program effect• There is a difference, gain• Our theory is correct
TYPE II ERROR
The odds of saying there isno effect or gain when in
fact there is one
# of times out of 100 when
there is an effect, we’ll say
there is none
H0 (null hypothesis) false
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 30/35
The Decision MatrixIn reality
Whatwe conclude
Alternative H1 true
In reality...
Reject null
Accept alternative
We say...
• There is a real programeffect
• There is a difference,gain
• Our theory is correct
• There is a real program effect• There is a difference, gain• Our theory is correct
H0 (null hypothesis) false
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 31/35
The Decision MatrixIn reality
Whatwe conclude
Alternative H1 true
In reality...
Reject null
Accept alternative
We say...
• There is a real programeffect
• There is a difference,gain
• Our theory is correct
• There is a real program effect• There is a difference, gain• Our theory is correct
1-
POWER
The odds of saying there isan effect or gain when in
fact there is one
# of times out of 100 when
there is an effect, we’ll say
there is one
H0 (null hypothesis) false
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 32/35
The Decision Matrix
x
In reality
Whatwe conclude
Alternative H1 false Alternative H1 true
In reality... In reality...
Accept null
Reject alternative
Reject null
Accept alternative
We say...
• There is no realprogram effect
• There is no difference,gain
• Our theory is wrong
We say...
• There is a real programeffect
• There is a difference,gain
• Our theory is correct
• There is no real program effect• There is no difference, gain• Our theory is wrong
• There is a real program effect• There is a difference, gain• Our theory is correct
1-
THE CONFIDENCE LEVEL TYPE II ERROR
The odds of saying there isno effect or gain when in
fact there is none
# of times out of 100 when
there is no effect, we’ll say
there is none
The odds of saying there isno effect or gain when in
fact there is one
# of times out of 100 when
there is an effect, we’ll say
there is none
1-
TYPE I ERROR POWER
The odds of saying there isan effect or gain when in
fact there is none
The odds of saying there isan effect or gain when in
fact there is one
# of times out of 100 when
there is no effect, we’ll say
there is one
# of times out of 100 when
there is an effect, we’ll say
there is one
H0 (null hypothesis) true H0 (null hypothesis) false
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 33/35
The Decision MatrixIn reality
Whatwe conclude
Alternative H1 false Alternative H1 true
In reality... In reality...
Accept null
Reject alternative
Reject null
Accept alternative
We say...
• There is no realprogram effect
• There is no difference,gain
• Our theory is wrong
We say...
• There is a real programeffect
• There is a difference,gain
• Our theory is correct
• There is no real program effect• There is no difference, gain• Our theory is wrong
• There is a real program effect• There is a difference, gain• Our theory is correct
1-
THE CONFIDENCE LEVEL TYPE II ERROR
1-
TYPE I ERROR POWER
H0 (null hypothesis) true H0 (null hypothesis) false
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 34/35
The Decision MatrixIn reality
Whatwe conclude
Alternative H1 false Alternative H1 true
In reality... In reality...
Accept null
Reject alternative
Reject null
Accept alternative
We say...
• There is no realprogram effect
• There is no difference,gain
• Our theory is wrong
We say...
• There is a real programeffect
• There is a difference,gain
• Our theory is correct
• There is no real program effect• There is no difference, gain• Our theory is wrong
• There is a real program effect• There is a difference, gain• Our theory is correct
1-
THE CONFIDENCE LEVEL TYPE II ERROR
1-
TYPE I ERROR POWER
CORRECT
CORRECT
H0 (null hypothesis) true H0 (null hypothesis) false
7/31/2019 Good and Valid Conclusion
http://slidepdf.com/reader/full/good-and-valid-conclusion 35/35
Thank you
Questions ?
