prev

next

of 61

Published on

07-Apr-2015View

91Download

0

Transcript

Getting a Grip on Statistics:Whats Right & Wrong with Numbers in the NewsCalifornia Endowment Health Journalism Fellowships Journalism Seminar, Los Angeles, CA Saturday, October 23

SelfSelf-exams effective?

SelfSelf-exams, Take II

SelfSelf-exams, Take III

Study: Breast self-exams may not matter y One study, and peace of mind is of no use y Self-exams dont cut breast-cancer death risky

SelfSelf-exams, Take IV (in print)

SelfSelf-exams, Take V

Monthly self-breast exams still essential y Women wasting their time y More confusing information tonight y Breast self-exams may not matter y Self-exams dont cut breast-cancer death risk y Everything weve heard about breast cancer prevention is upside down [may be wrong]y

Breast self-exam headlines self-

y

Whats at stake?More Americans get their news from local sources Media coverage can and does shape public agendas, public opinion, and ultimately behavior The words you choose and the numbers you present matter!

Coverage of Statistics Matters!

y

y

Scientists keep changing their minds. They tell us that coffee does or doesnt cause various medical problems, time after time offering different advice. They tell us that a drug works fine, then take it off the market because its too risky to use. To some people, this switching gives science a bad name. Actually its science working just as its supposed to work.- Victor Cohen & Lewis Cope (2001), p.9-10

The Certainty of Uncertainty

Probabilistic (proof) Cannot control for everything Rely on the accumulation of evidence, continually testing and re-testing ideas y Requires careful examination of the merits and limitations of individual studies y New results need to be put in context y Importance of replication y Remember: all studies can be improved!y y y

Nature of Medical/Social Science

y

y

Just because the subject is science and the researchers are medical professionals does not mean usual skepticism should be suspended Be wary of:Anecdotal evidence Expert opinion Scientific studies Conflicts of interest

y

Ask for numbers and evaluate the evidence

A Note of Caution!

1. 2. 3. 4. 5. 6. 7.

Alternative explanations Data, distributions, and variability Probability and significance Sampling Power Types of studies Questions to ask

Overview

Crime rates increase with ice cream sales y People who live together before marriage are more likely to get divorced y The longer patients wait for surgery, the larger their chances of survival y The Denver Broncos lose more often when I dont watch the gamey

Cause and Effect? (Bivariate Relationships)

Correlation Causation

y

Alternative explanations?Crime rates increase with ice cream sales People who live together before marriage are more likely to get divorced The longer patients wait for surgery, the better their chances of survival The Denver Broncos lose more often when I dont watch the game

y y

Be wary of spurious relationships Does association persist controlling for other factors?

Confounding Factors TheImportance of Multiple Controls!

y

Measures of central tendency (what they can and cannot tell you)Mean arithmetic average

Data & Distributions

y

Measures of central tendency (what they can and cannot tell you)Mean arithmetic average Median midpoint Mode most common

Data & Distributions

http://www.brighton-webs.co.uk/statistics/images/central_tendency.gif

Mean, Median, & Mode

New hypothetical disease POA y Compare ERs of Hospitals AMW and ALAy

Problem with Central Tendencies

City Los Angeles Phoenix San Diego San Francisco Seattle Overall

Patients 559 233 232 605 2146 3775

% Surviving 88.9 96.8 91.7 83.1 85.8 86.7

ER at Hospital ALA

City Los Angeles Phoenix San Diego San Francisco Seattle Overall

Patients 811 5255 448 449 262 7225

% Surviving 85.6 92.1 85.5 71.3 76.7 89.1

ER at Hospital AMW

Treatment of POA ALA Number of patients: 3775 Percent surviving: 86.7 Number of patients: 7225 Percent surviving: 89.1

AMW

POA Survival Rate Comparison

Percent Surviving POA ALA AMW Overall 86.7 89.1 Los Angeles 88.9 85.6 Phoenix 96.8 92.1 San Diego 91.7 85.5 San Francisco 83.1 71.3 Seattle 85.8 76.7

POA Survival Comparison

Percent Ontime ArrivalsAlaska Air AM West

Overall Los Angeles Phoenix San Diego San Francisco Seattle

86.7 88.9 96.8 91.7 83.1 85.8

89.1 85.6 92.1 85.5 71.3 76.7

Source: www.cs.cmu.edu/afs/cs/academic/class/15299/handouts/lecture20

Averages can be misleading!

y

Given a measure (or measures of central tendency), we still need to know something about the spread or scatter of the distribution of valuesRange (low to high) Percentiles Standard deviation

Measures of Dispersion

Aristotle: the probable is what usually happens y Not was always happens y Improbable events can and do occurand may be more frequent than we realize!y

Probability

In a probabilistic world, all results and events can be affected by chance y A p-value is a measure of the probability that a result is actually meaningful, that is not due to random variation (chance) y The lower the p-value, the higher the likelihood that the finding is a real resulty

P-values (probability values)

y

By convention, a p-value of 0.05 or less, is consider statistically significantp=0.05 means that 1 in 20 times (5 percent), the observed result could have happened by chance p=0.001 means that 1 in 1,000 times (1 percent), the observed result is due to chance

y

Note: this does NOT mean that chance is ruled out!

P-values (probability values)

y

Finding a result that is not there (Type I Error)At standard significance levels, 5 out of 100 researchers will conclude that a treatment helps, when it really has no effect

y

Not finding a result that is there (Type II)y

A study may simply include too few subjects to detect a real result sufficient power is necessary (more on this in a minute)

Error: Type I & Type II

Repeated tests will produce different results y Confidence Level the percentage of times that repeated trials should produce a result within the confidence interval y Confidence Interval the range within the true value of the result probably liesy

Confidence Intervals

Small confidence intervals indicate that the true effect is unlikely to deviate much from the studys findings y Large confidence intervals mean that the studys findings are not very precisey

Confidence Intervals

Neg. effect

No effect

Pos. effect

Confidence Intervals

Just because a result is statistically significant does not necessarily mean the effect is large y In addition to knowing the size of the confidence interval, we also want to know the size of the effect (substantive significance)y

Statistical Confidence vs. Substantive Size of Effects

A

B

C

D

Neg. effect

No effect

Pos. effect

Statistical Confidence vs. Substantive Size of Effects

y

What does is mean for a result to be important?Statistically significant results are not always important Most powerful findings are those that are BOTH statistically and substantively significant

Statistical vs. Substantive Significance

y

Results of new treatment for disease in puppies:33.3% survived 33.3% died during treatment and the other one ran away!

y y

What if the third puppy had survived?The study would have a 66.7% survival rate

Lesson: small changes in small samples can drastically affect the results!

Size of the population(and why it matters!)

Sample size increases our confidence y Law of Large Numbers as the number of cases increases, we can be more confident in the validity (accuracy) and reliability (reproducibility) of the findings y Always ask for the numerator and denominator!y

Large Numbers Yield Power

Bad coins? y Expected number of heads? y Lets say we conduct coin-flipping trials, with 10 flips per trialy

Problems with Small Samples

y

If we repeat our 10 flips per trial a thousand times, how many trials should we expect to get exactly 5 heads?a) b) c) d) About About About About 500 900 400 250 (50 (90 (40 (25 percent percent percent percent of of of of the the the the trials) trials) trials) trials)

Problems with Small Samples

y

If we repeat our 10 flips per trial a thousand times, how many trials should we expect to get exactly 5 heads?a) b) c) d) About About About About 500 900 400 250 (50 (90 (40 (25 percent percent percent percent of of of of the the the the trials) trials) trials) trials)

Problems with Small Samples

Expected Distribution 10 flips, 1,000 times

y

M&M Mars produces blue, green, yellow, orange, red, and brown M&Ms according to a specified distribution. Based on your M&M packet, which color do you think they produce most?

Sampling example: M&Ms

y

Which color do they produce most? Blue 24% Orange 20% Green 16% Yellow 14% Red 13% Brown 13%

Sampling example: M&Ms

Researchers use samples to represent larger populations y Findings can only be generalized to the population for which the sample is drawn be wary of unrepresentative samples!y

Examples:x VA hospital results x Surveys assessing disease rates

Samples & Generalizability

y y

Anecdotes, ideas, opinions Descriptive reportsCase studies Cross-sectional study

y

Analytic studiesCase-controls Cohort studies

y

Experimental studiesRandomized controlled trials Blinded randomized controlled trials

Types of Studies (Evidence)

y

Case studies Identifying some unusual or interesting cases that alert physicians to potential relationships Helpful when phenomena stand out by themselves

y

Cross-sectional (prevalence) study Wide-angle shot Rate of disease in population Make observations Snapshot in time Conclusions may be overstated

Descriptive Reports (Evidence)

y

Case-controls Very common in disease outbreaks Compare sick people (the cases) to well people (the controls) Additional work may be needed to identify culprit (relish example)

y

Cohort studies Motion picture studies Follow people over time, comparing individuals to their peers Watch for drop-outs

Analytic Studies (Evidence)

y

Randomized controlled trials Common for testing drugs Treatment group vs. control group Example: China breast self-exam study

y

Blinded, randomized controlled trials Double-blind, triple-blind Gold-standard

Experimental Trials (Evidence)

y

Simply because the randomized control trial is the gold standard for medical research does NOT mean the results should be believed:Did the randomization work? How large was the sample? What is the sample and to what population can it be generalized? Length of the study? Is the analysis appropriate?

Gold Standard?

y

A new study reveals a breakthrough treatment that reduces patients cancer risk by one-halfWhere should the story go?

Putting Results in Context: Absolute vs. Relative Risk

Putting Results in Context: Exercise & Cancer?

Antidepressants raise risk of suicideConcern mounts about Prozac, Paxil, Zoloft

Antidepressant-Suicide Link Borne Out in Review of 702 Studies Study links SSRIs to increased suicide risk Studies Raise Questions About AntidepressantSuicide Link Suicide Risk from Antidepressants Remains Unclear Study of antidepressants and suicide may expandScientists find mixed results after looking at the drugs impact on adults

Putting Results in Context: Are Findings Consistent?

Choose absolute over relative risk y Provide known cues (has the study been published?) y Situate the study in the existing body of evidence, especially if recent evidence is mixed y Avoid anecdotes that contradict the evidence (anecdotes illustrating the issue at hand, however, can be helpful)y

Putting Results in Context: Lessons Learned

y

Self-exams?266,000 women Randomly assigned to 2 groups Instruction group taught self exams (and reinforced) Followed women for 10 years

y

Findings:No difference in mortality Nearly twice as many benign tumors in instruction group than in control

Breast self-exams: selfWho is the audience?

y

Watch the words you use they matter!

Overhyped Health Headlines Revealed, Popular Science, Aug 2009

y

Questions to ask and things to consider:Where was the study published? What type of study was it? What was the size? What was the sample? And to what population can results be generalized? Whats the size of the effect in absolute terms? Does the study comport with previous findings? How soon will the treatment be available?

Tips to Use for Every Study

y

Things you can do when you have a small amount of time:Draw on relationships with trusted sources Choose your language carefully Use known cues

Under Deadline

1. Qualify the results (e.g. what was the sample?) and use known cues (e.g. has the study been published and reviewed by other experts?) 2. Avoid overreaching statements (e.g. proves, cure, etc) 3. Choose absolute over relative risk (e.g. report a 1 to 2 percent increase rather than saying risk doubles) 4. State who funded the research 5. Explain medical terminology

Lessons LearnedPart I

6. Provide information on alternative treatments where possible 7. Mention when treatment will be available to the public if applicable 8. Avoid anecdotes contradicting evidence 9. Mention known negatives of products (previous wisdom) 10.Put the results into context and insert public health messages where possible 11.Provide follow-up resources!

Lessons LearnedPart II

Getting a Grip on Statistics:Whats Right & Wrong with Numbers in the NewsCalifornia Endowment Health Journalism Fellowships Journalism Seminar, Los Angeles, CA Saturday, October 23