How to read
academic research
(even if you’re not an expert)
Dr. Russell James III, Texas Tech University www.EncourageGenerosity.com
Rule 1
Don’t Freak Out!
You don’t need to eat the whole cow!
You can get important
concepts out of a research
article without fully
understanding every detail
How do you eat a cake with rocks in it?
Don’t try to eat the rocks
Questions for an article
1.Do I care about the research topic?
2.Do I believe the findings?
3.So what?
Abstract: Do I care?Tables: What did they really find?Methods: Do I believe the table?Discussion: So what?Lit. Review: What did we already know?
Title and Abstract: Do I care?
Tables: What did they find?
Methods: Should I believe
the table?
Discussion: So What?
Literature Review:What did we
already know?
Should you believe the findings?Research is messy. Research often disagrees. We want to be able to distinguish strong results from weak ones.
Bad news
Knowing whether you should believe the findings usually requires some statistics
Core statistics concepts you must know
1. Association v. Causation2. Correlation v. Multiple Regression3. Significance v. Magnitude
Association v. Causation
Association: A & B tend to occur together more frequently than one would expect by random chance
Explaining Associations1. Random chance (stuff happens)2. A causes B (sometimes)3. B causes A (sometimes)4. Something else causes both A & B
(sometimes)
Sleeping in your shoes is associated with waking up with a headache.
Why?
1. Random chance2. Sleeping in shoes causes headaches3. The very early stages of a forthcoming
headache causes sleeping in shoes4. Going to bed drunk causes both results
Association v. Causation
• Statistics can show only association
• Statistics can NEVER show causation
We infer causation from experimental design or theory combined with statistical association
Explaining associations:1. Random chance2. A causes B3. B causes A4. Something else causes both A & B
Statistics can easily determine
this
less so with these
Correlationv.
Multiple Regression
Multiple Regression: Above is true when comparing those otherwise similar in certain ways
Correlation: A & B tend to occur together more frequently than one would expect by random chance
CorrelationHigher education and charitable giving tend to occur together (more frequently than one would expect by random chance)
Multiple RegressionHigher education and charitable giving tend to occur together (more frequently than one would expect by random chance)comparing those with otherwise similar income and wealth.
Explaining Associations:1. Random chance2. A causes B3. B causes A4. Something else
causes both A & B
Multiple regression allows us to exclude specific items from #4, unless we can’t or didn’t measure it.
G.E. Quinn, C.H. Shin, M. Maquire, R. Stone (University of Pennsylvania Medical School), 1999, Myopia and Ambient Lighting at Night, Nature, 399, 113.
Nature says kids’ nightlights cause myopia
“Although it does not establish a causal link, the statistical strength of the association of night-time light exposure and childhood myopia does suggest that the absence of a daily period of darkness during early childhood is a potential precipitating factor in the development of myopia.”
G.E. Quinn, C.H. Shin, M. Maquire, R. Stone (University of Pennsylvania Medical School), 1999, Myopia and Ambient Lighting at Night, Nature, 399, 113.
Nature says kids’ nightlights cause myopia
1. Random chance
2. A causes B
3. B causes A
4. Something else causes both A & B
J. Gwiazda, E. Ong, R. Held, F. Thorn (New England College of Optometry), 2000, Myopia and Ambient Night-Time Lighting, Nature, 399, 113.
Rebuttal: Maybe parents’ myopia causes both nightlights and child’s myopia?
“…we find that myopic parents are more likely to employ night-time lighting aids for their children. Moreover, there is an association between myopia in parents and their children…”
“…Quinn et al.’s study should have controlled for parental myopia.”
Significance v.
Magnitude
Statistics tests a small sample to predict the whole population
Significance shows how likely our result might have been due to an unusual random sample, rather than an actual difference in the population
Most papers report some measure of statistical significance (chance that the association was due to a weird random sample)
• p-value• confidence interval
How likely is it to randomly draw these five fruits from a truckload with as many apples as oranges?
p-value
p-value
p<.05 = there is less than a 5% chance that the result was caused by an unusual random sample where there was no actual (population) difference
Was there a significant gender difference in planned givers with a will v. a trust?
No
This (sample) difference could have easily occurred even if the two (population) groups were the same
It DOES NOT mean the two (population) groups do not differ, only that WE CAN’T TELL.
No “*” means we can’t confidently tell the effect of this item
95% Confidence intervalIf you kept taking random samples, 95% of the time the true (population) value would appear inside the confidence interval associated with each sample
PopulationAverage Strength
SampleAverage Strength
Confidence Interval
S. Huck and I. Rasul (2008) Testing consumer theory in the field: Private consumption versus charitable goods
Dashed line is a 95% confidence interval
How likely is it to randomly draw these five fruits from a truckload with as many apples as oranges?
Would your answer change if I got to draw 20 times to find this group?
Multiple Comparisons Problem
If all variables are random, about one out of 20 will have a p-value<.05
“We tested 100 items and found 5 to be significant at p<.05.”
Significance v. Magnitude
It is possible to be highly confident of a very small effect. This may be publishable, but not practically important.
Numbers (coefficients) resulting
from complex statistical techniques may not be directly
interpretable in terms of real world magnitude
The impact of children
on the probability
of exclusively
secular giving is
“-0.089”, but the meaning
of that number is not easily translated
Even with complex techniques, we can easily compare sign and
magnitude relative to other variables
Race and education factors are
3-4 times as large.
More children have an opposite
relationship compared with more education.
Odds ratios are differentUsually you can compare sign and size, but odds ratios are always positive
Odds ratios: the odds of an event occurring in one group over the odds of it occurring
in another group <1 negative; >1 positive; =1 none
Pamala Weipking (2008) Giving to particular charitable organizations: Do materialists support local organizations and do Democrats donate to animal protection?
Odds ratios <1 correspond with negative coefficient numbers in other reporting
Finding academic research articles
Includes everything, even working papers and industry literature
ISI ranked academic journals articles only
How to read
academic research
(even if you’re not an expert)
Dr. Russell James III, Texas Tech University www.EncourageGenerosity.com