How to read academic research (beginner's guide)

How to read

academic research

(even if you’re not an expert)

Dr. Russell James III, Texas Tech University www.EncourageGenerosity.com

Rule 1

Don’t Freak Out!

You don’t need to eat the whole cow!

You can get important

concepts out of a research

article without fully

understanding every detail

How do you eat a cake with rocks in it?

Don’t try to eat the rocks

Questions for an article

1.Do I care about the research topic?

2.Do I believe the findings?

3.So what?

Abstract: Do I care?Tables: What did they really find?Methods: Do I believe the table?Discussion: So what?Lit. Review: What did we already know?

Title and Abstract: Do I care?

Tables: What did they find?

Methods: Should I believe

the table?

Discussion: So What?

Literature Review:What did we

already know?

Should you believe the findings?Research is messy. Research often disagrees. We want to be able to distinguish strong results from weak ones.

Bad news

Knowing whether you should believe the findings usually requires some statistics

Core statistics concepts you must know

1. Association v. Causation2. Correlation v. Multiple Regression3. Significance v. Magnitude

Association v. Causation

Association: A & B tend to occur together more frequently than one would expect by random chance

Explaining Associations1. Random chance (stuff happens)2. A causes B (sometimes)3. B causes A (sometimes)4. Something else causes both A & B

(sometimes)

Sleeping in your shoes is associated with waking up with a headache.

Why?

1. Random chance2. Sleeping in shoes causes headaches3. The very early stages of a forthcoming

headache causes sleeping in shoes4. Going to bed drunk causes both results

Association v. Causation

• Statistics can show only association

• Statistics can NEVER show causation

We infer causation from experimental design or theory combined with statistical association

Explaining associations:1. Random chance2. A causes B3. B causes A4. Something else causes both A & B

Statistics can easily determine

this

less so with these

Correlationv.

Multiple Regression

Multiple Regression: Above is true when comparing those otherwise similar in certain ways

Correlation: A & B tend to occur together more frequently than one would expect by random chance

CorrelationHigher education and charitable giving tend to occur together (more frequently than one would expect by random chance)

Multiple RegressionHigher education and charitable giving tend to occur together (more frequently than one would expect by random chance)comparing those with otherwise similar income and wealth.

Explaining Associations:1. Random chance2. A causes B3. B causes A4. Something else

causes both A & B

Multiple regression allows us to exclude specific items from #4, unless we can’t or didn’t measure it.

G.E. Quinn, C.H. Shin, M. Maquire, R. Stone (University of Pennsylvania Medical School), 1999, Myopia and Ambient Lighting at Night, Nature, 399, 113.

Nature says kids’ nightlights cause myopia

“Although it does not establish a causal link, the statistical strength of the association of night-time light exposure and childhood myopia does suggest that the absence of a daily period of darkness during early childhood is a potential precipitating factor in the development of myopia.”

G.E. Quinn, C.H. Shin, M. Maquire, R. Stone (University of Pennsylvania Medical School), 1999, Myopia and Ambient Lighting at Night, Nature, 399, 113.

Nature says kids’ nightlights cause myopia

1. Random chance

2. A causes B

3. B causes A

4. Something else causes both A & B

J. Gwiazda, E. Ong, R. Held, F. Thorn (New England College of Optometry), 2000, Myopia and Ambient Night-Time Lighting, Nature, 399, 113.

Rebuttal: Maybe parents’ myopia causes both nightlights and child’s myopia?

“…we find that myopic parents are more likely to employ night-time lighting aids for their children. Moreover, there is an association between myopia in parents and their children…”

“…Quinn et al.’s study should have controlled for parental myopia.”

Significance v.

Magnitude

Statistics tests a small sample to predict the whole population

Significance shows how likely our result might have been due to an unusual random sample, rather than an actual difference in the population

Most papers report some measure of statistical significance (chance that the association was due to a weird random sample)

• p-value• confidence interval

How likely is it to randomly draw these five fruits from a truckload with as many apples as oranges?

p-value

p-value

p<.05 = there is less than a 5% chance that the result was caused by an unusual random sample where there was no actual (population) difference

Was there a significant gender difference in planned givers with a will v. a trust?

No

This (sample) difference could have easily occurred even if the two (population) groups were the same

It DOES NOT mean the two (population) groups do not differ, only that WE CAN’T TELL.

No “*” means we can’t confidently tell the effect of this item

95% Confidence intervalIf you kept taking random samples, 95% of the time the true (population) value would appear inside the confidence interval associated with each sample

PopulationAverage Strength

SampleAverage Strength

Confidence Interval

S. Huck and I. Rasul (2008) Testing consumer theory in the field: Private consumption versus charitable goods

Dashed line is a 95% confidence interval

How likely is it to randomly draw these five fruits from a truckload with as many apples as oranges?

Would your answer change if I got to draw 20 times to find this group?

Multiple Comparisons Problem

If all variables are random, about one out of 20 will have a p-value<.05

“We tested 100 items and found 5 to be significant at p<.05.”

Significance v. Magnitude

It is possible to be highly confident of a very small effect. This may be publishable, but not practically important.

Numbers (coefficients) resulting

from complex statistical techniques may not be directly

interpretable in terms of real world magnitude

The impact of children

on the probability

of exclusively

secular giving is

“-0.089”, but the meaning

of that number is not easily translated

Even with complex techniques, we can easily compare sign and

magnitude relative to other variables

Race and education factors are

3-4 times as large.

More children have an opposite

relationship compared with more education.

Odds ratios are differentUsually you can compare sign and size, but odds ratios are always positive

Odds ratios: the odds of an event occurring in one group over the odds of it occurring

in another group <1 negative; >1 positive; =1 none

Pamala Weipking (2008) Giving to particular charitable organizations: Do materialists support local organizations and do Democrats donate to animal protection?

Odds ratios <1 correspond with negative coefficient numbers in other reporting

Finding academic research articles

Includes everything, even working papers and industry literature

ISI ranked academic journals articles only

How to read

academic research

(even if you’re not an expert)

Dr. Russell James III, Texas Tech University www.EncourageGenerosity.com