Introduction to Research Design
Threats to Internal ValidityOne Group Pretest-Posttest Design
O X O
• Campbell & Stanley “pre-experimental.”• Wuensch “experimental.”• OK design if can achieve “experimental
isolation,” as in the chemistry lab.• Correlated samples t test or nonparametric
equivalent.
History
Events other than X between pretest and posttest
• Pre = subjects’ energy consumption.• X = Education on importance of
conserving resources.• Post = subjects’ energy consumption.• History = price of energy increases 50%
between pre and post.
Maturation
Processes that normally cause subjects to change across time.
• Subjects = newly hired employees• Pre = Test of morale• X = Six month program to elevate morale• Post = Test of morale• Maturation = end of honeymoon effect
Testing
Pretesting subjects can change them.• Pre = frequency of conservation behaviors
– have you installed a low-flow shower head?– et cetera
• X = Education on importance of conserving resources
• Post = frequency of conservation behaviors• Testing = just (pre) asking them about
certain behaviors might cause them to try them.
Instrumentation
The measuring instrument changes across time.
• The $1.99 scale for our fishing experiment.– AM versus PM weight of our catch– Spring stretched
• The human observer as instrument– effect of treatment on number of problems in
computer lab– changes in observers from pre to post
Statistical Regression
When scores have an error component, both high and low scores regress towards the mean upon retesting.
• My ESP demo in PSYC 2101• Educational research at Miami Univ.
– mean IQ in school district = 102– mean of selected students = 80– how much regression is expected?
Percent of regression towards the mean, PRM = 100(1 – ρ).
• If pretest-posttest corr = .8,• PRM = 100(1 - .8) = 20%.• Expect regression (up) of .2(102 - 80) =
4.4 points.• That may be enough to get significant pre-
post change.
Mortality
Subjects who drop out of the experiment may differ from those who stay in.
• Subjects = patients with wasting disease• Pre, Post = body weight, X = New drug• 20 patients at pre, 10 at post• Pre mean weight = 97 lb, post = 125.• Was the drug effective?
• Perhaps the sickest (and lightest) 10 dropped out.
• Maybe positive effect from some, who stay in, negative for others, who drop out.
• Correct by computing means only for those who complete study.– creating a problem of external validity.