Quasi-Experimental Designs Chapter 13

Quasi-Experimental Designs

In a quasi-experimental design, the researcher lacks control over the assignment to conditions and/or does not manipulate the causal variable of interest.

A quasi-independent variable is not manipulated by the researcher but rather is an event that occurred for other reasons.

Examples

Does smoking cause cancer?

Did 9/11 cause an increase in prejudice against people of middle-eastern decent?

Do Republican vs. Democratic presidents affect the economy?

Do extreme events (i.e., winning the lottery or being paralyzed) affect day-to-day happiness?

Does campus crime affect applicants to a university?

Questionable Internal Validity

No control over assignment of participants to the independent variable, so internal validity of quasi-experiments is always questionable.

However, some quasi-experimental designs are more internally valid than others.

The extent to which a quasi-experimental design can eliminate possible threats to internal validity determines its usefulness.

Quasi-Experiments and Internal Validity

To infer that X causes Y

1. X must precede Y in time

2. X <-> Y must be related to each other

3. all other alternative explanations of the results are eliminated through random assignment or experimental control

With quasi-experimental designs, you can’t rule out ALL alternative explanations, but you can try to minimize them.

One-group pretest-posttest design

Example: Sample of violent adolescent women

Treatment: anger management class

Measures: aggressive behavior 1 year pre/post

If aggressive behavior is lower at T2 than T1, can we conclude this decrease is caused by the treatment?

Time 1 Measure

Treatment

Time 2 Measure

O1 X O2

One-group pretest-posttest design

Internal validity problems?

Maturation effect: between observations, participants could have grown out of aggressive behavior

History effect: something else in the school environment caused a decrease—less overcrowding

Testing effect: the act of assessing aggression led to awareness of their own aggression…

Time 1 Measure

Treatment

Time 2 Measure

O1 X O2

One-group pretest-posttest design

For these reasons, simple pretest-posttest design should never be used without a control condition and other efforts to control third variables.

Time 1 Measure

Treatment

Time 2 Measure

O1 X O2

Nonequivalent Control Group Designs

Researcher obtains a groups of participants who are similar to the group that receives the quasi-independent variable.

Example: Do extreme events (winning lottery, being paralyzed) have a long-term effect on day-to-day happiness?

Nonequivalent groups: posttest only design

Measure both groups after one receives the quasi-independent variable.

Event (X): winning lottery, paralyzing accident

Advantage: unlike pre-post design, not subject to testing effect.

Threat to Internal Validity: Selection bias, you can not be sure the groups were the same before the treatment.

How could we try to address this?

Time 1: Event Time 2: Measure

X O

-- O

Time Series Designs

Measure the dependent variable on several occasions before and after the quasi-independent variable occurs.

T1 T2 T3 T4 T5 T6 T7 T8 T9

O1 O2 O3 O4 X O5 O6 O7 O8

8 Observations 4.0 4.3 3.9 4.2 2.9 3.1 3.0 3.3

2 Observations 4.2 2.9

Example: Does a well publicized crime spree (X) around Howard campus impact on number of observed applicants (O1-O8) to the university?

What if we only had the two observations?

Simple interrupted time series design

Advantage: Can rule out maturation.

Possible Threat to Internal Validity: Contemporary history –could be confounded with some other event that occurred at the same time.

Simple interrupted time series design

T1 T2 T3 T4 T5 T6 T7 T8 T9

O1 O2 O3 O4 X O5 O6 O7 O8

T1 2 3 4 5 6 7 8 9 10 11 12 13

O1 O2 O3 O4 X O6 O7 O8 -X O9 O10 O11 O12

Reputation 5 4 6 5 10 10 9 6 7 5 4

Interrupted Time Series with a Reversal

Example: Howard graduates earn 3 metals in one Olympics, and earn 0 metals in the next Olympics. What impact does that have on Howard’s reputation for athletics?

Shows the effect of adding AND removing quasi-independent variable.

This pattern of results rules out maturation and history.

Time Series Designs

Comparative Time Series Design

Examines two or more variables over time in order to understand how changes in one variable are related to changes in another variable

Provides indirect evidence that the change in one variable may be causing the change in the other variable

Example: Comparative Time Series Design (Friedberg, 1998)

Time Series Designs

Control Group Interrupted Time Series Design

nonequivalent control group included that does not receive the quasi-independent variable

helps rule out certain history effects

but could still get local history effects

Experimental:O1 O2 O3 O4 X O5 O6 O7 O8

Control :O1 O2 O3 O4 -- O5 O6 O7 O8

Control Group Interrupted Time Series Example: Economic Output of Cities

0

20

40

60

80

100

120

2001 2002 2003 2004 2005 2006 2007 2008 2009

New Orleans

Tampa FL

FEMA went into New Orleans

Longitudinal vs. Cross-sectional Designs

Design Participant #

1990 1995 2000 2005

Longitudinal

P’s 1-25 Age 5 Age 10 Age 15 Age 20

Cross-sectional

P’s 1-25 Age 5

P’s 26-50 Age 10

P’s 51-74 Age 15

P’s 75-100 Age 20

Longitudinal Designs

The quasi-independent variable is time; nothing has occurred from one observation to the next except for the passage of time

O1 O2 O3 O4 O5

Mostly used by developmental psychologists to study age-related changes in how people think, feel, and behave.

Developmental changes in memory, emotions, self-esteem, personality…

Longitudinal Design Example: Marital Satisfaction (Kurdek, 1999)

1 2 3 4 5 6 7 8 9 10

Years of Marriage

Ma

rita

l S

atisfa

ctio

n

Wife

Husband

Note: Years is within-participants. Would you offer to collect these data as a graduate student?

Longitudinal Designs

Three drawbacks:

Difficult to obtain participants who agree to be in a study over a long period of time

Difficult to keep track of participants once they are in the study

Repeatedly testing a sample requires time, effort, and money

Cross-Sectional Designs

Drawback: Age is confounded with generation. Examples:

1. Texting ability by age (10, 20, 30, 40, 50, and 60 year old Ps)

• Younger people are faster texting

• Does texting speed decrease with age?

• Or are there generational effects

with experience with texting?

2. Prejudice as a function of age.

3. Attitudes toward sex in the media as a function of age.

Evaluating Quasi-Experimental Designs

Quasi-experimental designs can show that:

the presumed causal variable preceded the effect in time

cause and effect covary

Quasi-experimental designs do not:

eliminate all other alternative explanations for the results

reason: no random assignment and no experimental control

Increasing Confidence in Quasi-Experimental Results 1. Measure both the effects of the quasi-IV and the

processes assumed to explain the relationship.

2. Include a noncomparable control condition to rule out global history effects.

3…

4…

3. Find a complex pattern of results that cannot be explained by anything except for your quasi-experimental factor.

Complex patterns of data are not sufficient by themseves…

4. Critical multiplism – employ multiple approaches that may converge to yield conclusions that are as concrete as those obtained in experimental research.

• Longitudinal and cross-sectional

• quasi and full experiments

• different quasi-experiments for the same factor

• replications in different populations...

Increasing Confidence in Quasi-Experimental Results

Design your own Quasi-Experiment

Points to consider

-Pick an event or set of events & one DV variable

-Is it longitudinal, cross-sectional, time-series,

pre-post test, or post-test?

-Is a non-equivalent control group or reversal included?

-What is you hypothesis (prediction)?

-What confounds threaten internal validity?

-Can you improve the design to address the confounds?