Quasi-experimental

Designs & Applied

Research

Outline

Quasi-experimental designs

Non equivalent control groups

Time series designs

Program Evaluation

Myths

Where program evaluations is helpful

Basic Steps

Quasi-experiments

Procedures resemble those of true

experiments

But lack the degree of control found in true

experiments

Generally occur when the IV involves:

subject variables (e.g., personality type)

an environmental event (e.g., hurricanes or

having a particular classroom teacher),

the passage of time

Hedrick, Bickman & Rog (1993) “a quasi-

experimental design is not the method of

choice, but rather a fallback strategy”

Cannot infer cause and effect, BUT well

designed quasi-experiments enable you to

demonstrate that rival interpretations are

rendered unlikely

Quasi-experiments

Non-equivalent Group Designs

Posttest only nonequivalent control group design (aka static group comparison)

X O (treatment grp)

O (nonequivalent control grp)

X=treatment O=measurement/observation

Because there is no random assignment to groups, confounding variables may explain any difference observed

Non-equivalent Group Designs

Most common threat to internal validity of

this type of design is SELECTION

No control over which participants are in each group

Experimental group=people who volunteer for the program

Control group = smokers who did not sign up for the training

Selection problem - smokers who choose to participate may differ in some important way from those who do not

Example

Participants

IV

Training Program

No Training Program Participants

Smoking measure

DV

Smoking measure

Pre-test/Post-test Non-equivalent

Control Group

Addition of the pretest measurement allows

researcher to compare the observations

before treatment

Design also allows a researcher to compare

the pretest scores and posttest scores for

both groups

O X O treatment group

O O non-equivalent control group

Pre Post

Example: Research Methods and

Reasoning Ability

0

1

2

3

4

5

Pre Post

Mea

n R

easo

nin

g S

core

Develop Methods

Intervention: critical

thinking seminar

Research Methods

students receive the

intervention (i.e.,

participate in the critical

thinking seminar)

Developmental

Psychology students are

used as a nonequivalent

control group

(i.e., do not attend the

seminar)

May have different experiences (selection-

history effect)

May mature at different rates (selection-

maturation effect)

Be measured more or less sensitively by the

instruments (selection-instrumentation effect)

May drop out of the study at different rates

(differential subject mortality)

May differ in terms of regression to the mean

(differential regression)

What threats of internal validity are

NOT ruled out?

Quasi-experiments very similar to those of

true experiments

When the quasi-experiments were

characterized by:

small differences between group and

controls on the pre-tests

low attrition rates

low levels of participant self-selection into

conditions

How severe are these problems?

Heinsman & Shadish (1996)

Interrupted Time Series Designs

Extension of the simple one group pre and

post design

Participants are pre-tested a number of

times and then post-tested a number of

times after being exposed to the treatment

intervention

O1 O2 O3 O4 X O5 O6 O7 O8

Useful when:

cannot randomize participants

it is possible to obtain a series of

assessments of the DV before and after

treatment

Goal: evaluate the influence of the treatment

by comparing the observations made before

treatment with the observations made after

Interrupted Time Series Designs

Example: Interrupted Time Series

Designs

Intervention:

Course to change

students’ study

habits,

implemented

during the summer

(after semester 4).

DV: semester GPA 0

0.5

1

1.5

2

2.5

3

3.5

4

1 2 3 4 5 6 7 8

Mea

n G

PA

X

Baseline

Post-tx

Maturation: We assume maturational

changes are gradual, not abrupt

discontinuities

Testing: If testing influences responses, these

effects are likely to show up in the initial

observations (i.e., before the intervention)

Regression: If scores regress to the mean,

they will do so in the initial observations

What threats are more easily ruled out?

What threats to internal validity still need

to be considered?

History threats are somewhat lessened

Instrumentation threats also are likely in

some studies

How Many Measurements Are Needed

for a Time Series Design?

Depends on:

amount of random fluctuation (noise) that

may occur in the outcome being measured

how much of an impact the intervention is

expected to have

Typically somewhere between 6 to 15

measurements

Multiple Time Series Designs

Add a comparison group to the simple interrupted time series design:

O1 O2 O3 O4 X O5 O6 O7 O8

------------------------------------------------------------------

O1 O2 O3 O4 O5 O6 O7 O8

Campbell & Stanley consider this an excellent quasi-experimental design

Conclusions About Quasi-

Experimental Designs

More flexible than experimental designs

Less internal validity than experimental

designs

Researcher can take steps to eliminate a

threat that isn’t automatically eliminated by

the design

Often, a threat can’t be eliminated

Program evaluation is carefully collecting

information about a program or some

aspect of a program in order to make

necessary decisions about that program

What is Program Evaluation?

What is Program Evaluation?

Application of social and behavioral research methods to determine the effectiveness of a program or an intervention

Effectiveness can be defined in terms of the following questions: To what extent did the program achieve its goals?

What aspects of the program contributed to its success?

How can the effects of a program be improved?

Cost analyses to determine if program benefits justify the funds expended

Myths About Program Evaluation

Useless activity that generates lots of

boring data with useless conclusions

Is a highly unique and complex process

that occurs at a certain time in a certain

way

Where Program Evaluation Is

Helpful

Improve delivery mechanisms to be more

efficient and less costly

Verify if the program is really running as

originally planned

Produce data or verify results that can be

used for public

Produce valid comparisons between

programs to decide which should be

retained

Key Steps in Program Evaluation

Step 1:

Needs

Assessment

determining that a problem exists and designing a solution for the problem

Key Steps in Program Evaluation

Determining that a problem exists and designing a solution for the problem

Step 2:

Program

Monitoring

Ensuring that the program is carried out as designed

3 major aspect:

1.reaches its intended client population

2.implemented completely and properly

Step 1:

Needs

Assessment

Sources of Implementation Failure

(Kazdin)

Lack of specific criteria and procedures for

program implementation

Insufficiently trained staff

Inadequate supervision of staff provides

opportunity for treatments to drift away

from their intended course

Resistance to the program by staff who do

not believe in its effectiveness or by clients

Key Steps in Program Evaluation

Determining that a problem exists and designing a solution for the problem

Ensuring that the program is carried out as designed

3 major aspect:

1.reaches its intended client population

2.implemented completely and properly

3.identification of unintended effects.

Step 2:

Program

Monitoring

Step 1:

Needs

Assessment

Key Steps in Program Evaluation

Step 3: Impact

Assessment

Has program been effective in meeting its goals

Researcher must devise a way of measuring the outcome and then study the impact of the program on the outcome measure

Key Steps in Program Evaluation

Step 3: Impact

Assessment

Has program been effective in meeting its goals

Researcher must devise a way of measuring the outcome and then study the impact of the program on the outcome measure

Step 4: Efficiency

Analysis

Cost-benefit analysis - compares the dollar cost of operating program to benefits (in dollars)

Cost effectiveness analysis - compares the cost of a program to the size of its outcomes (instead of dollar benefit)