Quasi-Experiments – Outline 1. True Experiments a. Characteristics b. Threats to validity controlled by experiments c. Threats not controlled by experiments

Quasi-Experiments – Outline

1. True Experimentsa. Characteristicsb. Threats to validity controlled by experimentsc. Threats not controlled by experimentsd. Obstacles to true experiments in the field

2. Quasi-experimentsa.The logic of quasi-experiments b.Non-equivalent control group design

• Example – Langer & Rudin (1976)c. Interrupted time-series design

• Example – Campbell (1969)

Quasi

True Experiments - Characteristics

• True experiments are characterized by:• A manipulation• A high degree of control• An appropriate comparison

(the major goal of exerting control)

• Manipulation in the presence of control gives you an appropriate comparison.

Quasi

Threats to validity controlled by true experiments

• History • occurrence of an event other than the treatment

Quasi


• Maturation • participants always change as a function of time. Is change in behavior due to something else?

Quasi


• Testing • improvement due to practice on a test (familiarity with procedure, or with testers expectations)

Quasi


• Instrumentation • especially if humans are used to assess behavior (fatigue, practice)

Quasi


• Regression • when first observation is extreme, next one is likely to be closer to the mean.

Quasi


• Selection • if differences between groups exist from the outset of a study

Quasi


• Mortality • if exit from a study is not random, groups may end up very different

Quasi


• Interactions of selection… • with History• with Maturation• with Instrumentation (ceiling

effects)

Quasi

Note difference between these threats:

• Maturation• One group; performance

better on post-test than on pre-test

• Interaction of Maturation & Selection• Two or more groups• Performance difference

larger on post-test than on pre-test

Quasi

Threats to validity not controlled by experiments

• Contamination• communication of

information about the experiment between groups of subjects

• Cook & Campbell (1979):• resentment• ‘compensatory rivalry’• diffusion of treatment:

control subjects use information given to others to change their own behavior.

Quasi

Contamination – an example

• Craven, Marsh, Debus, & Jayasinghe (2001)

• Journal of Educational Psychology

• Teachers trained to improve students’ academic self-concept through praise

• Internal control• External control

Quasi

Contamination – an example

• Craven, Marsh, Debus, & Jayasinghe (2001)

• Next slide shows T2 (post-test) academic self-concept scores as a function of T1 scores for control children only.

Quasi

External controlInternal control

Internal high focusInternal low focus

1.0

0.5

0.0

-0.5

-1.0Low Medium High T1 acad self concept

Diffusion

No diffusion

T2 acad self concept

Low focus group consistently higher than external control

Resentful demoralization?Overzealous cooperation?


• Threats to external validity • best way to deal with this is replication

Quasi


• Hawthorne effects • changes in a person’s behavior due to being studied rather than the manipulation.

• a special kind of reactivity.

Quasi

Hawthorne effects

• Demand characteristics • cues communicated by researcher

• subject’s under-standing of their role

Quasi

Hawthorne effects

• Role of “research subject” • Is subject behaving the way he thinks a person in that role should behave?

• (E.g., hypnotized person)

Quasi

Hawthorne effects

• Orne (1962) • ‘good subjects’ think they are contributing to science by complying with researcher’s demands

Quasi

Hawthorne effects

• What to do about Hawthorne effects?

• Orne (1962): Use quasi-control subjects as “co-investigators”

• They do your task, reflect on demand characteristics of the experiment.

Quasi

Obstacles to true experiments in the field

• Sometimes, we cannot bring the phenomenon we want to study into the lab, so we have to work in the field.

• Can we do experiments in the field?

Quasi


• Can’t get permission from individuals in authority?

• Your study may involve some time and effort on their part. But what’s in it for them?

• In schools, parents also have to agree.

Quasi


• Can’t assign subjects to groups randomly?

• have to work with intact groups (e.g., classes in a school)

Quasi

Quasi-Experiments

• Quasi-experiments resemble true experiments… • usually include a

manipulation, and provide a comparison.

• …but they are not true experiments.• lack high degree of control

that is characteristic of true experiments.

Quasi

Quasi-Experiments

• Quasi-Experiments are compromises

• They allow the researcher some control when full control is not possible.

Quasi

Quasi-Experiments

• Because full control is not possible, there may be several “rival hypotheses” competing as accounts of any change in behavior observed.

• How do we convince others that our hypothesis is the right one?

Quasi

The Logic of Quasi-Experiments

• Eliminate any threats you can• Show how each threat to

validity on list given above is dealt with in your study.

• Argue that others don’t apply.• using evidence or logic

Quasi

Two kinds of quasi-experiments

• Non-equivalent control group

• “non-equivalent” because not randomly assigned

Quasi


• Interrupted time-series design

• a series of observations over time, interrupted by some treatment

Quasi

Non-equivalent Control Group design

• Control group is “like” the treatment group.

• Chosen from same population

• Pre- and post-test measures obtained for both groups, so similarity can be assessed.

Quasi


• Control group is not equivalent

• subjects are not randomly-assigned to control & treatment groups

• so best you can do is argue that comparison is appropriate.

Quasi


• If the groups are comparable to begin with, this design potentially eliminates threats to internal validity due to:

• History• Maturation• Testing• Instrumentation• Regression

Quasi

Problems with the NECG design

• Threats to validity due to interactions with selection may not be eliminated using the NECG design.

• Selection and maturation

• Most likely when treatment group is self-selected (as in psychotherapy cases – people who sought help).

Quasi


• Selection and history • Does one group experience some event that has a positive or negative effect (e.g., teacher of one class leaves)?

Quasi


• Selection and instrumentation

• Does one group show ceiling or floor effects?

Quasi


• Regression to the mean • Are one group’s pretest scores more extreme than the other group’s?

Quasi

Possible NECG study outcomes

• both experimental and control groups show improve-ment from pretest to posttest

• appears not to be any effect of the treatment

Quasi

Pretest Posttest

Control group


• Looks like a treatment effect, but there may be a threat due to• selection and maturation, • selection and history

Quasi

Pretest Posttest

Control group


• Selection and maturation could be a threat

• Or interaction of selection and • history• testing• instrumentation• or mortality.

Quasi

Pretest Posttest

Control group


• Interaction of selection and regression looks like a serious threat here

• Selection and maturation probably not a threat here.

Quasi

Pretest Posttest


• Crossover effect• Clearest evidence for an

effect of the program of any of these graphs.

• Selection and instrumentation not a problem – no ceiling or floor effects

Quasi

Pretest Posttest

Quasi-experiment example

• Langer & Rudin (1976)• Research conducted in

retirement home.

• Residents on one floor given more control over their daily lives

• Residents of another floor given same interaction with staff, but no increased control.

Quasi

Langer & Rudin (1976) – Measures

• Ratings• Self-report of feeling of

control from residents• Staff assessments of mental &

physical well-being, by ‘blind’ assessors

• Objective measures • record of movie attendance• participation in “Guess how

many jelly-beans” contest on each floor

Quasi

L & R (1976) – limits on control

• L & R had no control over • who entered the home• who was assigned to either

floor.• no control over staff hiring

or firing / resigning.

Quasi

L & R (1976) – Possible Problems

• Interaction of Selection and Maturation• even if groups have similar

pretest scores, they may differ on things pretest didn’t measure

• probably not a problem here – people on both floors had similar SES

• assigned to floors randomly, not by health status.

Quasi


• Selection and history • suppose a popular (or unpopular) nurse left one of the floors during the study. That might influence well-being.

• L & R did not address this issue.

Quasi


• Selection and instrumentation

• did one group show ceiling or floor effects?

• L & R say, no.

Quasi


• Regression • were one group’s pretest scores more extreme than the others?

• L & R say, no.

Quasi


• Observer bias and Contamination

• observers in the L & R study were not aware of the hypothesis.

• L & R reported there was little communication between floors.

Quasi


• Hawthorne Effect • cannot be ruled out, but L & R took care to give both floors same attention.

• Message varied between floors, but “face time” was the same.

Quasi


• External Validity • might be an issue. • home involved was rated

“one of the finest” in the state

• subjects may have been atypical in their desire for control

Quasi


• Non-equivalent control group

• Interrupted time-series design

• a series of observations over time, interrupted by some treatment

Quasi

Time-Series Designs

• In T-S designs, performance is measured both before and after a treatment.

• If there is an abrupt change in performance at time of treatment, we conclude that treatment worked.

Quasi

Time-series designs example

• Campbell (1969)• Effect of speed limit

reduction on traffic fatalities in Connecticut

• incidence of traffic fatalities in years before and after the speed limit reduction,

• conclusion: speed limit change had a modest effect.

Quasi

Campbell (1969)

• Any threat to internal validity?

• other explanations for any change in traffic fatality incidence:• Changes in car safety• Weather• Record keeping

Quasi

Campbell (1969)

• Any threat to internal validity?

• Such effects should be similar in neighboring states

• Campbell found no change in fatality incidence in those states.

Quasi

Campbell (1969)

• Any threat to external validity?

• E.g., would treatment have same effect in other states, or are people in Connecticut more law-abiding?

Quasi

Campbell (1969)

• Time-series design eliminates most other threats to validity – e.g., maturation, testing, regression.

• For example, maturation would probably not produce a sudden change in performance of the kind found in Time-Series Designs.

Quasi

Documents

Quasi-Experiments – Outline 1. True Experiments a. Characteristics b. Threats to validity controlled by experiments c. Threats not controlled by experiments