Quasi-Experimental Designs 101: What Works?

Quasi-Experimental Designs 101:What Works?

The Need To Know Team January 31 – February 1,

2005

Patricia J. Martens PhD

Outline

Reviewing X’s and O’s Quasi-experimental

time series designs with comparison groups

The Population Health Research Data Repository: what data do we have?

Brainstorming ideas

Tic Tac Toe anyone?

Key features of study designs

Artificial manipulation?

(experimental or observational)

Experimental: Are the groups randomly assigned to receive or not

receive the intervention? (randomized controlled trial) Are the groups selected to be as similar as possible, not

randomly? (quasi-experimental comparison groups)

Research Design Schema

Research Designs

DescriptiveAnalytical

Experimental Observational

Randomly selected

Non-random (quasi-experimental)

Cross-Sectional

Longitudinal

Case-Control Cohort

ProspectiveHistorical Prospective

(Retrospective)

Key Features of Study Designs

Observational: – Information collected concurrently or over a time

period? (cross-sectional or longitudinal)– If over a time period, i.e. longitudinal, do you go

from exposure to disease (cohort) or from disease back in time to examine exposures (case-control)?

– Do you start now and go forward (prospective), or do you have a “cohort” somewhere in the past and you follow them forward (historical prospective)?


Research Designs



Randomly selected


Cross-Sectional

Longitudinal

Case-Control Cohort


(Retrospective)

Study design: observational

Cross-sectional studies – studying all factors at once - both the hypothesized

explanatory and outcome variables

Prospective studies – going forward in time, following a cohort and observing the

effect of exposure to a future outcome

Case-control studies – going backwards in time from the cases/controls to look at

differential exposures


Research Designs



Randomly selected


Cross-Sectional

Longitudinal

Case-Control Cohort


(Retrospective)

Study design: “What Works” proposal

Randomized Controlled (Clinical) Trial – designing a specific intervention and randomly assigning

people to receive it or not to receive it

Quasi-experimental– using a comparison group which is not randomly assigned– Each RHA is a comparison group– A quasi-experimental time series with many comparison

groups (all other RHAs in the province)

Diagrammed and described by Campbell & Stanley (1963)

X is an intervention

O is an outcome measure

X O

Let’s play X’s and O’s

O X O


O X O

O O


R means randomly assigned

R O X O

R O O

(pretest-posttest control group design)


_ _ _ _ means not randomly assigned (quasi-experimental comparison)

O X O- - - - - - - - O O


O X O

- - - - - - - -

O Oquasi-experimental pretest- posttest design(non-randomized control group)(non-equivalent pretest-posttest comparison group

design)


0

10

20

30

40

1 2

Hospital BFHI Compliance Scores

Time (8 month interval)

BF

HI

Com

plia

nce

siteArborgPine Falls

Ten Steps and WHOCode each assigned4 points, for totalcompliance of 44

control

intervention

Split-unit anova:p=0.0009

Martens 2001

Examples of a quasi-experimental pretest-posttest comparison group study to determineeffectiveness of hospital policy/education program

O O X O O

Time series (quasi experiments)


0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1992 1993 1994 1995 1996 1997

Breastfeeding Initiation 1992-97

year

prop

ortio

n in

itiat

ing

brea

stfe

edin

g

1994 Breastfeeding study:pregnant women interviewed

Video and breastfeedngbooklet completed, used inindividual prenatalinstruction by CHN

CHN at conference,uses new techniques toaddress prenatal feedingintent

CHN hired

PC Training begun

* p<0.05,one-tailed, adjustedfor birth weight and parity

*

Martens2002

Example of a quasi-experimental time seriesto determine effectiveness of a community-based breastfeeding strategy

Time series (quasi experiment with comparison group)

O O X O O- - - - - - - - - - - - - - -O O O O


Statistically significantdecline in Region A?

Figure 4: Sample Analysis of Regions A through D Teen Pregnancy Rates 1993 through 2002

0

50

100

150

200

250

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002Year

rate

of t

een

preg

nanc

ies pe

r 10

00

fem

ales

age

d 15

-19

year

s

Region A

Region B

Region C

Region D

Statistically significantdecline in Region A?



0

50

100

150

200

250

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002Year

rate

of t

een

preg

nanc

ies pe

r 10

00

fem

ales

age

d 15

-19

year

s

Region A

Region B

Region C

Region D

From CIHR proposal submission September 2004

Example of a quasi-experimental time serieswith comparison groups to determine effectiveness of a regional teen pregnancyreduction program

Additions of small amounts of phosphorus to one section of ELA Lake 226 caused surface blooms of blue-green algae, and vividly demonstrated the importance of phosphate as a cause of excessive algal growth or eutrophication. This experiment spurred legislation controlling the input of phosphorus to many water bodies.

http://www.umanitoba.ca/institutes/fisheries/eutro.html

A demonstration of the work ofDr. David Schindler and the ExperimentalLakes project in NW Ontario

Study design: Low internal validity

Anecdote/case study

Pre-experimental just doing a pretest and posttest on one group and

seeing its effect

Cross-sectional a snapshot in time: can’t tell which comes first, but only

that they are “associated”

Study design: medium internal validity

Time series; Time series with qualitative layer– looking over time to see change, with information about

when interventions occurred in the time frame

Case-control– going backwards in time from the cases/controls to look at

different exposures to possible risk factors

Observational (prospective)– going forward in time, observing the effect of exposure on

a cohort to a future outcome

Study design: high internal validity

Randomized Controlled (clinical) Trials, RCT designing a specific intervention and randomly

assigning people to receive it or not to receive it following people to observe the outcome of interest

Quasi-experimental comparison group studies

using a comparison group which is not randomly assigned, but very similar at onset

Inte

rnal

val

idit

y

Low

High

Cross-sectionalPre-experimentalAnecdote/case study

Time series with comparisonObservational (prospective)Case-controlTime series with qualitative layer

Randomized Controlled Trials RCTQuasi-experimental comparison group studies

“There is nothing so useless as doing efficiently that which should not be done in the first place.”

Peter Drucker

MCHP’s … “paperclips”“Population Health Research Data Repository”

Population-Based Health Registry

Hospital

Home Care

Pharmaceuticals

CostVital

Statistics

Provider

Nursing Home

Medical

Family Services

Education

Immunization

National surveys

Census Data EA/DA level

Brainstorming: “What Works” proposal

Pick (a) a policy; and (b) a program– Think of something that your region has done in the past,

somewhere between 1997 and the present (hopefully, with a few years of data AFTER the onset of this)

What OUTCOME measures would you think this would impact?

– Think of what you would expect to see if this intervention was “working”

– Are there specific target groups to which this intervention applies? (e.g. teens, people living in a certain district of your region?)

– What measures of this intervention would be available through the Repository data?

Brainstorm and report! (see sheet for recording)

Policy or Program

Outcome Measure(s)

Target Group

Outcome available in Repository?

Other comments

Teen pregnancy reduction

Teen pregnancy rate

12-19 year olds?

Certain district?

pregnancies

or

live births?

Maybe birth control pill use in Rx data?

Documents

Quasi-Experimental Designs 101: What Works?