Upload
reginald-fitzgerald
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Quasi-Experimental Designs 101:What Works?
The Need To Know Team January 31 – February 1,
2005
Patricia J. Martens PhD
Outline
Reviewing X’s and O’s Quasi-experimental
time series designs with comparison groups
The Population Health Research Data Repository: what data do we have?
Brainstorming ideas
Tic Tac Toe anyone?
Key features of study designs
Artificial manipulation?
(experimental or observational)
Experimental: Are the groups randomly assigned to receive or not
receive the intervention? (randomized controlled trial) Are the groups selected to be as similar as possible, not
randomly? (quasi-experimental comparison groups)
Research Design Schema
Research Designs
DescriptiveAnalytical
Experimental Observational
Randomly selected
Non-random (quasi-experimental)
Cross-Sectional
Longitudinal
Case-Control Cohort
ProspectiveHistorical Prospective
(Retrospective)
Key Features of Study Designs
Observational: – Information collected concurrently or over a time
period? (cross-sectional or longitudinal)– If over a time period, i.e. longitudinal, do you go
from exposure to disease (cohort) or from disease back in time to examine exposures (case-control)?
– Do you start now and go forward (prospective), or do you have a “cohort” somewhere in the past and you follow them forward (historical prospective)?
Research Design Schema
Research Designs
DescriptiveAnalytical
Experimental Observational
Randomly selected
Non-random (quasi-experimental)
Cross-Sectional
Longitudinal
Case-Control Cohort
ProspectiveHistorical Prospective
(Retrospective)
Study design: observational
Cross-sectional studies – studying all factors at once - both the hypothesized
explanatory and outcome variables
Prospective studies – going forward in time, following a cohort and observing the
effect of exposure to a future outcome
Case-control studies – going backwards in time from the cases/controls to look at
differential exposures
Research Design Schema
Research Designs
DescriptiveAnalytical
Experimental Observational
Randomly selected
Non-random (quasi-experimental)
Cross-Sectional
Longitudinal
Case-Control Cohort
ProspectiveHistorical Prospective
(Retrospective)
Study design: “What Works” proposal
Randomized Controlled (Clinical) Trial – designing a specific intervention and randomly assigning
people to receive it or not to receive it
Quasi-experimental– using a comparison group which is not randomly assigned– Each RHA is a comparison group– A quasi-experimental time series with many comparison
groups (all other RHAs in the province)
Diagrammed and described by Campbell & Stanley (1963)
X is an intervention
O is an outcome measure
X O
Let’s play X’s and O’s
O X O
Let’s play X’s and O’s
O X O
O O
Let’s play X’s and O’s
R means randomly assigned
R O X O
R O O
(pretest-posttest control group design)
Let’s play X’s and O’s
_ _ _ _ means not randomly assigned (quasi-experimental comparison)
O X O- - - - - - - - O O
Let’s play X’s and O’s
O X O
- - - - - - - -
O Oquasi-experimental pretest- posttest design(non-randomized control group)(non-equivalent pretest-posttest comparison group
design)
Let’s play X’s and O’s
0
10
20
30
40
1 2
Hospital BFHI Compliance Scores
Time (8 month interval)
BF
HI
Com
plia
nce
siteArborgPine Falls
Ten Steps and WHOCode each assigned4 points, for totalcompliance of 44
control
intervention
Split-unit anova:p=0.0009
Martens 2001
Examples of a quasi-experimental pretest-posttest comparison group study to determineeffectiveness of hospital policy/education program
O O X O O
Time series (quasi experiments)
Let’s play X’s and O’s
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1992 1993 1994 1995 1996 1997
Breastfeeding Initiation 1992-97
year
prop
ortio
n in
itiat
ing
brea
stfe
edin
g
1994 Breastfeeding study:pregnant women interviewed
Video and breastfeedngbooklet completed, used inindividual prenatalinstruction by CHN
CHN at conference,uses new techniques toaddress prenatal feedingintent
CHN hired
PC Training begun
* p<0.05,one-tailed, adjustedfor birth weight and parity
*
Martens2002
Example of a quasi-experimental time seriesto determine effectiveness of a community-based breastfeeding strategy
Time series (quasi experiment with comparison group)
O O X O O- - - - - - - - - - - - - - -O O O O
Let’s play X’s and O’s
Statistically significantdecline in Region A?
Figure 4: Sample Analysis of Regions A through D Teen Pregnancy Rates 1993 through 2002
0
50
100
150
200
250
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002Year
rate
of t
een
preg
nanc
ies pe
r 10
00
fem
ales
age
d 15
-19
year
s
Region A
Region B
Region C
Region D
Statistically significantdecline in Region A?
Figure 4: Sample Analysis of Regions A through D Teen Pregnancy Rates 1993 through 2002
Figure 4: Sample Analysis of Regions A through D Teen Pregnancy Rates 1993 through 2002
0
50
100
150
200
250
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002Year
rate
of t
een
preg
nanc
ies pe
r 10
00
fem
ales
age
d 15
-19
year
s
Region A
Region B
Region C
Region D
From CIHR proposal submission September 2004
Example of a quasi-experimental time serieswith comparison groups to determine effectiveness of a regional teen pregnancyreduction program
Additions of small amounts of phosphorus to one section of ELA Lake 226 caused surface blooms of blue-green algae, and vividly demonstrated the importance of phosphate as a cause of excessive algal growth or eutrophication. This experiment spurred legislation controlling the input of phosphorus to many water bodies.
http://www.umanitoba.ca/institutes/fisheries/eutro.html
A demonstration of the work ofDr. David Schindler and the ExperimentalLakes project in NW Ontario
Study design: Low internal validity
Anecdote/case study
Pre-experimental just doing a pretest and posttest on one group and
seeing its effect
Cross-sectional a snapshot in time: can’t tell which comes first, but only
that they are “associated”
Study design: medium internal validity
Time series; Time series with qualitative layer– looking over time to see change, with information about
when interventions occurred in the time frame
Case-control– going backwards in time from the cases/controls to look at
different exposures to possible risk factors
Observational (prospective)– going forward in time, observing the effect of exposure on
a cohort to a future outcome
Study design: high internal validity
Randomized Controlled (clinical) Trials, RCT designing a specific intervention and randomly
assigning people to receive it or not to receive it following people to observe the outcome of interest
Quasi-experimental comparison group studies
using a comparison group which is not randomly assigned, but very similar at onset
Inte
rnal
val
idit
y
Low
High
Cross-sectionalPre-experimentalAnecdote/case study
Time series with comparisonObservational (prospective)Case-controlTime series with qualitative layer
Randomized Controlled Trials RCTQuasi-experimental comparison group studies
“There is nothing so useless as doing efficiently that which should not be done in the first place.”
Peter Drucker
MCHP’s … “paperclips”“Population Health Research Data Repository”
Population-Based Health Registry
Hospital
Home Care
Pharmaceuticals
CostVital
Statistics
Provider
Nursing Home
Medical
Family Services
Education
Immunization
National surveys
Census Data EA/DA level
Brainstorming: “What Works” proposal
Pick (a) a policy; and (b) a program– Think of something that your region has done in the past,
somewhere between 1997 and the present (hopefully, with a few years of data AFTER the onset of this)
What OUTCOME measures would you think this would impact?
– Think of what you would expect to see if this intervention was “working”
– Are there specific target groups to which this intervention applies? (e.g. teens, people living in a certain district of your region?)
– What measures of this intervention would be available through the Repository data?
Brainstorm and report! (see sheet for recording)
Policy or Program
Outcome Measure(s)
Target Group
Outcome available in Repository?
Other comments
Teen pregnancy reduction
Teen pregnancy rate
12-19 year olds?
Certain district?
pregnancies
or
live births?
Maybe birth control pill use in Rx data?