Embed Size (px)
Citation preview
1
A Framework for StudyingVariation in Program Effects
June 14, 2015Michael J. Weiss
Based on: Weiss, M. J., Bloom, H. S., & Brock, T. (2014). A Conceptual Framework for Studying the Sources of
Variation in Program Effects. Journal of Policy Analysis and Management, 33(3).
2
Outline
Section I: Introduction
Section II: Proximal Sources of Variation in Program Effects
Section III: Connecting the Treatment Contrast to Program Implementation & Treatment Fidelity
3
Section 1
Introduction
4
Consequences of Effect Variation
• Targeting services (types of clients / locations)
• Equity / fairness• Best case / worst case scenarios• Generalizability / Scaling• Research design–What research questions to address?– Sample size– Etc.
5
Three Motivating Studies
• After School Programs– certain types “work” (sequenced activities, active
learning, limited # of goals, activities tied to goals)
• Charter Schools– certain types “work” (clear behavior standards,
zero-tolerance, coaching new teachers)
• Welfare-To-Work– certain types “work” (counseling to get jobs quick,
personalized attention)– context mattered (low unemployment rate larger
effects)
6
DEFINING A PROGRAM EFFECT FOR AN INDIVIDUAL
Section 1: Introduction
7
Weight Loss Program
No Program
Potential Outcome
Potential Outcome
ProgramEffectOnNeyman
Neyman
8
ESTIMATING THE AVERAGE PROGRAM EFFECT FOR A GROUP OF INDIVIDUALS
Section 1: Introduction
9
Estimating the Average Program Effect
Randomize individuals (or groups) to: 1. Receive program offer2. Not receive program offer
Compare two group’s average future outcomes
10
DEFINING VARIATION IN PROGRAM EFFECTS
Section 1: Introduction
11
Weight Loss Program
No Program
Potential Outcome
Potential Outcome
ProgramEffectOnNeyman
Neyman
Weight Loss Program
No Program
Potential Outcome
Potential Outcome
ProgramEffectOnRubin
Rubin
The Framework
Outcomes
Treatment Contrast
Program Effect
12
Treatment Offered
Take-up
Treatment Offered
Treatment Fidelity
Implementation Process
Client Characteristics and Context
Characteristics of Implementing Organization
Implementation Plan
Treatment Planned
Prog
ram
Mod
el
Take-up
OutcomesTreatment Received
Treatment Received
13
Section 2
Proximal Sources of Variation in Program Effects
14
The Three C’s
1. Treatment Contrast
2. Client Characteristics
3. Program Context
15
1. THE TREATMENT CONTRAST
Section 2: Proximal Sources of Variation in Program Effects
16
Weight Loss Program
No Program
Potential Outcome
Potential Outcome
ProgramEffectOnNeymanContrast 1
Neyman
Weight Loss Program
Alternate Program
Potential Outcome
Potential Outcome
ProgramEffectOnNeymanContrast 2
Neyman
Treatment Contrast and Program Effect
Treatment Received
Outcomes
Outcomes
Treatment Received
Treatment Contrast
Program Effect
17 Client Characteristics and Context
Characteristics of Implementing Organization
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el Treatment Offered
Take-up
Take-up
Treatment Offered
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el
Outcomes
- Employment - Achievement - Recidivism - etc.
Outcomes
- Employment - Achievement - Recidivism - etc.
Treatment Received
- Content - Quantity - Quality - Conveyance
Mediators
Treatment Received
- Content - Quantity - Quality - Conveyance
Mediators
18
2. CLIENT CHARACTERISTICS
Section 2: Proximal Sources of Variation in Program Effects
19
Weight Loss Program
No Program
Potential Outcome
Potential Outcome
ProgramEffectOnHigh-risk Type
High-Risk
Weight Loss Program
No Program
Potential Outcome
Potential Outcome
ProgramEffectOnLow-risk Type
Low-Risk
Client Characteristics as Moderators
Treatment Received Outcomes
Treatment Contrast
Program Effect
OutcomesTreatment Received
20 Client Characteristics and Context
Characteristics of Implementing Organization
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el Treatment Offered
Take-up
Take-up
Treatment Offered
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el
21
Author’s Suggestions (Client Characteristics as Moderators)
1. Risk• E.g., pre-test, pre-RA income level
2. Readiness• E.g., acknowledgment of problem
(substance abuse)
3. Program Theory• E.g., age (adult education that relies on
technology)
22
3. PROGRAM CONTEXT
Section 2: Proximal Sources of Variation in Program Effects
23
Weight Loss Program
No Program
Potential Outcome
Potential Outcome
ProgramEffectOnNeymanContext 1
Neyman
Weight Loss Program
No Program
Potential Outcome
Potential Outcome
ProgramEffectOnNeymanContext 2
Neyman
Context as a Moderator
Treatment Received
Outcomes
Outcomes
Treatment Received
Treatment Contrast
Program Effect
24 Client Characteristics and Context
Characteristics of Implementing Organization
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el Treatment Offered
Take-up
Take-up
Treatment Offered
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el
25
Section 3
Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity
26
FROM TREATMENT OFFERED TO TREATMENT RECEIVED: CLIENT TAKE-UP
Section 3: Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity
Client Take-Up
Treatment Received
Outcomes
Outcomes
Treatment Received
Treatment Contrast
Program Effect
27 Client Characteristics and Context
Characteristics of Implementing Organization
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el Treatment Offered
Take-up
Take-up
Treatment Offered
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el
28
Author’s Suggestions (Strategies to Increase Client Take-Up)
1. Incentives / Inducements • ASAP’s MetroCard tied to attending advising
sessions
2. Requirements / Mandates • Opening Doors required student success course
3. Remove Barriers• H&R Block’s FAFSA simplification• 401(k) saving plans “opt-out”
4. Improved Outreach• High school to college transition, summer
outreach
29
FROM TREATMENT PLANNED TO TREATMENT OFFERED: PROGRAM IMPLEMENTATION
Section 3: Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity
Program Implementation
Treatment Received
Outcomes
Outcomes
Treatment Received
Treatment Contrast
Program Effect
30 Client Characteristics and Context
Characteristics of Implementing Organization
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el Treatment Offered
Take-up
Take-up
Treatment Offered
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el
1
2
3
31
Treatment Plan
Definition: Planned content, quantity, quality & conveyance of treatment services for clients
• The plans for “what, how much, how, and by whom?” of treatment
• Varies in specificity
1
32
Implementation Plan
Definition: Set of instructions for how the treatment plan is to be realized
Worth Considering:• Staff recruitment and selection• Staff training• Staff monitoring and supports
2
33
Implementing Organization(Author’s Suggestions )
1. Strong Leadership
2. Resources / Capacity
3. Supportive Culture / Climate
4. Involvement of Outside Monitor or “Fixer”
3
Treatment Fidelity
Treatment Received
Outcomes
Outcomes
Treatment Received
Treatment Contrast
Program Effect
34 Client Characteristics and Context
Characteristics of Implementing Organization
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el Treatment Offered
Take-up
Take-up
Treatment Offered
Treatment Fidelity
Implementation ProcessImplementation
Plan
Treatment Planned
Prog
ram
Mod
el
35
A Framework for StudyingVariation in Program Effects
Michael J. Weiss Howard S. Bloom Thomas Brock [email protected] [email protected] [email protected]
