L A U RA RA L S T O N , E C O N O M I S T , C C S D

FINDING TRUE PROGRAM IMPACTS THROUGH

RANDOMIZATION

SESSION OVERVIEW

1. Background2. What is a randomized experiment?3. Why randomize?4. Key Takeaways

Materials used from MIT Open Courseware http://ocw.mit.edu JPAL Executive Training: Evaluating Social Programs 2011 Chris Blattman “Can swords be turned into ploughshares? Experimental effects of an agricultural program on employment, lawlessness, and armed recruitment”with Jeannie Annan

IMPACT: WHAT IS IT?

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 60

2

4

6

8

10

12

Time (mths before and after)

Pri

mary

Ou

tcom

e

Intervention Impact

Counterfactual

IMPACT: WHAT IS IT?

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 60

2

4

6

8

10

12

14

16

Time (mths before and after)

Pri

mary

Ou

tcom

e

Intervention Impact

Counterfactual

IMPACT: WHAT IS IT?

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 60

0.5

1

1.5

2

2.5

3

3.5

4

Time (mths before and after)

Pri

mary

Ou

tcom

e

Intervention Impact

Counterfactual

HOW TO MEASURE IMPACT?

Impact is defined as the comparison between:

1. The outcome some time after the program has been introduced

2. The outcome at that same point in time had the program not been introduced (the counterfactual)

COUNTERFACTUAL

The counterfactual represents the state of the world that program participants would have experienced in the absence of the program (i.e., had they not participated in the program)

• Problem: counterfactual cannot be observed

• Solution: We need to “mimic” or construct the counterfactual

IMPACT EVALUATION METHODS

Randomized Experiments

• Also known as:• Random Assignment Studies• Randomized Field Trials• Social Experiments• Randomized Controlled Trials (RCTs)• Randomized Controlled Experiments

IMPACT EVALUATION METHODS

Non- or Quasi –Experimental Methods

• Includes:• Pre-Post• Simple Difference• Difference-in-Difference• Multivariate Regression• Statistical Matching• Instrumental Variables• Regression Discontinuity

• More on these tomorrow

2. What is a randomized experiment?

SESSION OVERVIEW

THE BASICS

Start with simple case:• Take a sample of program applicants

• Randomly assign them to either:• Treatment Group – is offered treatment• Control Group – not allowed to receive treatment

(during the evaluation period)• Note:

• Randomization does not mean denying people the benefits of the program

• Usually existing constraints in project roll-out allow randomization• Randomization is often the fairest way to allocate treatment

KEY ADVANTAGE OF EXPERIMENTS

• Because members of the groups (treatment and control) are randomly selected, they do not systematically differ at the start of the experiment,

• Any difference that subsequently arises between them can be attributed to the program rather than to other factors.

EXAMPLE: “WOMEN AS POLICYMAKERS” TREATMENT VS. CONTROL VILLAGES AT BASELINE

Standard Errors in parentheses. Statistics for West Bengal, India. Source: Chattopadhyay and Duflo (2004)

RANDOMIZATION EXAMPLE CORE INTERVENTION: CASH PLUS SKILLS TRAINING

IMPLEMENTED BY AVSI UGANDA 2009-11

Target: 15 poorest, most marginalized rural women in villages of 80 - 300 households; Nominated by community and screened by NGOAge 27, Work 15 hours/week, earn <$10/month in cashResearch questions: What limits the growth of self-employment and income among the poorest and marginalized? Does more work and income “empower” them?

SAMPLE: 120 VILLAGES IN 6 SUBCOUNTIESS E L E C T E D B A S E D O N B E I N G U N D E R S E R V E D

V I L L A G E S R E P R E S E N T 2 5 % O F S U B C O U N T Y P O P U L A T I O N

BUCKET RANDOMIZATION BY VILLAGE TO TREATMENT OR WAITLIST

F I R S T 6 0 V I L L A G E S R E C E I V E I M M E D I AT E T R E AT M E N T ( P H A S E 1 )6 0 R E C E I V E D E L AY E D T R E AT M E N T 1 8 M O N T H S L AT E R ( P H A S E 2 )

SOME VARIATIONS ON THE BASICS

• Assigning to multiple treatment groups

• Assigning of units other than individuals or households:• Health Centers• Schools• Local Governments• Villages

KEY STEPS IN CONDUCTING AN EXPERIMENT

1. Design the study carefully: what is the objective of your impact evaluation? What do you most want to test?

2. Randomly assign people to treatment or control

3. Collect baseline data

4. Verify that assignment looks random

5. Monitor process so that integrity of experiment is not compromised

KEY STEPS IN CONDUCTING AN EXPERIMENT (CONT.)

1. Collect follow-up data for both the treatment and control groups

2. Estimate program impacts by comparing mean outcomes of treatment group vs. mean outcomes of control group

3. Assess whether program impacts are statistically significant and practically (size) significant.

3. Why randomize?

SESSION OVERVIEW

WHY RANDOMIZE? – CONCEPTUAL ARGUMENT

If properly designed and conducted, randomized experiments provide the most credible method to estimate the impact of a program

WHY “MOST CREDIBLE”?

Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment,

Any differences that subsequently arises between them can be attributed to the program rather than to other factors.

EXAMPLE – CAN EMPLOYMENT PROGRAMS REDUCE LAWLESSNESS AND REBELLION? A FIELD EXPERIMENT WITH

HIGH-RISK YOUTH IN LIBERIA (BLATTMAN 2014)

Knowledge Gaps:1. Little experimental evidence of employment or

incomes on crime or violence (Freeman 1999, Blattman and Miguel 2010)

• Exceptions are with low-risk populations (Blattman et al 2013, 2014)

• US experiments test adolescent schooling, neighborhoods

2. Few experimental job programs generate jobs• Demobilization and reintegration (Kingma and Muggah 2009)

• Vocational and business training (Card et al 2010, Attanasio et al 2011, McKenzie & Woodruff 2012)

• Cash to microenterprises target the already employed (de Mel et al 2008, Fafchamps et al 2012)

3. Where there is evidence, theoretical mechanism unclear• Adult education: Opportunity cost, socialization, or peer effects?• Income-conflict correlation: Opportunity cost or grievance?

INTERVENTIONOffer high-risk young people in hotspots:

1. 4-month residential training program• Highly practice-based agricultural skills

2. “Life skills” and counseling• Handling conflict, dealing with trauma and

PTSD, career counseling• Mentoring by former ex-combatants

3. Assistance returning to a community• Leader permission, land access, transport

4. Package of agricultural inputs• $125 in tools and materials in two stages• Choice between vegetable farming,

animals• $50 cash (Sinoe site only)

AIMS1. Increase farm

incomes and activity

2. Shift occupational incentives away from illicit resource extraction

3. Socialize into peacetime, non-violent life

4. Reduce risk of mercenary recruitment

RANDOMIZED EXPERIMENT

Suppose we evaluated this program using a randomized experiment

Question 1: What would this entail? How would we do it?

Question 2: What would be the advantage of using this method to evaluate the impact of the program?

METHODS TO ESTIMATE IMPACTS

Let’s look at different ways of estimating the impacts using the data from young people who were enrolled in this program:

1. Pre – Post (Before vs. After)2. Simple difference3. Difference-in-difference4. Other non-experimental methods5. Randomized Experiment

PRE-POST (BEFORE VS. AFTER)

• Look at average change in:• Involvement in crime (drug selling, illicit extraction,

stealing)• Hours per week worked in legal activities (raising

animals, farming)

• Question: under what conditions can these differences be interpreted as an impact of the program?

Crime Rate Hours worked

Average Pre Program 47% 31

Average Post Program 41% 37

Difference -6% +6

WHAT WOULD HAVE HAPPENED IN ABSENCE OF THE PROGRAM?

Pre Post28

29

30

31

32

33

34

35

36

37

38

Hou

rs w

ork

ed

per

week

Impact = 6 hrs ?

SIMPLE DIFFERENCE

Compare crime rates of…

Young men who got program with those that didn’t

SIMPLE DIFFERENCE

• Look at average difference in:• Involvement in crime (drug selling, illicit extraction,

stealing)• Hours per week worked in legal activities (raising

animals, farming)

• Question: under what conditions can these differences be interpreted as an impact of the program?

Crime Rate Hours worked

Average Outside of Program 47% 28

Average Inside of Program 41% 37

Difference -6% +9

WHAT WOULD HAVE HAPPENED IN ABSENCE OF THE PROGRAM?

Impact= 9 hrs ?

DIFFERENCE-IN-DIFFERENCES(MORE ON THIS TOMORROW)

Compare decrease in crime rates of…

Young men who got program with those that didn’t

OTHER METHODS

There are more sophisticated non-experimental methods to estimate program impacts:• Regression• Matching• Instrumental Variables• Regression DiscontinuityBut all these methods rely on being able to mimic the counterfactual under certain assumptionsProblem: Assumptions are not testable

IMPACT OF PROGRAM - SUMMARY

Method Impact on Crime Rates

Impact on Hrs worked

1. Pre-post -6% +6*

2. Simple Difference -10%* +9

3. Difference-in-Difference -8% +11*

4. Regression -4% +5

5. Randomized Experiment -5%* +5.5*

*: Statistically significant at the 5% level

Bottom Line: which method we use matters!

4. Key Takeaways

SESSION OVERVIEW

KEY TAKEAWAY #1

The single best way to evaluate the true average impact of a program is by

randomizing treatment

KEY TAKEAWAY #2

Randomization is more flexible than you think:

• It does not require withholding of benefits

• It can take advantage of necessary staggered roll-out

• It can test different reforms or packages across groups at the same time

EXAMPLE OF ROLL-OUT RANDOMIZATION

1800 clients in 120

villages

60 villages:Training, grant and follow-up

30 villages:Intensify

group formation and cooperation

30 villages:No added services

“Phase 1”

300 clients:No follow-up

visits

300 clients:“Accountability & advice”: 3-5 follow-up visits

300 clients:Accountability: 1-2 follow-up

visits

60 villages:Training and

grant 18 months

later

“Phase 2”

KEY TAKEAWAY #3

It is more ethical to test programs rigorously before universally implementing them than it is to use scarce public resources to implement a universal program with uncertain benefits.

Thank you !