2014.05.19 - OECD-ECLAC Workshop_Session 3_Marc GURGAND

Randomized Social Experiments

Marc GurgandParis School of Economics and J-PAL

OECD - May 19, 2014

Randomized experiments in three sentences

Evaluate the impact of social policies on measurable outcomes

Compare treatment and control goup

Randomize status to make the two groups comparable

Randomized experiments in three sentences

Evaluate the impact of social policies on measurable outcomes

Compare treatment and control goup

Randomize status to make the two groups comparable

Influential experiments

Class size : STAR (US 1985)

Welfare : Self-sufficiency project (Canada, 1994)

Local environment : Moving to Opportunity (US, 1994)

Conditional Cash Transfer : Progressa (Mexico, 1997)

Renewal : Development (Esther Duflo), Europe (Fondsd’experiementation jeunesse, Education Endowment Fundation,Trygfonden/Aarhus, ...)

Methodology in two slides

Counterfactual and causal effects

Objective : measure the causal effect of some intervention (ex.class size reduction) on some outcomes (ex. test scores)

Counterfactual

What would have happened (to a given individual)had he/she not been subject to the intervention

Causal impact defined (and measured) by the difference betweenoutcomes in the obseved state and outcomes in the counterfactualstate

Counterfactual and causal effects

Objective : measure the causal effect of some intervention (ex.class size reduction) on some outcomes (ex. test scores)

Counterfactual

What would have happened (to a given individual)had he/she not been subject to the intervention

Causal impact defined (and measured) by the difference betweenoutcomes in the obseved state and outcomes in the counterfactualstate

Counterfactual and causal effects

Objective : measure the causal effect of some intervention (ex.class size reduction) on some outcomes (ex. test scores)

Counterfactual

What would have happened (to a given individual)had he/she not been subject to the intervention

Causal impact defined (and measured) by the difference betweenoutcomes in the obseved state and outcomes in the counterfactualstate

Counterfactual and causal effects

Objective : measure the causal effect of some intervention (ex.class size reduction) on some outcomes (ex. test scores)

Counterfactual

What would have happened (to a given individual)had he/she not been subject to the intervention

Causal impact defined (and measured) by the difference betweenoutcomes in the obseved state and outcomes in the counterfactualstate

Randomized experiments

Objective : build-up counterfactuals (“same individual”)

Treament and control group randomized

The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact

Any systematic difference between the two groups must result formthe intervention

Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely

Randomized experiments

Objective : build-up counterfactuals (“same individual”)

Treament and control group randomized

The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact

Any systematic difference between the two groups must result formthe intervention

Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely

Randomized experiments

Objective : build-up counterfactuals (“same individual”)

Treament and control group randomized

The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact

Any systematic difference between the two groups must result formthe intervention

Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely

Randomized experiments

Objective : build-up counterfactuals (“same individual”)

Treament and control group randomized

The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact

Any systematic difference between the two groups must result formthe intervention

Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely

Randomized experiments

Objective : build-up counterfactuals (“same individual”)

Treament and control group randomized

The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact

Any systematic difference between the two groups must result formthe intervention

Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely

Two examples in labor policy

Example : Counseling the unemployed

Important and costly active policy

Recent trend in subcontracting to private operators

Comparing counseled and not counseled ; or before and afterimplementation of counseling : does not make sense.

Example : Counseling the unemployed

Important and costly active policy

Recent trend in subcontracting to private operators

Comparing counseled and not counseled ; or before and afterimplementation of counseling : does not make sense.

Example : Counseling the unemployed

Important and costly active policy

Recent trend in subcontracting to private operators

Comparing counseled and not counseled ; or before and afterimplementation of counseling : does not make sense.

Example : Counseling the unemployed

Important and costly active policy

Recent trend in subcontracting to private operators

Comparing counseled and not counseled ; or before and afterimplementation of counseling : does not make sense.

Experiment 1

200 PES offices in France, 44,000 unemployed in the target

Upon registration randomized into :

Intensive public service counselingIntensive private provider counselingstandard PES track

Mesure exit towards employment in the three groups

Experiment 1

200 PES offices in France, 44,000 unemployed in the target

Upon registration randomized into :

Intensive public service counselingIntensive private provider counselingstandard PES track

Mesure exit towards employment in the three groups

Experiment 1

200 PES offices in France, 44,000 unemployed in the target

Upon registration randomized into :

Intensive public service counselingIntensive private provider counselingstandard PES track

Mesure exit towards employment in the three groups

Experiment 1

200 PES offices in France, 44,000 unemployed in the target

Upon registration randomized into :

Intensive public service counseling

Intensive private provider counselingstandard PES track

Mesure exit towards employment in the three groups

Experiment 1

200 PES offices in France, 44,000 unemployed in the target

Upon registration randomized into :

Intensive public service counselingIntensive private provider counseling

standard PES track

Mesure exit towards employment in the three groups

Experiment 1

200 PES offices in France, 44,000 unemployed in the target

Upon registration randomized into :

Intensive public service counselingIntensive private provider counselingstandard PES track

Mesure exit towards employment in the three groups

Experiment 1

200 PES offices in France, 44,000 unemployed in the target

Upon registration randomized into :

Intensive public service counselingIntensive private provider counselingstandard PES track

Mesure exit towards employment in the three groups

Intensive Counseling against standard track

After 6 months : entry rate increases from 20% to 30% with PES ;from 20% to 24% with private providers

(source : Behaghel et al. 2014)

Displacement ?

Two theories on overall impact of counseling :

1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker

2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation

The usual random experiment does not answer this question

Displacement ?

Two theories on overall impact of counseling :

1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker

2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation

The usual random experiment does not answer this question

Displacement ?

Two theories on overall impact of counseling :

1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker

2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation

The usual random experiment does not answer this question

Displacement ?

Two theories on overall impact of counseling :

1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker

2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation

The usual random experiment does not answer this question

Displacement ?

Two theories on overall impact of counseling :

1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker

2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation

The usual random experiment does not answer this question

Experiment 2

235 PES offices interpreted as local labor markets

Randomize offices : in some no intensive counseling at all

In others, randomize some unemployed that are offeredintensive counseling

11 000 workers followed during 20 months

Experiment 2

235 PES offices interpreted as local labor markets

Randomize offices : in some no intensive counseling at all

In others, randomize some unemployed that are offeredintensive counseling

11 000 workers followed during 20 months

Experiment 2

235 PES offices interpreted as local labor markets

Randomize offices : in some no intensive counseling at all

In others, randomize some unemployed that are offeredintensive counseling

11 000 workers followed during 20 months

Experiment 2

235 PES offices interpreted as local labor markets

Randomize offices : in some no intensive counseling at all

In others, randomize some unemployed that are offeredintensive counseling

11 000 workers followed during 20 months

Experiment 2

235 PES offices interpreted as local labor markets

Randomize offices : in some no intensive counseling at all

In others, randomize some unemployed that are offeredintensive counseling

11 000 workers followed during 20 months

Exit to jobs within 8 months

results

www.povertyactionlab.org 3

Only 35percentof the treatmentgroupenrolled in theintensive counseling program. This low take-up may have been because nearly half of the participants already had some form of employment when they were assigned to receive counseling. Take-up of intensive counseling services was higher among people who were not working, had less advanced degrees, and were receiving government unemployment benefits.

Individuals in the treatment group foundworkmorequickly than those in the comparison group in thesameareas,butthesegainsdisappearedafter twelvemonths.After eight months, job seekers in the treatment group were 2.5 percentage points more likely to have found long-term employment, an effect that was mostly driven by men. Increases in employment were also driven by job seekers who found fixed-term contracts, and there was no increase in permanent employment contracts. However, all employment gains disappeared after 12 months and individuals in the treatment group were no more likely to be employed than those in the comparison group (figure 2).

abdul latif jameel poverty action lab

Providing intensive job counseling created adisplacement effect, lowering the employment rateamongthecomparisongroupindividualswholivedintreatmentareas.In treatment areas, job seekers who were offered counseling were 2.5 percentage points more likely to have found long-term employment after eight months than their peers who did not receive counseling. However, these peers were 2.1 percentage points less likely to have found long-term employment than job seekers in areas where no one received counseling. The net effect of being offered the program is, therefore, small and insignificant (Figure 3, left panel).

The low overall impact of being offered intensivecounseling is partly due to the relatively low take-upof the program. Since only 35 percent of those offered intensive counseling chose to enroll, the direct benefit on those who took up the program was essentially compensated by the displacement effects on the job seekers who were assigned to the treatment group but did not enroll.

Displacement effectswere strongest in placeswherejobseekerseligiblefortheprogramwerecompetingforasmallerpoolofjobs.In treatment areas with extremely competitive job markets, the employment rate among the comparison group individuals was lower by 7.7 points than the employment rate in areas where no treatment was offered at all (figure 3, right panel).

figure 3: employment rates among treated and untreated job seekers

time since counseling offer

perc

ent

wit

h l

on

g-t

erm

em

plo

ymen

t 60%

55%

50%

45%

40%

35%

30%8 months 12 months 20 months

treatment group

comparison group in treated areas

40%

35%

30%

25%

Pure Comparison G

roup

Comparison G

roup in

Treated Areas

Treatment G

roup

Pure Comparison G

roup

Comparison G

roup in

Treated Areas

Treatment G

roup

competitive job marketsall job markets

Signi�cantly di�erent from the pure comparison group

figure 2: rates of long-term employment over time

(source : Crepon et al. 2013)

Lessons on job counseling

Efficient technology (but no long run effects)

Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)

Evidence in favor of displacement effects : benefit is aroundzero

Strong statements that require RCT to be credible(but additional evidence is needed)

Lessons on job counseling

Efficient technology (but no long run effects)

Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)

Evidence in favor of displacement effects : benefit is aroundzero

Strong statements that require RCT to be credible(but additional evidence is needed)

Lessons on job counseling

Efficient technology (but no long run effects)

Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)

Evidence in favor of displacement effects : benefit is aroundzero

Strong statements that require RCT to be credible(but additional evidence is needed)

Lessons on job counseling

Efficient technology (but no long run effects)

Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)

Evidence in favor of displacement effects : benefit is aroundzero

Strong statements that require RCT to be credible(but additional evidence is needed)

Lessons on job counseling

Efficient technology (but no long run effects)

Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)

Evidence in favor of displacement effects : benefit is aroundzero

Strong statements that require RCT to be credible(but additional evidence is needed)