Upload
martha-wilkinson
View
215
Download
1
Tags:
Embed Size (px)
Citation preview
There and Back Again:An Impact Evaluator’s Tale
Paul GertlerUniversity of California, Berkeley
July, 2015
Answer these questions
What is impact evaluation?
What makes a good impact evaluation?
Why is impact evaluation valuable?
1
2
3
Answer these questions
What is impact evaluation?
What makes a good impact evaluation?
Why is impact evaluation valuable?
1
2
3
“Impact Evaluation
An assessment of the causal effect of a project , program or policy on beneficiary outcomes.
Estimates the change in outcomes attributable to the intervention.
Impact Evaluation AnswersWhat is effect of information on hand
washing, hygiene and child health?
Does paying primary health care workers for performance improve access & quality?Do early childhood education programs improve subsequent learning?Does expansion of urban sewares reduce open defecation & improve health?
Impact Evaluation AnswersWhat was the effect of the program
on outcomes?
How much better off are the beneficiaries because of the program/policy?
How would outcomes change if changed program design?
Is the program cost-effective?Traditional
M&E cannot
answer these.
Answer these questions
What is impact evaluation?
What makes a good impact evaluation?
Why is impact evaluation valuable?
1
2
3
How are Impact Evaluations Useful?
To inform program design
As an input to funding decisions
As a means of influencing ideas
How are Impact Evaluations Useful?
As an input to funding decisions
PROGRESA / Oportunidades (Mexico)
Inform Program Design Influence IdeasInput to Funding
Decisions
Evaluation shows significant impacts on education and health → scaled up & adopted by new presidential
administration.
• Households paid to send children to school and regular health checkups
• First evaluation of large-scale CCT program
Families enrolled in Oportunidades
Roof Rain Water Cisterns -- Brazil
Inform Program Design Influence IdeasInput to Funding
Decisions
• Northeastern states very dry
• Collect rain from roofs during rainy season
• Store in cistern
Evaluation: • No significant impact on health
• Families value cisterns• Value of house & Depression
Low Cost Pre-Schools (Mozambique)
Inform Program Design Influence IdeasInput to Funding
Decisions
• Pre-schools constructed in 30 villages (Save the Children)
• Volunteer community members trained to staff schools
Evaluation shows significant impacts on health and education indicators → scaled up to cover 600 rural
communities.7.38
3.15
26.2117.82
5.375.55 2.86
14.01 11.483.60
Preschool Control
How are Impact Evaluations Useful?
To inform program design
Improving Access to Essential Medicines (Zambia)
Influence IdeasInput to Funding
DecisionsInform Program Design
SP
ACT Adu
lt
ACT Pe
diat
ric0%
25%
50%
75%58%
43%34%
46%
22%30%
BaselineEndline
Clinics receive monthly supplies from district
stores
SP
ACT Adu
lt
ACT Pe
diat
ric0%
25%
50%
75%
52% 48% 43%
16%6% 12%
BaselineEndline
Clinics receive supplies direct from central
stores
Stock-outs are reduced under both distribution systems:
Direct distribution more cost-effective → replicated across Zambia (World Bank, 2010)
$4.20 / day of averted stock-out
$14.50 / day of averted stock-out
Targeting the Poor (Indonesia)
Influence IdeasInput to Funding
DecisionsInform Program Design
• Randomized field experiments evaluating accuracy of 3 methods for targeting the poor:
• Community-based
• Proxy means test (PMT)
• Self-targeting
PMT & community-based found to be most accurate → findings used by Indonesian government to build registry of
poorest 40% (World Bank, 2012)
How are Impact Evaluations Useful?
As a means of influencing ideas
Conditional Cash Transfer (CCT) ProgramsInput to Funding
DecisionsInform Program Design Influence Ideas
Countries implementing CCT programs in
1997
Conditional Cash Transfer (CCT) ProgramsInput to Funding
DecisionsInform Program Design Influence Ideas
Countries implementing CCT programs in
2011
Results-Based Financing (Rwanda)
Input to Funding Decisions
Inform Program Design Influence Ideas
• Health clinics paid to immunize children and encourage women to give birth in a clinic
• Treatment group received bonuses according to performance;
• Control group received grant regardless of performance
Evaluation shows RBF has significant impact on prenatal care → results inspire RBF designs in other
countries, including Nigeria (Basingra et al, 2010)
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
Treatment (PBF) Facili-ties
Control Facilities
Sta
nd
ard
ized
Pre
nata
lEff
ort
Sco
re
Baseline (2006) Follow-up (2008)
15% Standard deviation increase
due to RBF
Treatment (RBF) Facilities
Control Facilities
Results-Based Financing (Argentina)Input to Funding
DecisionsInform Program Design Influence Ideas
• Plan Nacer used to provide additional health care coverage to women and young children
• Federal resources allocated to provinces based on enrollment and health results achieved by each province
Evaluation shows RBF has significant impact on prenatal care, infant/ maternal mortality → results inspire
RBF program in Dominican Republic & Peru
Answer these questions
Why is evaluation valuable?
What makes a good impact evaluation?
1
2
How to assess impact
What is beneficiary’s diarrhea incidence of diarrhea in last 3-days with program compared to without program?
Compare same individual with & without programs at same point in time
Formally, program impact is:
α = (Y | P=1) - (Y | P=0)
e.g. How much does an safe water intervention reduce diarrhea?
Solving the evaluation problem
Estimated impact is difference between treated observation and counterfactual.
Counterfactual: what would have happened without the program.
Use Control group to estimate counterfactual.
Never observe same individual with and without program at same point in time.
Counterfactual is key to impact evaluation.
Possible SolutionsNeed to guarantee comparability of treatment and control groups.
ONLY remaining difference is intervention.
Consider:o Experimental design/randomization o Quasi-experiments
Counterfactual Criteria
Treated & Counterfactual(1) Have identical characteristics,(2) Except for benefiting from the intervention.
No other reason for differences in outcomes of treated and counterfactual.
Only reason for the difference in outcomes is due to the intervention.
2 Counterfeit CounterfactualsBefore and After
Those not enrolledo Those who choose not to
enroll in the programo Those who were not
offered the program
Same individual before the treatment
Problem: Cannot
completely know
why the treated
are treated and
the others not.
What's wrong?Selection bias: People choose to participate for specific reasons
1
2
3
Many times reasons are related to the outcome of interest
Cannot separately identify impact of the program from these other factors/reasons
Need to know…
All the reasons why someone gets the program and others not.
All the reasons why individuals are in the treatment versus control
group.If reasons correlated w/ outcome cannot identify/separate program impact from
other explanations of differences in outcomes.
Possible SolutionsNeed to guarantee comparability of treatment and control groups.
ONLY remaining difference is intervention.
In this seminar we will consider:o Experimental design/randomization o Quasi-experiments (Regression
Discontinuity, Double differences)o Instrumental Variables.
These solutions all involve…Knowing how the data are
generated.Randomizationo Give all equal chance of being in control
or treatment groups o Guarantees that all factors/characteristics
will be on average equal btw groupso Only difference is the intervention
If not, need transparent & observable criteria for who is offered program.
Working Smarter in IE
• Be strategic in selecting programs to evaluate
• Don’t assume costly data collection is always necessary
• Use IE to maximize program efficiency, not just impactImpact evaluation as an operational research tool
Use administrative data when possible
Use IE’s to fill knowledge gaps & assess alternatives for key programs
Ensuring the Impact of Impact Evaluations
Engage early. Engage Often.
Work locally. Think globally.
• Tale of 2 Evaluations
• Involve stakeholders at every stage
• Prospective – IE part of design
• Foster relationships on the ground with decision-makers
• Results inform decisions beyond the borders of country studied
Water Privatization -- Argentina
o Municipal water & sanitation
o Public versus Private management
o Evaluation:o Increase in water
qualityo Increase in access
by pooro Large reductions in
child mortality
Figure 4: Evolution of Mortality Rates for Municipalities with Privatized vs. Non-Privatized Water Services
3.5
4
4.5
5
5.5
6
6.5
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999
Year
Mo
rta
lity
Ra
tes
Mortality Rates, Non-privatized MunicipalitiesMortality Rates, Privatized Municipalities
No Influence on policy: Academic study without stakeholders involvement
Improving Housing in Urban Slums(Piso Firme)
Inform Program Design Influence IdeasInput to Funding
Decisions
Replacing Dirt Floors with Cement Floors (Piso Firme)
• Evaluation shows significant impacts on reducing diarrhea, parasitic infections & improving cognitive development
• Scaled up to 3 million householdso Moving families to new housing
developments (Tu Casa)o No Impact – canceled program
Ensuring the Impact of Impact Evaluations
Engage early. Engage Often.
Work locally. Think globally.
• Tale of 2 Evaluations
• Involve stakeholders at every stage
• Prospective – IE part of design
• Foster relationships on the ground with decision-makers
• Results inform decisions beyond the borders of country studied
Messageso IE Useful for policy
o Resource allocationo Benefit design
o Influence global ideas
o What to evaluate
o High cost programs with large #’s beneficiaries
o Little existing knowledge
o Start early – prospective
o Build control groups into rollout
o Work locally think globally
Figure 4: Evolution of Mortality Rates for Municipalities with Privatized vs. Non-Privatized Water Services
3.5
4
4.5
5
5.5
6
6.5
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999
Year
Mo
rta
lity
Ra
tes
Mortality Rates, Non-privatized MunicipalitiesMortality Rates, Privatized Municipalities