How do impact evaluations improve development operations and policy?David K. Evans, World BankAfrica Chief Economist’s Office
June 1, 2016 | 3ie/IFAD [email protected] | @tukopamoja
2
How do impact evaluations affect operations and policy?Own project• Clarify thinking about the
theory of change• Improve quality of data
systems• Improve efficiency of projects• Shifts in project design• Influence scale-up of projects
Other projects• Design of new projects• Choosing between designs• Culture of evidence• Contribute to a body of
knowledge that changes how we do business
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
3
I’m going to tell you some stories (and show some analysis).
But how do we measure impact?
Isn’t impact evaluation about careful causal inference? Where’s the counterfactual?
Image source: http://imperialcentrepaintingstudio.blogspot.com/2012/02/reading-comprehension-teach-your-child.html
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
4
Measuring the impact of impact evaluationSecond cousin to the event study
• Before-after is a poor evaluation method– Because usually many other
things are changing• But useful when no other
credible cause
Embrace the ambiguity
• The best impacts from impact evaluation can come when the resulting decisions are completely internalized by government (Goldstein 2016)
• Short of an RCT of RCTs, embrace the ambiguity.
Jensen (2007)
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
5
Indicators for IE impact3ie• Closing knowledge or evidence
gaps?• Changes in policy or programs?• Key stakeholders and key
influencers?• Plan to engage with them at
each stage of the IE?• Cleary theory of change for
policy influence?
DIME (World Bank)• Capacity building
– IE results discussed with client– Training provided for data analysis
• Quality of data– Availability of data– Improved administrative data
• Quality of policy decisions– Used to motivate scale-up/scale-down of
policy– Adopted delivery mechanism based on IE
results– Baseline informed design or
implementation
Source: Legovini 2014Source: 3ie 2012
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
6
• Impact evaluation helps clarify thinking on exactly how one gets from inputs to impacts
• By focusing on questions of – Causality– Measurement
• Impact evaluation improves theories of change
Own projectClarify thinking about the theory of change
Figure sources: Imas & Rist 2009
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
7
Own projectImprove quality of data systems• Kenya Judicial Performance IE
– Impact of incentives for greater efficiency
– Co-developed tool for measuring efficiency of courts: How quickly do they get their cases out
– Tool was adopted for all courts as official monitoring tool
• Kenya Patient Safety IE– Impact of inspections of health
facilities– Reform to regulatory framework of
inspection system for minimum patient safety.
– Design of quality of care measuresSources: World Bank 2013; i2i n.d. (1); i2i n.d. (2)
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
8
• Sample of 100 impact evaluations, 1000+ projects between 2005 and 2011
• Include time, sector, country, and project leader effects
• Significant increase in projects carrying out their activities
Own projectImprove efficiency of projects
Source: Legovini et al. 2015
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
9
• Notably, this increase in disbursements happens after impact evaluation begins
• Can’t completely control for selection into project
• BUT suggestive that IEs help projects roll out and that they don’t slow projects down (as commonly claimed)
Own projectImprove efficiency of projects
Source: Legovini et al. 2015
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
10
Own projectShifts in project design Ethiopian Nile Irrigation and Drainage Project
• Baseline survey in Ethiopia• Women have
– Lower access to extension– Greater time poverty– Much lower agricultural value-
added for women than men• Result: Pilot strategies in
Farmer’s Initiative Fund, designed for female farmers
Source: World Bank 2014
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
11
Own projectInfluence scale-up of projectsFinancial Literacy in Brazil• Pilot in high schools• Modest but positive, significant
impacts• Scale-up
– High schools around the country– Elementary schools
Other example: Land titling in Rwanda
Conditional Cash Transfers• Mexico
– Initial program showed positive results
– Expansion to urban– Survival across administrations
• PROGRESA Oportunidades Prospera
• Tanzania– Pilot showed positive results– Key evidence informing scale-up
Source: Dhaliwal & Tulloch 2012; Mkelemi & Evans 2016Source: Bruhn et al. 2013; Ali et al. 2011
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
12
Other projectsDesign of new projectsProjects reports that IEs influence design
• Education Modernization Project in Kazakhstan– Influenced by synthesis of
education impact evaluations• Adolescent Girls Empowerment
Project in India– Influenced by IEs of programs to
empower adolescent girls in Liberia and Uganda
Citations of IEs in World Bank documents ↑ 6x
05 06 07 08 09 10 11 12 0
5,000
10,000
15,000
20,000
25,000
3,691
21,271
Fiscal Year
Tota
l IE
Cita
tions
Source: Legovini 2014Source: Gender Innovation Lab 2015 Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
13
• Zambia Essential Drugs Public Pilot Program• Consistent problem with stockouts of essential malaria drugs• Randomized impact evaluation to test two different models • A: District planner• B: Direct orders from clinic to central medical stores
Other example:Early child development in the Gambia
Other projectsChoosing between designs
Source for Zambia: Vledder et al. 2015; World Bank 2010
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
Source for the Gambia: Blimpo & Carneiro 2014
14
Other projectsCulture of evidenceMexico• Progresa/Oportunidades 1997-
2000• 2000 Congress’ Degree: Annual
external evaluations for projects
• 2004 Social Development Law: National Council for Evaluation of Social Policy (CONEVAL)
Tanzania• Two donor-initiated impact evaluations
under TASAF-2– Community-based conditional cash
transfers– Vulnerable groups
• Three client-initiated impact evaluations under TASAF-3– Scaled-up cash transfers– Cash transfers on women’s empowerment– Cash transfers on transition to adulthood
• False dichotomy: Donor-initiated vs client-initiated. But a shift along the spectrum.
Source: Hernández Licona 2008
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
15
Other projectsContribute to a body of knowledge that changes how we do developmentSmall questions or big questions?• Should we give away
insecticide treated nets or charge for them?
• Higher prices – lower use– don’t improve targeting– don‘t increase use
• Between 2000 and 2015: “450 million cases of malaria and 4 million deaths averted from ITN distribution”
Many small studies add up to big answers
Pricing experiments around the world
Source: Dupas & Miguel 2016Source: Glennerster 2016
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
16
3ie. “Policy Influence Plan.” 2012. http://evaluationvigie.fr/wp-content/uploads/2014/05/2012_PIP_template.docx Ali, Daniel, Klaus Deininger, and Markus Goldstein. “Environmental and gender impacts of land tenure regularization in Africa: pilot evidence from Rwanda.” World Bank Policy Research Working Paper 5765. 2011. http://documents.worldbank.org/curated/en/2011/08/14833489/environmental-gender-impacts-land-tenure-regularization-africa-pilot-evidence-rwanda Blimpo, Moussa, and Pedro Carneiro. “Early childhood development: Some preliminary findings to guide future actions.” PowerPoint Presentation. 2014. Bruhn, Miriam, Luciana de Souza Leao, Arianna Legovini, Rogelio Marchetti, and Bilal Zia. “The impact of high school financial education: experimental evidence from Brazil.” World Bank Policy Research Working Paper 6723. 2013. http://documents.worldbank.org/curated/en/2013/12/18640673/impact-high-school-financial-education-experimental-evidence-brazil Dhaliwal, Iqbal, and Caitlin Tulloch. “From research to policy: using evidence from impact evaluations to inform development policy.” 2012. https://www.povertyactionlab.org/sites/default/files/publications/With%20JDE%20Revisions%20and%20Footnotes%202012.09.03.pdf Dupas & Miguel. “Impacts and Determinants of Health Levels in Low-Income Countries.“ Prepared for the Handbook of Field Experiments. 2016. http://emiguel.econ.berkeley.edu/assets/miguel_research/73/Health_DupasMiguel_Handbook.pdf Gender Innovation Lab. Africa Gender Innovation Lab (GIL) Influence. November 2015. Glennerster, Rachel. “Not so small.” Running Randomized Evaluations blog. May 27, 2016. http://runningres.com/blog/2016/5/27/not-so-smallGoldstein, Markus. Conversation with Dave Evans on a bus. Rome, Italy. May 31, 2016.Hernandez Licona, Gonzalo. “Impact evaluation in Mexico.” January 2008. http://siteresources.worldbank.org/INTISPMA/Resources/Training-Events-and-Materials/449365-1199828589096/GonzaloHernandezLicona.pdf
References - 1
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y
17
i2i. Changing Policy through Research: DIME IE Portfolio in Kenya. n.d.i2i. Selected Examples of DFID’s Contribution to Evidence-Based Policy through i2i. n.d.Imas, Lnda G. Morris, and Ray C. Rist. The Road to Results: Designing and Conducting Effective Development Evaluations. World Bank. 2009. https://openknowledge.worldbank.org/bitstream/handle/10986/2699/52678.pdf Jensen, Robert. “The Digital Provide: Information (Technology), Market Performance, and Welfare in the South Indian Fisheries Sector.” Quarterly Journal of Economics 122(3), August 2007. http://www.jstor.org/stable/pdf/25098864.pdf Legovini, Arianna. “The development impact of impact evaluation.” World Bank. 2014. http://www.worldbank.org/content/dam/Worldbank/Feature%20Story/japan/pdf/event/2014/120514_Arianna_Legovini.pdf Legovini, Arianna, Vincenzo di Maro, Caio Piza. “Impact evaluation helps deliver development projects.” World Bank Policy Research Working Paper 7157. 2015. http://documents.worldbank.org/curated/en/2015/01/23173058/impact-evaluation-helps-deliver-development-projects Mkelemi, Godfrey, and David K. Evans. “The Challenge and Success of Impact Evaluation for Tanzania Community-Based Conditional Cash Transfer Pilot.” Conference presentation, IFAD/3ie. May 31, 2016. http://www.slideshare.net/DavidEvans198/the-challenge-and-success-of-impact-evaluation-for-tanzania-communitybased-conditional-cash-transfer-pilot Prottas, Chris (@ChrisProttas). “I understand why, but real mental disconnect w anecdotes used to justify impact of impact evals.” May 27, 2016, 3:10am. Tweet. https://twitter.com/ChrisProttas/status/736137568954294272?lang=enVledder, Monique, Jed Friedman, Mirja Sjoblom, Thomas Brown, and Prashant Yadiv. “Optimal Supply Chain Structure for Distributing Essential Drugs in Low Income Countries: Results from a Randomized Experiment.” Ross School of Business working paper 1269. March 2015. https://deepblue.lib.umich.edu/bitstream/handle/2027.42/110800/1269_Yadav.pdf World Bank. From Evidence to Policy: The Challenge of Ensuring Adequate Stocks of Essential Drugs in Rural Health Clinics. November 2010. World Bank. “Improving patient safety in Kenya.” November 6, 2013. PowerPoint Presentation. https://www.wbginvestmentclimate.org/results/upload/Kenya-PS-BBL_November-6-2013.pdfWorld Bank. Africa Impact Evaluation Update: Agriculture and Land. March 2014.
References - 2
Evan
s |
Wor
ld B
ank|
im
pact
eva
luat
ions
, op
erat
ions
& p
olic
y