#ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 ARIANNA LEGOVINI...
Preview:
Citation preview
- Slide 1
- #ieGovern Impact Evaluation Workshop Istanbul, Turkey January
27-30, 2015 ARIANNA LEGOVINI DEVELOPMENT IMPACT EVALUATION IMPACT
EVALUATION FOR REAL TIME POLICY MAKING
- Slide 2
- WHAT IS THE VALUE OF A POLICY? measuring HOW TO GET THE MOST
OUT OF A POLICY? reaching 2
- Slide 3
- Policy objective Invest in what gets us results 3 Invest in the
cause to get an effect
- Slide 4
- Evaluation objective Identify cause - effect 4 enable good
policy decisions
- Slide 5
- 55 WHY IS UNDERSTANDING CAUSALITY SO IMPORTANT? I THINK I WILL
STOP USING UMBRELLAS WHEN I USE MY UMBRELLA MY SHOES GET WET It is
easy to confuse correlation with causation
- Slide 6
- 6 Invest in the cause to get results Mentorship interacts with
quality of mentor to change attitudes and value for youth to act
upon their life-skill training and change their behavior (Bushway
and Reuter, 2002, 2007) Charismatic mentor + behavioral therapy
Attitudes and values Vocational training EMPLOYMENT More motivated
youth
- Slide 7
- Evaluation Narrow down causality by identifying a
counterfactual and compare 7 WHAT HAPPENED WHAT WOULD HAVE HAPPENED
TO
- Slide 8
- 8 What is counterfactual analysis? Compare same individual with
& without intervention at the same point in time Missing data
Compare statistically identical groups of individuals with &
without intervention at the same point in time Comparable data
- Slide 9
- 9 Counterfactual criteria Treated & control groups Have
identical initial average characteristics (observed and unobserved)
So that, the only difference is the treatment Therefore the only
reason for the difference in outcomes is due to the treatment
- Slide 10
- 10 USE SCIENTIFIC EXPERIMENTS TO GUIDE POLICY DECISIONS
- Slide 11
- 1111 Get the delivery right Subsidy for malaria treatment $350
million Sealing packages saved 16,600 under five + 2,200 adult
lives in 5 years Problem: poor public sector distribution of
medicines Solution: found system that reduced stockouts by 83%
Source: Vledder, M.; Friedman, J; Yadav, P., and Sjoblom M. RANDOM
ASSIGNMENT OF HEALTH DISTRICTS TO 3 DIFFERENT DISTRIBUTIO N
SYSTEMS
- Slide 12
- 12 Price a CURE $6
- Slide 13
- 1313 Get people to do it Source: Goldstein, Thirumurthy, and
Zivin HIV treatment $ 2.1 billion Reminders increase adherence to
treatment by 35% Problem IE scalable solution RANDOM ASSIGNMENT OF
DAILY AND WEEKLY REMINDERS TO HIV PATIENTS
- Slide 14
- 14 Price Faiths LIFE $7
- Slide 15
- 15 TRANSFORMATIONAL (not marginal) IMPACT
- Slide 16
- 16 Public recognition DOUBLES performance [Ashraf et al.,
2011]
- Slide 17
- What does dime do? 17 Transform development policy 1.Run
experiments to inform decisions 2.Build agencies capacity to do
this systematically 3.Draw lessons and share them face-to-face to
global audiences
- Slide 18
- 175 IEs in 47 Countries Distribution of dime s IEs by GP and
CCSA (number of IEs) 18
- Slide 19
- 1919 dime in action Inform policy design Guide mid- course
corrections Inform adoption and scale-up Systematic use of evidence
Train & apply Learn by doing Apply knowledge Capacity building
IE Product IE DESIGN IE IMPLEMENTATION IE DISSEMINATION
- Slide 20
- Clear roadmap 20 Clearer roadmap WhatTo whomWhenWhere
- Slide 21
- 2121 UNDERSTAND THE THEORY OF CHANGE Inputs Financial and human
resources Activities Actions to convert inputs into outputs Outputs
Tangible products Intermediate outcomes Effective access Use
Compliance Performance Service quality Final Outcomes Health
outcomes Lives saved Productivity WHAT (PROGRAM)HOW TO (RESPONSE)
& RESULTS
- Slide 22
- awareness disposition Take-up ADOPTION SKILLS WILLINGNESS
motivation participation Accountability quality sustainability 22
BREAK IT DOWN
- Slide 23
- Policy questions in dime s portfolio (% of questions) 23
- Slide 24
- Better capacity 24 Better capacity Research team Full time
field coordinator Training More and better data
- Slide 25
- dime results indicators 25
- Slide 26
- Workshops increase participantsknowledge (pre/post test)
- Slide 27
- Impact of Workshop on (External Double Blind) Technical Ratings
of IE proposals 27
- Slide 28
- dime results indicators 28
- Slide 29
- More effort into implementation 29 Motivation to perform Goal:
this is what we are trying to achieve Control: this is how we want
to do it and BTW somebody measures (and cares)
- Slide 30
- Better decisions 30
- Slide 31
- dime results indicators 31
- Slide 32
- IE increases disbursements IE helps projects stick to plans 32
Does IE help project implementation? 1300 World Bank projects - 100
impact evaluations (2005-2011) Legovini, Di Maro, Piza (2015) It
does!
- Slide 33
- IE increases disbursements & closes implementation gap
statistically significant at 1% level 33
- Slide 34
- AT WHAT COST? SOME SIMPLE COST-BENEFIT OF IMPACT EVALUATION
34
- Slide 35
- Is the benefit worth the cost? Zambia best supply chain IE
solution: 83% reduction in stock out 21-25% reduction in malaria
mortality Pilot: Benefit ($7.4M) / Cost ($3.8M) = 2 Scaled up:
Benefit ($60.7M) / Cost ($12.4M) = 4.9 Malawi commitment saving
account IE solution: 22% increase in farmer production value Pilot:
Benefit ($0.7M)/Cost ($0.5M) = 1.4 If scaled: Benefit ($15.6M) /
Cost ($1.7M) = 9.2 35 IE PAYS FOR ITSELF A PUBLIC GOOD WORTH
FINANCING
- Slide 36
- dime & i2i IE has real impacts on developmental
effectiveness and project delivery High return on investment
Spillover effects i2i opportunity to increase scale and scope Open
invitation to donors and governments 36 partners CRIME LAB
- Slide 37
- Human Development Education Health Social Protection i2i
pillars and thematic areas 37 Shared Prosperity Finance, Private
Sector and Jobs Agriculture Infrastructure Governance Public Sector
Governance Justice Local Development Governance Public Sector
Governance Justice Local Development Climate Change Energy
Environment Natural Resource Management Agriculture Transport
Fragility & Conflict Reintegration Governance Gender-based
Violence Urban crime & violence Gender Human Capital Economic
Opportunities Voice/Agency 5 123 64
- Slide 38
- Discussion Which development interventions are effective and
worth scaling up? Which promising mechanisms are worth testing
across different sectors? No one size fits all Importance of
building evidence-based institutions 38
- Slide 39
- #ieGovern Impact Evaluation Workshop Istanbul, Turkey January
27-30, 2015 Thank You! facebook.com/ieKnow #impacteval
blogs.worldbank.org/impactevaluations
microdata.worldbank.org/index.php/catalog/ impact_evaluation
http://dime.worldbank.org WEB