Upload
david-evans
View
461
Download
0
Embed Size (px)
Citation preview
$
Manage Successful Impact Evaluations
FIELD COORDINATOR WORKSHOP
6-10 JUNE 2016WASHINGTON, DC
How Much Does It Cost To Get That Impact?
Measuring Cost Effectiveness8 June 2016
David K. Evans, Senior Economist, Office of the Chief Economist for Africa
The ChallengeIncreasing Evidence of Impact
Lego
vini
et a
l. 20
12
We have all this amazing evidence of interventions that
will work in your country!
Yes, but how much is this going to cost me?
Not just you, man! How much is it going to cost me?
Government MinisterEntrepreneur
IE Expert #1
1. Summarize a complex program in terms of a return on investment
2. Permit comparison across contexts & times
Combining cost & impact data has two goals
Dhal
iwal
et a
l. 20
11
Intuitive appeal of cost-effectiveness estimates
JPAL
201
4
Cost Effectiveness Can Reverse Ranking of Effective
Interventions
What are the most effective education programs?
Source: J-PAL, “Increasing test score performance”
Cost Effectiveness Can Reverse Ranking of Effective
Interventions
What are the most cost-effective education programs?
No overlap between those two groups.
Source: J-PAL, “Increasing test score performance”
Cost Effectiveness Can Reverse Ranking of Effective
Interventions
If we want to responsibly use impact evaluation to inform policy,cost analysis is crucial.
Source: J-PAL, “Increasing test score performance”
Definitions and Debates
Cost benefit analysis
Cost effectiveness analysis
Cost-utility analysis
Cost Benefit Analysis
𝑁𝑒𝑡 𝑝𝑟𝑒𝑠𝑒𝑛𝑡𝑣𝑎𝑙𝑢𝑒=∑𝑡=0
𝑛 𝐵𝑡
(1+𝑟 )𝑡−∑
𝑡=0
𝑛 𝐶𝑡
(1+𝑟 )𝑡
1. Monetize the benefits of the
program
2. Sum those up over time (for n
years) 3. Discount future gains
4. Do the same thing for costs
6. Voilà! A summary measure of the lifetime
net value of the intervention!
5. Subtract costs from benefits
McE
wan
201
2
Cost Effectiveness Analysis
McE
wan
201
2
1. Calculate the cost
2. Divide it by incremental
effect
𝐶𝐸=𝑐𝑜𝑠𝑡 𝑝𝑒𝑟 𝑎𝑑𝑑𝑖𝑡𝑖𝑜𝑛𝑎𝑙𝑢𝑛𝑖𝑡 𝑜𝑓 𝑏𝑒𝑛𝑒𝑓𝑖𝑡
What the h#@! is an additional unit of benefit?
10% increase in business profits 1 standard deviation increase in crop yield
10% increase in farmer knowledge Additional woman receiving prenatal care
Additional student in school 1 s.d. increase in student learning
CBA Absolute desirability of
intervention Allows comparison across
totally different interventions (health versus agriculture)
Requires very strong assumptions (lifetime benefits of intervention…)
Pros and Cons
CEA Much more transparent: No
need to monetize range of benefits
Simpler to explain Useful for comparing
interventions with common outcome (agricultural extension versus targeted savings)
Implicit assumption: Common post-intervention trajectory
McE
wan
201
2
Question: The Minister of Finance is deciding whether to scale up an Agricultural Extension program or a Business Training
program. What kind of analysis will be most useful?
1. Cost-Benefit Analysis
2. Cost-Effectiveness Analysis
Answer: (1)
Question: The Minister of Education wants to increase student test scores. What kind of analysis will be most useful?
1. Cost-Benefit Analysis
2. Cost-Effectiveness Analysis
Answer: (2)
Whose costs?Take a social perspective
Seed distribution program
Ministry of Finance
DeliveryCommunity leadersHouseholds
Ministry of Agriculture NGO Partner
McE
wan
201
2
Question: The government is giving grants to entrepreneurs. Where do these enter the costing equation?
1. As cost to the government.
2. As benefit to the entrepreneur.
3. Both.
Answer: It depends. In CBA, the answer is (3), so it cancels out.
In CEA, we don’t include other benefits (e.g., increased revenues, better health), so why this? So (1).
McEwan 2012 and Dhaliwal et al 2011
Ingredients method• Personnel• Facilities• Equipment• Materials• Other program inputs• Client inputs
Recipe for Cost AnalysisMetaphor for
successful cost analysis
Looks delicious!
McE
wan
201
2
Where do we get the info?Budget• What it was supposed to cost• Not actual cost• Often misses client costs
Interviews & direct observations• Did it go how it was supposed to go? (No)• How many seeds were actually distributed? How much time did
it actually require?
Data collected in impact evaluation• Remuneration• Time use
McE
wan
201
2
Need early: Guides rest of cost analysis
Very difficult to gather ex post
Must be designed ex
ante
Where do we get the info?Budget• What it was supposed to cost• Not actual cost• Often misses client costs
Interviews & direct observations• Did it go how it was supposed to go? (No)• How many seeds were actually distributed? How much time did
it actually require?
Data collected in impact evaluation• Remuneration• Time use
McE
wan
201
2
Need early: Guides rest of cost analysis
Very difficult to gather ex post
Must be designed ex
ante
Field coordinators are uniquely
placed to gather these data.
Value each ingredientAdjust for *– Inflation ---– Time-value *
– Currency ---
+ Add it all togetherCost of the program
What do we do with the info?
McE
wan
201
2
Question: My agricultural extension program has community volunteers who spend 10 hours a week and
receive a bag of rice. How do I cost that?1. Market value of a bag of rice2. Market value of 10 hours of
time for people with similar background
3. Both4. I don’t know; I was checking
my email.
Answer: (3). Hopefully not (4). Come on team, heads in the game. Really, it depends.
If “government only” perspective, could leave out, but volunteers in pilot may not be available at scale.McEwan 2012 and Dhaliwal et al 2011
JPAL
201
4
What Cost Effectiveness Looks Like in Practice
• Cost analysis is extra work• Most studies don’t do it– Of 77 RCTs, 56% reported
zero data on incremental costs (McEwan 2014)
• Not doing it significantly limits policy relevance
Well, that was easy. Not!
That was exhausting!
Now that you know how to do it
A few caveats to keep in mind
Caveats
Cost-Effectiveness Sensitivity to Errors in
Estimates
Additional s.d. of student learning per $100
Evan
s & P
opov
a 20
14
Gray: Measured cost
effectiveness
Black: 90% confidence interval on
impact estimateLesson: Be humble about cross program ordering.
Cost-Effectiveness Sensitivity to Place
Evan
s & P
opov
a 20
14
Actual effectiveness-cost ratio for
India programGray: Other programs
Additional s.d. of student learning per $100
Red: Use other country costs of
community teachers
Lesson: Adapting cost-effectiveness across countries takes work.
Pilot vs Scaled up
Evan
s & P
opov
a 20
14
Cost Sensitivity to Exchange Rates
Dhal
iwal
et a
l. 20
11
Standard exchange rate Purchasing power parity
Assumes relative prices in country Adjusts for price levels across countries but messes up relative prices
Ignoring:1. Categories of ingredients: facilities, equipment,
client inputs2. Sensitivity (to place, to scale, etc.)
Common Pitfalls
Adap
ted
from
De
La B
riere
3. Exchange rates4. Reality (costing ideal vs.
actual program).
Not providing enough details to verify 1 to 4
1. No policymaker adopts a program without knowing the cost (we hope!)
2. Costing and cost effectiveness take us from scholarship to practical policy implementation
3. It is just as important to be prospective with costs as with impact evaluation
4. Field coordinators are particularly well placed to gather cost data
Remember!
• Papers!– Cost-Effectiveness Analysis of Education & Health Interventions in Dev
eloping Countries, by McEwan (2012)
– Comparative Cost-Effectiveness Analysis to Inform Policy in Developing Countries, by Dhaliwal et al (2011)
– Cost-Effectiveness Measurement in Development: Accounting for Local Costs & Noisy Impacts, by Evans & Popova (2014)
• A book! – Cost-Effectiveness Analysis: Methods & Applications, by McEwan &
Levin (2000)
Additional Resources - 1
• IDB Evaluation Hub: Cost Benefit and Cost-Effectiveness– Background materials– Terms of reference– Examples– Templates
• Poverty Action Lab– Examples for student participation and student learning– What does this look like in Excel?
Additional Resources - 2
Credits• All clip art from openclipart.org• “Just Do It” from Adweek.com• NGO partners from OneAcreFund.org• Some other photos from
World Bank Flickr stream
Thank you!