Does Aid Work?Michael Kevane

Dept. of Economics Santa Clara University

• What is aid? How much is going where?

• Who is answering the big question

• Why this is a bad question

• Reframing the question into smaller questions

• My experiences with small-scale aid project: Friends of African Village Libraries

Does Aid Work? Causality hard to determine

Bad shock to

economyBad

outcomes for people

Increased flows of

aid

?

Time frame may not be knowable

Project done Wow!

Little to show

5 years 5 years

Aggregating outcomes: Net cost-benefit

Treadle pumps

Cleft palate

Solar lighting -$2,000,000

$1,000,000

$50,000

Aggregating outcomes: Moral sentiments

Treadle pumps

Cleft palate

Solar lighting

Save the planet

Huh?

Of course

If we knew the answer to the question…

Does aid work?

So what? How to make it better?

Yes/No

Examples of evaluation issues

• Social entrepreneurship- selling to the bottom of the pyramid –How to distinguish social good vs. social

bad? E.g. cell phones as donuts

• Microfinance

– Joint liability lending to women with no collateral

– Systemic risk in Burkina Faso

Examples of evaluation issues

• Darfur fuel efficient stoves

–Fuel collection poses multiple risks

–Laboratory tests say FES do very well

–What about in the camps? Not very effective

–Markets for carbon credits

Reframing the question

• Using randomized evaluations of small-scale aid projects to measure impact and learn about effectiveness – Among well-implemented projects, which have high

impact? – Which have high measured “taxable” revenues relative to

costs?

• All projects are bundles of attributes (process, messages, context) and have multiple causal pathways to outcomes. – Key enable scaling and replicating

Reframing the question• What makes some projects work and others not?– Example: A water purification project is wildly successful

• Technology – system actually does more relaible purification at lower cost

• Marketing – users think pure water makes them smarter• Design – users “see” the water getting pure• CEO – hands on and full of energy• Incentives – someone gets paid when water purification actually

used• Community engagement – local activists are motivated to work on

project– Learning about which dimension of bundle is more

important, and how bundle affects outcomes, is expensive.

Reframing the question

• What is the mechanism that changes behavior? Example: Establish coffeehouses to reduce social stress. Outcome, people frequent coffeehouses and social stress reduced.

• But why did this happen?– Behavioral predilection (i.e. want to see people’s faces)– Addiction – people addicted to coffee, channel stress away

from social stress– Good taste – Any good food product makes people less

stressed out– Want to hold warm thing – All warm and fuzzy projects

have similar effects- distribute puppies

Randomized experiments to estimate intervention impact

• Randomly select samples to measure differences in outcomes

• One sample gets treatment, the other is control• Use covariates to be more precise in statistical

significance of measure of impact• Use power calculation to figure out sample size

needed to estimate expected effect size as statistically significant difference

• Problem of external validity

Other approaches: Non-Experimental Designs

• Difference in differences – E.g. radio stations established across country at different times,

so look at changes in behavior over time and across regions• Regression Discontinuity

– E.g. Families with older grandparent above 65 receive pension, so examine differences in behavior between families with grandparents aged 62-64 and those 65-67

• Matching– Some schools received solar panels, so match to similar sized

schools with similar populations• Instrumental Variables

– Find a variable that explains variation in “treatment” but not “outcome”

– e.g. dust from rock crushing operation increases asthma, so acts like a random “treatment” of effects of asthma on schooling outcomes (unless dust directly changes schooling, or crushing operations not located near high asthma centers)

Community participation

• Defining community is a tricky issue– Villages are often divided, government is not

democratic

• Community assessment– How to ensure multiple voices are heard? Lots of

barriers

• Bottom-up– But why is this a value?

• Because of evidence that more successful?• Because intervener values it?

Problems of the pelleteuse

• Cost-benefit: Much more expensive to run, fewer benefits, appropriate technology: Were there any uses of pelleteuse?

• Credit constraint: Costs upfront, benefits over time• Incentives: Who was the residual claimant?• Village politics: Success changes distribution of power;

power might be zero sum• Land tenure may not be resolved• Imagination and knowledge skills• Tradition? Often overplayed• Organization of foreign aid itself