Randomized Control Trials forAgriculture

Pace Phillips, Innovations for Poverty Action

www.poverty-action.org

• What we do• IPA Ghana • IPA’s Mission and Approach• Why evaluate? • How to evaluate impact? • Examples of evaluations

Outline

What does IPA do?• IPA was created in 2002 to determine what

works and what does not work in poverty alleviation through rigorous evaluation.

• Estimate magnitude of effects– Most cost-effective solutions; can be surprising

• Learn how to improve programs through testing operational questions

• Determine where to spend limited resources

IPA Ghana• In Ghana since 2005• 16 projects nationwide• Permanent offices in Accra and Tamale• Current Agric Partners:

– Ghana Insurers Association– GIZ – IFPRI – Presbyterian Agricultural Service– MoFA– SARI– ISSER

• Trainings /Conferences

• Surprisingly little hard evidence on what works

• Can do more with given budget with better evidence

• If people knew money was going to programs that worked, could help increase pot for anti-poverty programs

• Instead of asking “do aid/development programs work?” should be asking:– Which work best, why and when?– How can we scale up what works?

Why focus on impact evaluation?

6

Process and impact evaluation

InputsInputs OutputsOutputs OutcomesOutcomes

•Process evaluations

•Monitoring

In the control of the organization

Impact

Impact evaluation: how, and how much, does the organization affect the welfare of beneficiaries (and non beneficiaries)

Monitoring/Process evaluation: tracking development and outputs of a program as compared to stated objectives, targets and timelines

• In Northern Ghana, a development organization undertakes a program to promote fertilizer use to improve yields. How do we know if the program was successful?

How do you evaluate a program?

Impact: What is it?

Time

Cro

p Y

ield

s

Impact

Counterfactual

Intervention

What is the impact?

Time

Cro

p Y

ield

s

ImpactCounterfactual

Intervention

What is the impact?

Time

Cro

p Y

ield

s

ImpactCounterfactual

Intervention

• Counterfactual: What would have happened in the absence of the program?

• Problem: Counterfactual is not observable the key goal of all impact evaluation methods

is to construct or “mimic” the counterfactual.

• Solution: Counterfactual is often constructed by selecting a group not affected by the program.

Counterfactual

• Without Random Selection, the control group will have intrinsic differences that can bias the study.

• Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment

• Any difference that subsequently arises between them can be attributed to the program rather than to other factors

• If properly designed and conducted, randomized experiments provide the most credible method to estimate the impact of a program

Why randomize?

13

Randomly samplefrom area of interest

Random sampling and random assignment

Randomly samplefrom area of interest

Randomly assignto treatmentand control

Random sampling and random assignment

Randomly samplefrom both treatment and control

Which method we use matters!

Method Impact Estimate

(1) Pre-post 26.42*

(2) Simple Difference -5.05*

(3) Difference-in-Difference 6.82*

(4) Regression 1.92

(5)Randomized Experiment 5.87**: Statistically significant at the 5% level

Underinvestment in Agriculture• A study in Northern Ghana to understand if farmers do not invest in their farms because either capital constraint or risk aversion.

Year 2: 1377 farmers in 74 villages

Year 2 Randomization

Insurance: 729 farmers Both: 108 farmersCapital: 363 farmers

Rainfall index insurance offered at varying prices

GHS 350 provided

Control: 177 farmers

Rainfall index insurance offered at varying prices, AND GHS 350 provided

Results• Capital only farmers– Used more inorganic fertilizer only

• Capital and insurance farmers– Increased farm investment by 20 percent

• Insurance farmers– Increased total farm expenditure by 13 percent

• Increased inorganic fertilizer use by 25 percent• Increased cultivation area by 8 percent• Increased expenditures on land preparation by 12 percent • Increased total labor use on plots by 13 percent• Increased harvest output by 8 percent

• Increase in investment but no increase in profitability

Demand for Agriculture InsuranceUptake

A Well Timed Nudge• A study in Western Kenya tested whether farmers purchases of fertilizer was dependent on the time of the year it was offered.

Baseline: 1,125 Farmers in Suri Kenya

Randomization

Fertilizer offered AFTER HARVEST

with free delivery

50% Subsidy Fertilizer offered

just BEFORE GROWING SEASON with free delivery

CHOICE of when they want the

fertilizer offer with free delivery.

Control – No Offer

Fertilizer offered just BEFORE

GROWING SEASON with free delivery

Results• Farmers had high demand and ability to

purchase fertilizer in advance. • Impact of the After Harvest offers were

comparable to that of a 50% subsidy at fertilizer application time.

• Increase in Fertilizer Use• Control Group 28%• After Harvest 39%• Choice of Time 47%• Before Growing Season 33%• 50% Subsidy Before Growing Season 41%

• Once the program stopped, fertilizer usage went back to what it had been.

Thank You

www.poverty-action.org

IPA Ghana • Why do farmers under-invest in farms?• What is the impact of financial savings programs

in schools?• What is the willingness to pay for clean water?• What is the impact of remedial education

programs, smaller class size, teacher training and tracking on literacy and numeracy?

• What is the impact of health insurance education on enrollment and reenrollment in NHIS?

• Are mobile reminders an effective way to improve adherence to ACT regimens?

Problems Selecting a Control Group

Difference between Groups

Intrinsic Differences Impact of the program

Observables Non Observables

Selection Bias

Without Random Selection, the control group will have intrinsic differences that can bias the study.

Our approach

Innovate Evaluate Replicate Communi-cate Scale

- Understand market failures-Develop innovative solutions to poverty- Use frontier knowledge from economics, and psychology

Randomized Controlled Trials-Impact evaluations- Comparing variations of an intervention- Experiment with product designs

Replicate evaluations in various settings to :- Generalize research findings - Tell practitioners what works (and not), when

Effectively communicate to practitioners: -Conferences-Workshops with policy makers and practitioners-Policy memos and focus notes

Facilitate scale-up of effective solutions :- Active policy outreach -Practitioners’ toolkits - Hands-on technical assistance

We generate insights on what works and what does not through randomized evaluations, and ensure that those findings will be useful to, and used by practitioners and policy makers