How to Run an Impact Evaluation: An Example from Malawi€¦ · How to Run an Impact Evaluation: An...

1

Dean Yang

University of Michigan

How to Run an Impact Evaluation: An Example from Malawi

Fertilizer use, cash crop farmers in central Malawi

2

39%

8%

22%

9%

2%

17%

1% 1% 0% 0%

2%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

0 to 9 10 to 19 20 to 29 30 to 39 40 to 49 50 to 59 60 to 69 70 to 79 80 to 89 90 to 99 100 +

KG per acre categories

Urea per Acre of Maize

Note: Per-acre fertilizer recommendation for central Malawi is ~60kg urea, ~40 kg 23:21 (per-hectare recommendation is 150 kg urea and 100 kg 23:21).

Official government recommended level

Raising farm output with rural finance

• Insure farmers against adverse events

– Provide insurance against poor rainfall

• Facilitate credit for agricultural inputs

– Improve repayment via biometric identification

• Encourage farmers to save for their own input purchases

– Facilitate access to ordinary and “commitment” savings accounts

3

Raising farm output with rural finance

• Insure farmers against adverse events

– Provide insurance against poor rainfall

• Facilitate credit for agricultural inputs

– Improve repayment via biometric identification

• Encourage farmers to save for their own input purchases

– Facilitate access to ordinary and “commitment” savings accounts

4

Agenda

• Motivation – MRFC’s perspective

• Intervention: fingerprinting

• Sample

• Randomized assignment to treatment

– Choice of club-level randomization

• Data for analysis

– Baseline survey

– Follow-up survey

– MRFC administrative data on loan repayment

5

Malawi Rural Finance Company

• Malawian microfinance institution

• One of most important providers of credit in rural areas

• Group-liability loans to rural entrepreneurs as well as farmers

6

MRFC loan officer

• Club visits for loan orientation and repayment collection carried out by MRFC loan officer on motorcycle

7

Repayment problems

• Serious repayment problems among farmers

– Default rate often >10%

• Among other factors, perception that many defaults are due to imperfect personal identification

– MRFC can’t tell if a person applying for a new loan has defaulted in the past

– A borrower can take out a loan, default, and then apply for a new loan in future by using a different name

– If MRFC could tell who had defaulted in past, would not lend to them in future

• Problem: no formal system to identify people

– Loan officers can use their personal knowledge, but imperfect

8

Intervention

• To identify past defaulters, lenders in other countries use a variety of methods

– E.g., national ID cards

• Approach used in our study with MRFC:

– Collect fingerprints of farmers applying for a loan

– Tell farmers that prior to all future loans, they will be fingerprinted again

• Fingerprint will be used to look up individual’s past repayment history

• If they repay this loan on time, MRFC will be able to see that, and will approve next loan

• If they default on this loan, MRFC can see that as well, and they will be denied future loans

9

Fingerprinting

• Aug-Sep 2007

10

Demonstrating fingerprint identification

11

• What is the impact of fingerprinting on loan repayment?

Our evaluation question

12

intervention outcome

The loan program

• For paprika farmers in Central Malawi

• MRFC in-kind loans of ~MK17,000 (US$120) for paprika farming inputs

• Cheetah Paprika provides extension services and purchases paprika output

– Loan repayment forwarded to MRFC by Cheetah

– Farmers paid remainder

• How can farmers “cheat” MRFC?

– Take inputs, but don’t grow paprika

– Sell inputs for cash value

– Or sell paprika to other buyer, not to Cheetah

13

Malawi Study Areas

N

Young paprika in the field (Jan 2008)

15

A healthy paprika crop

16

Packing paprika for sale (May-Jun 2008)

17

Where did the sample come from?

• Cheetah Paprika provided us with list of farmers (and clubs) who had sold paprika to them in past in central Malawi

• MRFC agreed to consider all clubs for loans

• Study population:

– 3,206 MRFC borrowers in central Malawi

– In 214 farmer “clubs”

18

Paprika extension officer (plus our research manager)

19

A paprika club

20

Treatment and control groups

• Control farmers:

– Educational module on credit history

• Treatment farmers:

– Educational module on credit history (identical to module given to control group)

– plus:

• Biometric fingerprint collected from all farmers as part of loan application

• Use of fingerprints for unique identification explained

• Fingerprint identification demonstrated within group

• Question: Why give educational module to both groups? 21

Randomization of treatment

• Key question in study design:

• At what level do we randomize treatment?

• First idea: randomize at individual level

– What are the pros and cons of randomizing at individual level?

22

Group-level randomization

• We randomized at the level of the farmer club

• 214 clubs randomly assigned to treatment (fingerprinting) or control

• Rationale for doing this:

– Farmers may be upset by knowing they were treated differently from others

• And this may affect repayment

• (Although not clear whether farmers would be more upset if they were fingerprinted or not…)

• Clubs are geographically spread out, so clubs were generally not aware that other clubs might have had different treatment

23

Study schedule

July 2007

August 2007

Sep. 30, 2008

Clubs organized

Baseline survey and fingerprinting begin

November 2007

Loans disbursed

Loans due

September 2007

Baseline survey and fingerprinting end

Follow-up survey

August 2008

Baseline surveys (Aug–Sep 2007)

25

Study design incorporated from start

• Question: Why was it important that randomized evaluation was part of plan from the beginning?

July 2007

August 2007

Sep. 30, 2008

Clubs organized

Baseline survey and fingerprinting begin

November 2007

Loans disbursed

Loans due

September 2007

Baseline survey and fingerprinting end

Follow-up survey

August 2008

What were the impacts?

• Data for analysis:

– Administrative data from MRFC on loan repayment

– Follow-up survey

• In treatment and control groups, same percentage of farmers took out loans

• Other outcomes to compare between treatment and control:

– Total borrowed

– Repayment performance

27

Total borrowed

28

16 590 17 279

0

2 000

4 000

6 000

8 000

10 000

12 000

14 000

16 000

18 000

20 000

Treatment Control

Total borrowed (MK)

Repayment rate (on-time)

29

88%

80%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Treatment Control

% of loan repaid

Repayment rate (eventual)

30

90% 88%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Treatment Control

% of loan repaid

Are effects different for different types of borrowers?

• Overall differences between treatment and control groups are relatively modest

• So next step in analysis: are effects different for different types of individuals?

• Question: What different types of individuals would you look at?

• What we did: look at different types of borrowers by risk of default

– Calculate a “credit score” for each borrower

– Examine treatment vs. control for borrowers in 5 different categories

31

1%

3%

6%

8%

3%

6%

9%

7%

18%

39%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

less than

10%

10% to

20%

20% to

30%

30% to

40%

40% to

50%

50% to

60%

60% to

70%

70% to

80%

80% to

90%

more than

90%

Predicted Repayment for Loan Recipients

Predicted percentage repaid (“credit score”)

Repayment: % of balance paid on-time

33

88%

79%

91% 93%

89%

26%

74%

92%

96% 98%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Worst 2nd quintile 3rd quintile 4th quintile Best

Fingerprinted

Control

Repayment: % of balance paid (eventual)

34

92%

83%

93% 94% 92%

67%

77%

93%

96% 99%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Worst 2nd quintile 3rd quintile 4th quintile Best

Fingerprinted

Control

Fraction of land allocated to paprika

35

19%

15%

21% 22%

23%

11%

16%

19%

21%

23%

0%

5%

10%

15%

20%

25%

Worst 2nd quintile 3rd quintile 4th quintile Best

Fingerprinted

Control

Market inputs used on paprika (MK)

36

9 600

8 381

9 858

8 088

8 874

2 503

4 911

11 803

11 262

12 378

0

2 000

4 000

6 000

8 000

10 000

12 000

14 000

Worst 2nd quintile 3rd quintile 4th quintile Best

Fingerprinted

Control

In sum

• A simple RCT in microfinance found a cheap and easy way to raise loan repayment performance

• Lessons for RCT design:

– Randomization at group is feasible and avoids possible problems of individual-level randomization

– Randomized study design needs to be built into program from the beginning

37