FIGHTING POVERTY WITH WHAT WORKS (EVIDENCE):

The case for impact evaluations

Thomas Chataghalala Munthali, PhD

ECAMA Inaugural Research Symposium:

Golden Peacock, 10th October 2014

Reflection

But test everything; hold fast what is good.

1 Thessalonians 5:21 (ESV)

Introducing IPA • Innovations for Poverty Action (IPA) is an international

non-profit research organization that has a strong local presence through its 14 country offices serving 51 countries.

• IPA works towards the goal of reducing poverty by ensuring that policy is based on scientifically proven evidence.

IPA’s VISION More evidence, less poverty

IPA’s MISSION To discover and promote effective solutions to global poverty problems.

IPA works on three core areas:

• Research

• Capacity building on randomized evaluations

• Policy outreach and scaling-up

HOW ARE WE CURRENTLY ACHIEVING OUR GOALS?

CREATE EVIDENCE

• Randomized evaluations, with a focus on understanding why programs work or not

• Partnership with academics and implementing organizations

• Creating research infrastructure around the world

TURN EVIDENCE INTO ACTION

• Dissemination through policy-briefs and conferences

• Working closely with implementation partners throughout the evaluation

• Direct role in implementing successful programs at scale

WHAT WE HAVE ACHIEVED

CREATE EVIDENCE

• Over 150 Principal Investigators

• 14 Country Offices

• 300 number of randomized evaluations completed or underway

• More than 500 staff trained in implementing randomized evaluations across 50 countries

TURN EVIDENCE INTO ACTION

• Two programs scaled across countries, leading to 40 Mn children dewormed and thousands chlorine dispensers

• Several organizations changed their program based on research results.

• Thousands of practitioners attended conferences .

To see studies done by IPA in Malawi and beyond visit: www.poverty-action.org

WHERE WE WORK? 14 country Offices – planning to expanding to 20 in next five years.

Innovations for Poverty Action (IPA) Malawi

Basic History

IPA started work in Malawi in 2009 but got registered in 2010. IPA-Malawi has grown over the years and now has long-term staffing of 17 with an annual project budget of over US$1.6million.

Over 11 research projects have been completed with considerable influence on policy especially at the decentralized levels. Most of the research work has been in private sector and enterprise development as well as agriculture. Notably, most of the research has this far been largely driven by PI research interests. Currently, IPA-Malawi is developing a research agenda that reflects PI interests as well as Government’s development priorities as defined in the Malawi Growth and Development Strategy (MGDS II: 2011-2016).

Priority sectors

IPA-Malawi now has a five year Strategic Plan (2014-2018) that focuses on four priority sectors:

i. Education (Especially basic education and early

child development), ii. Health (especially reproductive health), iii. Enterprise development (especially SMEs), and iv. Agriculture (especially climate smart and

smallholder commercial farming).

Main Challenge with evidence-based planning

• Randomized Control Trials (RCTs) not fully known/appreciated/utilised by stakeholders as an important impact evaluation tool for measuring attribution of programmes to observed impacts.

Introduction to impact evaluation

Reflection

Friend tells Marx: “Life is difficult!”

Marx asks Friend: “Compared to what?”

What is Evaluation?

Evaluation

Program Evaluation

Impact Evaluation

Monitoring and Evaluation

Evaluation

Program Evaluation

Impact Evaluation

Monitoring

Components of Program Evaluation

• What is the problem? Needs Assessment

• How, in theory, does the program fix the problem?

Program Theory Assessment

• Does the program work as planned? Process Evaluation

• Were its goals achieved? The magnitude?

Impact Evaluation

• Given magnitude and cost, how does it compare to alternatives?

Cost Effectiveness

Evaluation should usually be conducted:

A. Externally and independent from the implementers of the program being evaluated

B. Externally and closely integrated with program implementers

C. Internally

D. Don’t know

Impact evaluation methods

1. Non- or Quasi-Experimental Methods

a. Pre-Post

b. Simple Difference

c. Differences-in-Differences

d. Multivariate Regression

e. Statistical Matching

f. Interrupted Time Series

g. Instrumental Variables

h. Regression Discontinuity

Ice Breaker

For every complex question there is a

simple answer -- and it's wrong.

-- H.L. Mencken

Impact evaluation methods Contd.

2. Randomized Experiments, also known as:

– Random Assignment Studies

– Randomized Field Trials

– Social Experiments

– Randomized Controlled Trials (RCTs)

– Randomized Controlled Experiments

Clip on randomized control experiments/trials

What is a randomized experiment?

Start with simple case:

• Take a sample of program applicants

• Randomly assign them to either:

Treatment Group – is offered treatment

Control Group - not allowed to receive treatment (during the evaluation period)

Key advantage of randomized experiments

Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment,

any difference that subsequently arises between them can be attributed to the program rather than to other factors.

25

Intervention

Time

Pri

mary

ou

tco

me

Counterfactual

Impact

What is Impact?

How to measure impact?

• What would have happened in the

absence of the program?

• Take the difference between

what happened (with the program)

…and

- what would have happened (without

the program) = IMPACT of the program

Some responses Contd.

Access to obstetric care and distribution of reproductive health commodities such as medication and contraceptives may be seriously hampered after disasters caused by climate change such as floods, putting women at risk for unintended pregnancies, sexually transmitted infections, including HIV.

Women and girls are generally expected to care for

the sick, particularly in times of disaster and environmental stress. This limits the time they have available for income generation.

Some of the research (impact evaluations) done in Malawi

Active Projects:

• Finance Sector and private sector development:

– Direct Deposits and Financial Literacy : measuring the changes in household financial behavior that are caused by promoting savings in the form of financial literacy trainings and the introduction of regular or labeled savings accounts

• PIs: Dean Yang, Jessica Goldberg, Lasse Brune

– Fingerprinting to reduce risky borrowing: aims at expanding the knowledge base about the impact of fingerprinting on repayment behavior and thus costs and risks of lending that microfinance organizations generally encounter in rural markets.

• PIs: Dean Yang, Jessica Goldberg, Xavier Gine

– Business Registration and Impact Evaluation Project (BRIE): Evaluating whether or not business registration improves enterprise performance.

• PIs: Francisco Campos and Marcus Goldstein

• Agriculture

– ‘Making family farmers more productive’ – Fomento project: The goal of this project is to implement and evaluate a modified version of a capital and information transfer program for extremely poor farmers .

• PIs: Susan Godlonton, Kate Ambler, Allan de Brauw

Overview: IPA Malawi

Completed Projects:

• Agriculture

– Making Networks Work (MNW) : The Project aims at evaluating how best to leverage communication within networks to increase the spread of two productive but under-utilized agricultural technologies in Malawi (pit planting for maize and improved crop residue management).

• PIs: Lori Beaman, Jeremy Magruder, Paul Fatch

• Rural Development:

– Access to Transport in Rural Malawi: Assessing the socio-economic impact of access to public transport on the lives of those living in remote rural areas.

• PIs: Jessica Goldberg, Dean Yang, Rebecca Thornton

• Social protection:

– Public Works Programme: aims at examining the impact of the World Bank supported public works program (PWP) as an income promoting mechanism via its impact on agricultural productivity through increased access to yield-improving inputs and any impacts on labor markets.

• PIs: Jessica Goldberg, Kathleen Beegle, Emanuela Galasso

Completed projects Contd.

• Enterprise Development:

– Malawi Savings Project (GLASE): to measure impact of increasing access to saving services on business creation, survival and growth, as well as on farming productivity and returns

• PIs: Dupas, Karlan, Robinson

– Borrower Responses to Fingerprinting for Loan Enforcement in Malawi: Assessing ability of fingerprinting in improving borrowers’ repayment rates

• PIs: Dean Yang, Jessica Goldberg, Xavier Gine

– Reducing Barriers to Saving in Malawi: the impact of enhanced savings products on the use of different agricultural inputs, farm output, and overall well-being in rural farming households?

• PIs: Dean Yang, Jessica Goldberg, Xavier Gine

• Governance

– Domestic Election Observers: to assess the impact of domestic election observers on detecting and preventing electoral fraud.

• PIs: George Ofosu and Daniel Posner

Ice Breaker

According to a new survey, women say they feel more comfortable undressing in front of men than they do undressing in front of other women. They say that women are too judgmental, where, of course, men are just grateful.

-- Jay Leno

For More …

Visit: www.poverty-action.org/malawi

You need technical support?

• Partner with Innovations for Poverty Action

(IPA) for impact evaluations on your

project(s)!

• Remember we are not a team of

consultants – won’t charge you

professional fees!

Sister organizations • The Abdul Latif Jameel Poverty Action Lab (J-PAL), based at the

Massachusetts Institute of Technology (MIT) and the University of Cape Town (UCT), consists of a network of leading academics who use randomized impact evaluations to test the effectiveness of alternative ways to improve welfare.

– Runs free Executive Education course on randomized evaluations

www.povertyactionlab.org

• Evidence Action responsible for taking proven impact initiatives to scale – just successfully chlorine dispensers in Zomba.

www.evidenceaction.org

Reflection

Therefore, I will always remind you about

these things, [since] you are well-

grounded in the truth that you now have.

2 Peter 1:12 (God’s Word Translation)

ZIKOMO KWAMBIRI!

IPA-Malawi

•

• THANK YOU!

Each question you ask will be evaluated, so do you really want to ask questions?

CONTACTS

Thomas Chataghalala Munthali, PhD COUNTRY DIRECTOR

Innovations for Poverty Action (IPA)

Malawi Country Office P.O Box 31093

Lilongwe 3 Email: tmunthali@poverty-action.org Skype: thomas.chataghalala.munthali

Malawi Office: + 265 1 762 424 | Mobile: +265 999 803140 | USA Office (HQ): 101 Whitney Ave. New Haven, CT 06510 USA |

+1 203-772-2216

www.poverty-action.org/malawi