Using Impact Evaluation to Improve Development Projects

Arianna Legovini World Bank

USG Evaluation Forum

December 2, 2015

What does DIME do?

2

Transform development policy

1. Run experiments to inform decisions

2. Build agencies’ capacity to do this systematically

3. Draw lessons and share them face-to-face to global audiences

ideas

1. Improve evidence-base at design stage

2. Experiment with causal mechanisms that can change results during implementation

3

We evaluate the impact of packages of interventions (“what”) and experiment with mechanisms (“why/how”)

to make them work better:

IE evaluation questions in IEs with DIME Involvement

(at a minimum) IE increases project disbursements & closes implementation gap

statistically significant at 1% level

5

40.8%***

-54.2%***

40.0%***

15.6%

-45.3%**

49.1%***

Rate of Disbursement(Up to Mid-Term)

Gap (Up to Mid-Term)

Rate of Disbursement- Adjusted by SectorAverage (Up to Mid-

Term)

Rate of Disbursement(overall)

Gap (overall) Rate of Disbursement- Adjusted by Sector

Average (overall)

What is i2i impact evaluation to development impact?

A global platform to

generate and share IE knowledge and

transform the path of development policy

(especially in under-researched areas)

6

What makes i2i different from other IE initiatives?

1. Bottom-up approach – Empower government agencies to solve their own problems/find their own solution

– Open platform for expert advice and support

– Equalize opportunities in selection for technical and financial support

2. High research quality and policy relevance – methods for causal analysis

– data quality

– attention to implementation

– analysis and peer review

– collaboration and policy dialogue to ensure policy actions

3. Generate evidence throughout policy lifecycle – policy design

– mid-course adjustments

– policy decisions on scale up/down.

7

We value equally the generation of knowledge and its application to policy decisions

i2i pillars and thematic areas

8

Shared Prosperity

• Finance, Private

Sector and Jobs • Agriculture • Infrastructure

Governance

• Public Sector Governance

• Justice • Local Development

Climate Change

• Energy • Environment • Natural Resource

Management • Agriculture • Transport

Fragility & Conflict

• Reintegration • Governance • Gender-based

Violence • Urban crime &

violence

Gender

• Human Capital • Economic

Opportunities • Voice/Agency

5

1 2 3

6

Cross Cutting Themes

Human Development

• Education • Health • Social Protection

4

What type of technical support does i2i provide?

• Build capacity through workshops and learning by doing

• Match researchers and operations

• Full implementation support

• In-house research

• Coordination across thematic areas in close collaboration with operational departments

– Knowledge priorities

– Operationalize results

DIME i2i Workshops

No. teams

Impact Evaluation 4 Peace Mar 2014 18

Innovations for Agriculture Apr 2014 8

Impact Evaluation for Legal Empowerment May 2014 11

Innovations for Agriculture June 2014 23

Energy & Environment Oct. 2014 18

Governance Jan. 2015 27

Trade & Competitiveness & Finance May 2015 22

Transport and ICT June 2015 20

Number of participating teams per region

DIME i2i workshops: 128 teams in FY15-16

Putting in perspective: The Bank approves around 400

projects a year

AFR 41%

LAC 31%

SAR 12%

ECA 6%

MENA 5%

EAP 5%

Workshops increase participants’ knowledge

61

65

69

60

72

73

79

70

AGRICULTURE ENERGY & ENVIRONMENT T&C AND F&M TRANSPORT & ICT

PRE/POST TEST

Pre post

56%

79%

89%

63%

77% 80%

FCS Agriculture Energy & Environment Governance T&C and F&M Transport & ICT

workshop projects that submitted an EoI

note: projects with ongoing IE are excluded

Impact of Workshop on (External Double Blind) Technical Ratings of IE proposals

14

2.25

2.04 2.18

2.55

2.14 2.04 2.01

1.76

2.08 2.13

1.85 1.91

FCS Agriculture Energy & Environment Governance T&C and F&M Transport & ICT

Technical ratings of proposals (0-3 scale)

Attended workshop Did not attend workshop

Applications received across all windows

In total 229 applications received:

• Ongoing IEs: 49

• New IEs: 180

FCS 33

Agriculture 26

Energy & Environment 27

Governance 28

T&C and F&M 35

Transport & ICT 31

A THEORY OF CHANGE IMPROVING DEVELOPMENT PROJECTS

16

Clear roadmap

17

Clearer roadmap

What

To whom When

Where

Better capacity

18

Better capacity

Research team

Full time field coordinator

Training

More and better data

dime results indicators

88%

74%

75%

36%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Client participated in IE workshop(s)

Baseline results discussed with client

IE results discussed with client

Training provided for data analysis

Capacity Building

19

dime results indicators

84%

85%

48%

50%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

High-quality baseline survey

High-quality follow-up survey(s)

Improved administrative data

Availability of Data as a Public Good

Quality of Data

20

More effort into implementation

21

Motivation to perform

Goal: this is what we are

trying to achieve

Control: this is how we want

to do it

…and BTW somebody

measures (and cares)

Better decisions 22

dime results indicators

36%

23%

30%

0% 5% 10% 15% 20% 25% 30% 35% 40%

Baseline informed policydesign/implementation

Adopted delivery or behavioral mechanismbased on IE results

IE results were used to motivate scale-up/scale-down of policy at national level

Quality of Policy Decisions

23

In short…

• Put development in the hands of the implementing agencies

• Challenge them to improve project design and experiment with new ideas

• Provide them with the technical resources to generate great data and evidence

• Help them make smart decisions

Thank You! facebook.com/ieKnow

#impacteval

blogs.worldbank.org/impactevaluations

microdata.worldbank.org/index.php/catalog/impact_evaluation

http://dime.worldbank.org WEB