Upload
trinhkhue
View
215
Download
0
Embed Size (px)
Citation preview
Impact Evaluation to Development ImpactTransforming Development through Impact Evaluation
Annual ReportMarch 2016–March 2017
IMPACT EVALUATION TO DEVELOPMENT IMPACTiii
vACKNOWLEDGMENTS
vii–viiiFOREWORD
ix–xii1. EXECUTIVE SUMMARY
1–162. DIME MODEL FOR REAL-TIME EVIDENCE-BASED POLICY2.1 Engagement with the Global Practices. . . . . . . . . . . . . . . . . 12.2 Engagement with the Clients . . . . . . . . . . . . . . . . . . . . . . . . . 32.3 Collecting Data and Setting Up Data Infrastructures. . . . 82.4 Generating Evidence and Motivating Change . . . . . . . . . . 12
17–223. SELECTION & QUALITY ASSURANCE FOR POLICY RELEVANCE & TECHNICAL RIGOR
23–304. DELIVERING ON OUR PROMISES
31–425. HOW WE CONTRIBUTE TO DEVELOPMENT ISSUES5.1 Growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.2 Shared Prosperity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.3 Risk and Vulnerability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.4 Governance and Accountability . . . . . . . . . . . . . . . . . . . . . . 375.5 Global Public Goods and Externalities . . . . . . . . . . . . . . . . .39
IMPACT EVALUATION TO DEVELOPMENT IMPACT ivIMPACT EVALUATION TO DEVELOPMENT IMPACT iv
43–686. ECONOMIES OF SCALE IN LEARNING6.1 Fragility, Conflict and Violence . . . . . . . . . . . . . . . . . . . . . . . 436.2 Agriculture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .466.3 Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .496.4 Climate Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526.5 Financial and Private-Sector Development . . . . . . . . . . . .546.6 Transportation and ICT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576.7 Gender. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .596.8 Edutainment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64
69–747. INNOVATION IN DATA QUALITY AND MONITORING OF POLICY INFLUENCE7.1 Monitoring System. . . . . . . . . . . . . . . . . . . . . . . . . . 697.2 Review of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
75–828. CASES OF POLICY INFLUENCE
83–869. COMMUNICATING LESSONS AND RESULTS
IMPACT EVALUATION TO DEVELOPMENT IMPACTv
ACKNOWLEDGMENTS
This annual report has been prepared by World Bank
DIME staff and consultants. It covers the reporting
period from March 2016 to March 2017. Contribut-
ing authors include:
Overall program management, progress, and communications:
n Arianna Legovini, DIME and i2i program
manager
n Florentina Mulaj, Operations Officer and i2i
program coordinator, DIME
n Anushka Thewarapperuma, Operations Officer
and i2i communications, DIME
n Chloe Fernandez, Research Analyst, DIME
n Rebecca De Guttry, Consultant, DIME
n Josine Umutoni Karangwa, Consultant, DIME
Fragility, Conflict, and Violence n Eric Mvukiyehe, Economist, DIME
n Marcus Holmlund, Economist, DIME
n Sarah Elise Elven, Consultant, DIME
Agriculture and rural development n Florence Kondylis, Senior Economist, DIME
n Paul Christian, Economist, DIME
n Maria Ruth Jones, Survey Specialist, DIME
n Astrid Zwager, Research Analyst, DIME
Governance n Vincenzo Di Maro, Senior Economist, DIME
n Daniel Rogger, Economist, DIME
n Bilal Siddiqi, Economist, DIME
n Michael Anthony Roscitt, Public Sector
Specialist, Governance GP
Climate Change n Aidan Coville, Economist, DIME
n Arndt Reichert, Economist, DIME
Financial and Private-Sector Development n Caio Piza, Economist, DIME
n Guadalupe Bedoya, Economist, DIME
n Aidan Coville, Economist, DIME
Transportation and ICT n Kevin Croke, Economist, DIME
Health and Edutainment n Victor Orozco, Economist, DIME
n Marcus Holmlund, Economist, DIME
Gender n Florence Kondylis, Senior Economist, DIME
n Eric Mvukiyehe, Economist, DIME
n Maria Ruth Jones, Survey Specialist, DIME
Monitoring system n Guadalupe Bedoya, Economist, DIME
n Chloe Fernandez, Research Analyst, DIME
n Rebecca De Guttry, Consultant, DIME
Collecting Data and Setting Up Data Infrastructures
n Florence Kondylis, Senior Economist, DIME
n Maria Ruth Jones, Survey Specialist, DIME
n Kristoffer Gustav Bjarkefur, Research Analyst,
DIME
IMPACT EVALUATION TO DEVELOPMENT IMPACTvii
FOREWORD
The purpose of Impact Evaluation to Development
Impact (i2i) is to change development practice
through greater emphasis on the dialectics of devel-
opment and the empirical testing of competing hy-
potheses, while leveraging international assistance
to help governments discover how they can make
best use of their overall resources. Some call it im-
pact evaluation (IE), some, science of delivery, and
others, adaptive programming. But the underlying
idea is to use operational research to generate useful
data and evidence to inform operational decisions in
real time toward greater policy effectiveness. In oth-
er words, using evidence to save and improve lives.
The core and seed funding that United Kingdom’s
Department for International Development (DFID)
provided in phase one of i2i created the dynamics
to bring a program like DIME to the forefront of the
World Bank discussions. First, it elicited an institu-
tion-wide process that generated a Bank-wide gov-
ernance for IE; second, it generated the incentives
for large and substantial participation and co-fi-
nancing; and, third and very importantly, it allowed
i2i to be present in all of the Bank’s Global Practices
and start to affect the operational culture across
the Bank and its partners- especially DFID and
Multilateral Development Banks (MDBs)-, as they
joined and contributed projects to i2i programs. As
a result, DIME is currently evaluating $8.2 billion in
the World Bank’s and $1.3 billion in DFID’s develop-
ment financing.
The problems we help address in development prac-
tice are in the conceptualization and implementa-
tion of policies and in the focus and content of de-
velopment research. These problems are common to
all development agencies. The first problem is that
policy designs are often weakly based on solid theo-
ry, that policies repeatedly (and predictably) fail the
implementation test, and that failure to generate
evidence compromises feedback loops into future
policies. Second, often development research is not
grounded in a deep understanding of reality and
research questions are not problem-driven. Third,
the traditional separation between development
research and practice compromises the quick ad-
vancement of useful knowledge generation.
We address these weaknesses by creating quality
in the engagement between researchers and practi-
tioners, and a process through which (i) data and ev-
idence is systematically generated and used through
the policy cycle and (ii) the capacity of implementer
agencies for data and evidence-based policy nur-
tured through a learning-by-doing approach. In this
report, we show that the benefits of handholding de-
velopment practice through a test-learn-and-adopt
model can affect development outcomes by large
margins. In Rwanda, earnings from farming doubled;
in Brazil, youth savings increased 11–31 percent; in
Senegal, legal delays were cut by 22–30 percent;
and in Mozambique, access to demonstration plots
increased by 20 percent. In the overall Bank portfo-
lio, we observed a 40 percent speeding up in project
implementation due to IE.
Phase one of i2i planted the seeds. The next phase
is designed to consolidate gains, expand donor part-
nerships, and continue to improve the quality of en-
gagement between research and policy.
IMPACT EVALUATION TO DEVELOPMENT IMPACT viii
First, we need to deepen and intensify the model
by investing in existing relationships to speed up
the process of problem-solving and solution-seek-
ing, thus pushing economies toward their efficient
frontier. What we learned from phase one is that,
in low-capacity and low-data environments, set up
times are long. Much time is devoted to building the
data infrastructure, and sometimes the regulato-
ry environment, before field experiments can take
place. We want to take advantage of the invest-
ments we are making across some 60 countries and
intensify experimentation and learning. This makes
eminent sense due to cost-effective use of resourc-
es and to continue building the dynamics to find
permanent and definitive solutions to development
problems.
Second, we need to work in sectors that currently
make only minor appearances in our program. Third
we need to collaborate with Global Practices in the
Bank to improve the quality of operations in the
pipeline by working systematically through project
typologies and creating an appetite for adaptive de-
signs and implementation frameworks. While it is
not always easy to distill operational learning, we
increasingly see the importance of group dynam-
ics and of operationalizing learning to change proj-
ects and address common failures in policy design.
Fourth and lastly is to test the possibility of moving
the profession from case-takers to case-givers and
propose a series of interventions to test big ideas in
development.
Arianna Legovini
Manager, Development Impact Evaluation Unit,
Development Research Group
IMPACT EVALUATION TO DEVELOPMENT IMPACTix
1. EXECUTIVE SUMMARY
OverviewImpact Evaluation to Development Impact (i2i) is a
World Bank program and global platform launched
in March 2014 with support from the United King-
dom’s Department for International Development
(DFID). It leverages international assistance to
generate high-quality and operationally relevant
impact evaluation (IE) research to transform devel-
opment policy, help reduce extreme poverty, and se-
cure shared prosperity. Its purpose is to change de-
velopment practice through a greater emphasis on
the dialectics of development and empirical testing
of competing hypotheses while embedding learning
into each element of the project cycle, from defin-
ing policy, through the decision to continue a pro-
gram or not, to the structure of the next phase of
learning.
Program components include:
n Developing and running experiments in
collaboration with government partners to
inform their policy decisions.
n Building agency capacities to do this
systematically.
n Drawing lessons and sharing them face-to-face
with direct clients and global audiences.
i2i provides support in various ways by: (i) allowing for
a more programmatic approach to evidence-based
policymaking; (ii) capacity building around a broad
set of stakeholders through policy-relevant re-
search agendas; (iii) expanding the reach of under-
developed areas in IE and evidence-based policy.
These are all critical elements to facilitate impact
and influence policy at different stages of a project
and the policymaking process. i2i has also funded
a web-based monitoring system (MyIE) that com-
plements the current advocacy and campaigning
efforts. MyIE reports on periodic progress of IEs
through the life of i2i as IEs move through different
phases of their lifecycle. These indicators showcase
the diversity of influence at all levels, which is well
beyond the typical IE results and impact.
i2i funds IEs across all World Bank sectors, covering
twelve operational sectors of the Bank across four
pillars and two cross-cutting themes. The pillars are
shared prosperity, governance, climate change, and
human development. The cross-cutting themes are
gender and fragility, and conflict and violence. Cur-
rent donor financing does not support the human
development sector, including education, health,
and social protection. i2i aims to support this sec-
tor as new donors join.
i2i adopts DIME’s operational model. DIME is a glob-
al program managed by DECIE in the World Bank
Research Group. DIME has developed institutional
structures to ensure that its IE products are relevant
and can influence the decision-making process for
development. Internally, DIME works with a cross-in-
stitutional council composed of chief economists
and directors from operational vice-presidencies.
The council provides strategic guidance. Working
groups in each global practice set learning priorities
and select IE cases. Each operational team for the
selected cases shape project design and structure
IMPACT EVALUATION TO DEVELOPMENT IMPACT x
The i2i portfolio was developed through close
collaboration with the Bank’s Global Practices,
operational teams, and client counterparts.
experiments to guide project implementation to-
ward greater effectiveness. Externally, DIME engag-
es with clients from the beginning to set research
questions, adapt policy implementation, and agree
on entry points to affect policy decisions using ex-
perimental results. This model transfers knowledge
and tools needed to support evidence-informed pol-
icymaking to country institutions through a medi-
um-term learning-by-doing approach.
Deliverables i2i has a portfolio of 131 IEs, spanning 53 countries
and covering all of the Bank’s regions and sectors.
Portfolio implementation is on track. As of January
2016, 24 percent of the portfolio has been complet-
ed. Most of these IEs have already produced final
outputs, such as working papers, publications, or
project completion reports. 43 percent of the port-
folio is in implementation phase, having passed
technical and policy relevance review and received
World Bank country director and sector manager
approval. A remaining 33 percent of the portfolio
is in preparation phase, with teams having passed
technical and policy relevance for expressions of in-
terest (EOI) and received preparation grants from
i2i to develop full technical proposals. This usually
entails research and Bank team travel to the coun-
tries to meet with the clients and work together
towards finalization of the methodology and imple-
mentation plans.
The i2i portfolio was developed through close col-
laboration with the Bank’s Global Practices, oper-
ational teams, and client counterparts. Each the-
matic program was initiated with a launch of an
impact evaluation workshop for capacity building,
aimed at developing strong IE team capacity and
collaborations between the operational staff, coun-
try governments, and researchers. From March
2014 to January 2017, i2i completed 14 workshops,
covering the following sectors: i) Fragility, Conflict,
and Violence, ii) Agriculture and Rural Development,
iii) Energy and Environment, iv) Governance, v)
Trade and Competitiveness, and vi) Transport and
ICT. Additionally, it completed sector cross-cutting
workshops focused on methods such as distribu-
tional impact analysis and innovations in big data
analytics. In all workshops combined, around 400
organizations participated and around 2000 peo-
ple were trained in impact evaluation methods.
i2i workshops serve three main purposes. They train
counterparts in IE methods to equip them with the
tools necessary for them to actively participate in
the design and implementation of their IEs. They
expose participants to latest available evidence so
that they can incorporate it in their own programs
and policy designs. And, they pair operational and
country teams with highly qualified IE researchers
to develop prospective IE proposals. Most impor-
tantly, workshops equalize opportunities to access
financial support across the development commu-
nity; by pairing policy teams with researchers they
improve the technical quality of proposals. On aver-
age, proposals submitted by teams that attend i2i
workshops receive higher technical and policy rele-
vance ratings from independent reviewers.
Policy InfluenceThe value of IE as a tool for more effective policy-
making is increasingly recognized by governments
and donors. Prospective multi-arm IEs designed to
reliably identify cause-effect relationships based
IMPACT EVALUATION TO DEVELOPMENT IMPACTxi
on counterfactual analysis can guide decisions
over which policies and programs are better able
to achieve desired objectives and how to design
these programs for maximum impact. A study
using data from Bank projects approved between
2005 and 2011 finds that projects with IEs imple-
ment development activities in a timelier manner.
Using disbursements against agreed activities as
an objective measure of implementation, the study
estimates that IE increases average cumulative
disbursements by two-fifths (40.8 percent) and re-
duces the planned-to-actual disbursements gap by
half (54 percent). The results suggest that IE is a
powerful tool to move projects from design to imple-
mentation. These projects are more likely to achieve
their objectives, supporting the idea that project
financing and IE research are complementary ser-
vices provided by the World Bank.
Further, i2i’s key feature is that its model enables
policy influence through all phases of project cycles.
The November 2016 update of the i2i Monitoring
System shows that 75 percent of the portfolio re-
ported that the IE informed design of programs
and policies, based on a clear understanding of the
underlying theory of change and highlighted areas
of uncertainty and critical assumptions. 61 per-
cent of the portfolio reported that IE evidence from
experimental testing of alternative mechanisms
was used by governments or other stakeholders to
adopt the most effective program alternatives or
to inform policy decisions. And 59 percent of the
portfolio reported that IE evidence was used to mo-
tivate scale-up or scale-down of policy.
For example, a large-scale evaluation of a school-
based financial education program in Brazil found
improved financial knowledge, attitudes, and behav-
ior of students. This resulted in the scaling up of the
program nationally by the Ministry of Education.
Another evaluation in Mozambique showed that
training and placing women in extension delivery
positions in agriculture benefits the broader popu-
lation of women in terms of technology awareness
and adoption. Results are being considered in the
design of a new agriculture extension policy in Mo-
zambique. In Nigeria, evidence and learning from
evaluations of maternal and child-health interven-
tions indicates that the lives of mothers and babies
are being saved. This is informing health policy dis-
cussions with the incoming Nigerian government.
Also in Nigeria, the results from the MTV television
drama Shuga evaluation show that the treatment
group was twice as likely to get tested, reported
fewer concurrent sexual partnerships, and reduced
gender-based attitudes and behaviors. Based on
these results, the Gates Foundation is discussing
scaling up its support for edutainment interven-
tions. In Kenya, the Patient Safety IE just recently
started the rollout of the intervention, but the IE
already reported significant contributions to poli-
cy during pre-implementation phase by enhancing
regulatory framework for health inspections and
strengthening institutional capacity through in-
spection protocols and monitoring systems.
Way Forward The last decade has seen a remarkable increase in
demonstrating value for money and effectiveness
in developmental programs run by the World Bank
and other multilateral lending and grant-making in-
stitutions. The standards for demonstrating impact
of development projects have also been raised sig-
nificantly. Rigorous and well-designed impact eval-
uation can help answer the “what” and the “how”
of economic development and help design better
policies. i2i is the largest international initiative de-
signed to systematically learn from development
experience on the basis of rigorous impact evalua-
tion. Its model is specifically designed to overcome
challenges from traditional approaches to evalua-
tion. This includes research-capacity constraints,
coordination failures and transaction costs related
to establishing researcher-policymaker relation-
ships, and a limited understanding of and the ability
to integrate IE into the implementation of policies
and programs at scale.
All i2i-supported IEs build on ongoing and complet-
ed work to create virtuous cycles of learning and
policy impact. Most importantly, by engaging the
government counterparts throughout all phases of
the IE cycle, i2i empowers governments to test in-
novations and scale up solutions. Through this ap-
proach of identifying pathways from policy to re-
sults, i2i will continue to improve accountability and
learn from development interventions and shape
the way development work is done at scale and in a
large number of contexts and practices.
IMPACT EVALUATION TO DEVELOPMENT IMPACT1
2. DIME Model for Real Time Evidence-Based Policy
2.1 Engagement with the Global Practices
Engagement with Global Practices and CCSAEngagement with the World Bank Global Practices
(GP) and Cross-cutting Solution Areas (CCSA) is a
fundamental ingredient to developing economies of
scale in learning and a strategic approach to chang-
ing development practice. GP and CCSA engage-
ment serves multiple purposes: (i) knowledge prior-
ities definition; (ii) strategic case selection; and (iii)
portfolio learning and feedback. The definition of
knowledge priorities is an iterative process that re-
flects both the composition of the project portfolio
and an evolving understanding of what is important
for GPs and CCSA to learn. They are usually reflect-
ed in selected thematic preferences in each round
of program development. Strategic case selection
is fundamental to ensure that important policies
and investments are included in each IE program
Defining Thematic Priorities in T&CThe work with T&C dates back to 2009 in the Africa region. The Director for Finance and Private Sector recognized the importance of measuring the impact of matching grant and SME service projects. These represented the bulk of the GP’s portfolio. Nine matching grant projects became the subject of IE but all of them failed to materialize for the lack of sample. This was because
the projects failed to attract firms’ interest. The problem of take-up was highlighted in the T&C IE Workshop which took place in 2015 as a central problem in private sector operations and generated awareness and willingness among project teams to rethink the way projects market and facilitate firm participation. In 2010, engagement with the business climate group expanded the program in the area of business climate reforms including simplification of firm registration procedures, tax reform, and inspection function. The evaluations of registrations procedures resulted in mild effects on registration rates and non-detectable effects on firm growth even among firms that registered their business. The limited effects of formalization on firm performance reported by those evaluations led to a reshape in the T&C approach. Formalization is no longer seen as sine qua non condition for growth of firms. The new focus shifted towards to identifying ways to make informal firms more productive. In 2016, the Trade & Competitiveness GP identified three areas of work that would move the discourse from eval-uating what projects do into stirring project designs into potential areas for firm growth: high-growth firms, firm linkages and regulatory efficiency. The evolution of T&C thematic area reflects an iterative process of learning about operational practices and the need to put emphasis on testing high-growth strategies that would help design operations in the future.
FIGURE 1: Engagement with GPs and Implementation
GP Sr. Director email
Meetings or clinics with IE experts
Workshop: Train and share evidence
Design facilitation
Call for proposals, double blind technical
review, technical committee
Concept note Bank
review
Concept note
Technical review
Step 5:Resource incentives
Step 5:Resource incentives
Step 3:Group
dynamics
Step 2:Project
feasibility
Step 1:Managerialincentives
TTL Response Preselection DesignSelection for preparation
Selection for implementation
Technical quality �2 DS Technical ratings
Technical quality �2 DS Technical ratings
Project characteristics �Operational opportunities �Budget opportunities
Project characteristics �Readiness �Nature of intervention �Thematic fit
TTL characteristics � IE awareness/ experience � Interact/incentivize �Management responsiveness
Country/ sector
strategic relevance
Strategic fit �Policy relevance �Learning opportunity
Client characteristics �Willingness �Understanding �Flexibility
TTL characteristics �Understand-ing �Flexibility
GP history �Lineage of IE program �Size of IE program
IMPACT EVALUATION TO DEVELOPMENT IMPACT 2
Furthermore, GP engagement is used to align the
priorities of Bank staff by increasing incentives for
investing in operational knowledge. These incen-
tives are both financial and non-financial to include
(i) clearly communicating managerial preferences
and incentives for project teams, (ii) developing
group dynamics and competition for excellence in
learning, and (iii) providing direct financial incen-
tives to conduct analytical work.
In practice, the level of engagement with each GP
and CCSA is endogenous to each GP and CCSA his-
tories, management interest in impact evaluation,
and whether or not i2i covered a GP or CCSA as
part of its program. GPs and CCSAs that have a
full level of engagement with DIME and i2i include
Governance, Trade and Competitiveness, Transport
and ICT, and Fragility Conflict and Violence (FCV),
each with senior management leadership, a dedi-
cated team working closely with the DIME team,
regular management consultations, co-sharing of
program costs, and active exchanges. The engage-
ments are reflected by the large shares of the i2i
portfolio in these areas. At a lesser level of engage-
i2i adopts a bottom-up and demand-driven approach, conducts high-quality and
policy-relevant research, and uses IE as a formative tool to generate
evidence throughout project lifecycles.
(as opposed to researcher-led project selection) and
that the cases reflect the priorities of the GP or
CCSA. Finally, portfolio learning is secured through
a continuous process of knowledge diffusion with
GPs and CCSAs that generates demands for sum-
maries, presentations to GP management and
knowledge boards, organization of IE events during
GF fora, etc.
IMPACT EVALUATION TO DEVELOPMENT IMPACT3
ment are Agriculture, Energy, Environment, Water
and Finance and Markets. With the exception of
Agriculture with a large and striving program, lower
levels of engagement are reflected by smaller port-
folios of IE. Finally the Health, Education and Social
Protection GPs were not directly targeted for pro-
grams and their large presence reflects non-i2i fi-
nanced work or the intersection with cross-cutting
areas such as Gender and FCV.
2.2 Engagement with the Clients
Workshops to Build Capacity and Stimulate Thinkingi2i adopts a bottom-up and demand-driven ap-
proach, conducts high-quality and policy-relevant
research, and uses IE as a formative tool to gen-
erate evidence throughout project lifecycles. Early
and sustained client engagement is a core element
of i2i’s work. Engaging with government agencies
and other clients early and often ensures their ac-
tive participation in defining research questions
and designing the IE from day one. Further, this
sets the foundation to build client capacity and
empower policymakers and practitioners to exert
control over their local environments. For example,
they can use data generated during an IE to make
mid-course corrections or use the final results to in-
form scale-up decisions. Ultimately, clients become
educated producers and consumers of evidence,
whether from IEs or other types of research, and
we create local capacity for the systematic use of
IE in policymaking.
IE researchers benefit from the early building up of
relationships with policymakers and practitioners.
It affords researchers a better understanding of
idiosyncratic contexts in which policies are crafted
and implemented, allowing them to better tailor re-
search questions and design. Early engagement be-
tween researchers, policymakers, and practitioners
bridges the gap between theory and practice and
kick-starts a process of feedback loops between
various stakeholders. This is a foundation for iter-
ative learning.
The IE workshop is the vehicle to initiate, stimulate,
and strengthen this process. Thematic workshops
are carried out periodically by each i2i program.
These bring together policymakers, practitioners
and operational staff, and World Bank and exter-
nal researchers to share knowledge on research
agendas set by respective steering groups. Table
1 contains a list of workshops and the number
of persons trained over World Bank fiscal years
2014–2017.
Workshops serve four main functions. First, gov-
ernment counterparts and other partners receive
instructions on IE methods and tools. This equips
them to participate actively in development and
implementation of their own IE and become bet-
ter-informed consumers of knowledge, whether
generated through IEs or other methods. Even if the
engagement were to stop here, this alone would be
valuable as policymakers are exposed to multiple
sources of information and a better understanding
of the type of evidence needed to understand caus-
al links is vital to better decision-making.
Second, participants are presented with the lat-
est rigorous evidence in each focus area. They can
then incorporate relevant findings into their own
program and policy design. For many policymakers
and practitioners, this is a rare opportunity to up-
date technical knowledge in their fields and interact
directly with top academics and peers from other
agencies and countries. Presentation of evidence,
often in engaging formats such as seven-minute
“ignite presentations”, convey key messages in a
memorable fashion, enhances cross-country learn-
ing, stimulates creative thinking and provides ideas
that may be incorporated and tested in one’s own
IE, and illustrates the feasibility of high-quality (ex-
perimental) IEs at scale in complex settings.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 4
TABLE 1: Workshops and Number of People Trained FY14–17
Workshops Date City Sector Objective Target Audience
# of people trained
Evaluating for Peace
March, 2014
Lisbon, Portugal
Fragility, Conflict, and Violence
Program develop-ment and capacity building
Project operational teams, government counterparts, researchers, and MDBs
95
Innovations for Agriculture
June, 2014
Kigali, Rwanda
Agriculture Program develop-ment and capacity building
Project operational teams, government counterparts, researchers, and MDBs
126
Energy and Environment
August, 2014
Berkeley, U.S.A.
Energy and Environ-ment
Leveraging new technologies to improve measurement
Engineers, researchers, and World Bank
60
Energy and Environment
October, 2014
Lisbon, Portugal
Energy and Environ-ment
Program develop-ment and capacity building
Project operational teams, government counterparts, researchers, and MDBs
71
Governance January, 2015
Istanbul, Turkey
Governance Program develop-ment and capacity building
Project operational teams, government counterparts, researchers, and MDBs
143
Trade and Competi-tiveness
May, 2015 Istanbul, Turkey
Trade and Competi-tiveness
Program development and capacity building
Project operational teams, government counterparts, researchers, and MDBs
139
Transport and ICT
June, 2015
Rio de Janeiro, Brazil
Transport and ICT
Program develop-ment and capacity building
Project operational teams, government counterparts, researchers, and MDBs
148
Energy and Environment
July, 2015 Chicago, U.S.A.
Energy and Environ-ment
Portfolio design follow-up
Researchers and World Bank 20
Governance February, 2016
Washington DC, U.S.A.
Governance Review of evidence and to identify emerging priorities
Government counterparts, researchers, and World Bank
179
Edutainment May, 2016 Mexico City, Mexico
Edutain-ment
Review of evidence and to identify emerging priorities
Producers and researchers from leading media organizations and universities
168
Kenya Evidence Forum
June, 2016
Nairobi, Kenya
Transport and ICT
Review of evidence and portfolio design follow-up
Practitioners, researchers, and World Bank
66
Transport and ICT Follow-Up
June, 2016
Nairobi, Kenya
Transport and ICT
Review of evidence and portfolio design follow-up
Researchers, civil society representatives, government counterparts, and World Bank
52
Evidence for Agriculture
November, 2016
Washington DC, U.S.A.
Agriculture Review of evidence and to identify emerging priorities
Practitioners, government counterparts, researchers, and MDBs
58
Distribution-al Impact Analysis
December, 2017
Washington DC, U.S.A.
Methods Methods to evalu-ate distributional impacts of pro-grams and policies
Researchers 33
IMPACT EVALUATION TO DEVELOPMENT IMPACT5
Third, each country team is matched with one or
more researchers based on compatibility of inter-
ests. Country teams are invited to workshops based
on preselection in collaboration with the relevant
Global Practices. Similarly, researchers are selected
based on their knowledge, interest, and availability
to work on long-term research collaborations with
governments. Researchers and country teams begin
their work together on the first day of the work-
shop, setting the foundation for collaboration and
compromise on what is feasible and what is ideal
for both sides. This ensures that the resulting prod-
FIGURE 2: Projects that Attend Workshops Consistently Achieve Higher Technical Scores in their i2i Proposals
Attended workshop Did not attend workshop
2.25
2.042.14
2.55
2.142.042.01
1.76
2.08 2.13
1.85 1.91
FCS Agriculture Energy &Environment
Governance T&C and F&M Transport & ICT
Technical ratings of proposals(0–3 scale)
FIGURE 3: Projects that Attend Workshops Consistently Achieve Higher Policy Scores in their i2i Proposals
Attended workshop Did not attend workshop
FCVAgriculture Energy &Environment
Governance T&C and F&MTransport & ICT
7.83 7.94 7.897.45
7.006.666.50 6.49 6.65
6.92
6.02 5.94
Policy relevance ratings of proposals by program(0–10 scale)
IMPACT EVALUATION TO DEVELOPMENT IMPACT 6
uct is of high technical quality, addresses pressing
policy questions directly, and contributes to an-
swering broader development questions.
While efforts are made to get this matching right
from the outset, in those cases where this is ful-
ly successful workshops serve as a marketplace
where both researchers and government teams
alike can find a suitable match. Successful match-
ing is critical for IEs as these are often multi-year
engagements. Using the workshop as a vehicle to
achieve this further equalizes opportunity for i2i
funding calls opened after workshops as well as
other funding opportunities where both technical
research quality and policy relevance count.
Fourth, IE teams develop the initial concept for a
prospective impact evaluation to answer questions
that have direct policy relevance to their program.
Providing dedicated “clinic” time for this during each
workshop day encourages teams to debate and in-
corporate relevant evidence, methods, and IE design
options discussed during the workshop week. Of-
ten, the IE process achieves its first policy influence
at this early stage by encouraging incorporation of
lessons learned elsewhere and testing of alterna-
tive policy designs (or “variations in treatment”). IE
teams are charged with presenting preliminary IE
designs on the final day of the workshop, which en-
sures that teams go home with a clear concept and
roadmap. Further, presentations are always done by
Case Study: IE 4 Peace Workshop, Lisbon, March 2014The first i2i workshop launched the IE 4 Peace program which, focuses on Fragility, Conflict, and
Violence (FCV). This was co-convened by three World Bank Group teams: DIME, the FCV Group (at
that point known as the Center for Conflict, Security, and Development), and the Latin America
and Caribbean Region’s Citizen Security Team. The workshop brought together practitioners, sub-
ject experts, and researchers to promote the strategic use of evaluation to inform policy and program
design and advance knowledge on key issues related to FCV under four themes. These themes—jobs for
resilience; public sector governance, urban crime and violence, and gender-based violence—became the
focus areas for the first two years of program implementation.
In total, 22 teams from across the world attended the workshop. The teams represented diverse coun-
tries such as Democratic Republic of the Congo, Colombia, Honduras, Russia, and Papua New Guinea.
Additionally, researchers from the World Bank and leading universities such as Harvard and MIT con-
tributed their expertise and worked with individual project teams on initial design concepts. Develop-
ment partners including DFID and USAID also participated, as did World Bank FCV and operational
staff, and research organizations such as Innovations for Poverty Action.
The first i2i call for proposals was launched following the workshop. Thirty-three eligible expressions of
interest (EOIs) were received and 14 projects were selected for funding. The selected projects included
several that had benefitted from training, experience and evidence sharing, and researcher matching at
the IE 4 Peace launch workshop. Several of these are now under implementation and incorporate ideas
first developed at the workshop. For example, after learning about the Becoming a Man program in
Chicago, the Honduras Safer Municipalities Project decided to incorporate cognitive behavioral therapy
(CBT) in a training and labor market insertion program for at-risk youth in high-violence municipalities.
The project is now testing variations in CBT delivery through its IE.
IMPACT EVALUATION TO DEVELOPMENT IMPACT7
government team-members. This reinforces owner-
ship and empowers practitioners and policymakers
to present the work within their governments.
From Design to Implementation
DIME IE workshops provide a platform for collec-
tive brainstorming: to create ideas and spur further
thinking. After the workshop, consistent follow up
consolidates ideas and turns them into concrete ac-
tions which, ultimately, lead to the successful com-
pletion of one or more IEs. A critical first step is to
begin securing buy-in from a wider range of stake-
holders than the workshop’s participants. This in-
cludes decision-makers in governments, World Bank
operational and country management staff, and the
IE research teams. The objective is to form a coa-
lition where everyone has a clearly defined role, in-
cluding rights and responsibilities. This is essential if
the IE is to correspond directly to country-specific
and broader policy-learning priorities; be carried out
at scale in the context of a government program;
and facilitate the use of its intermediate and final
findings as policy and program-management inputs.
An important impetus for building such a coalition
is the possibility of seed funding from i2i. While
preparing their expressions of interest (EOI), teams
build on initial concepts developed at the work-
shops by refining key IE design details including the
types of interventions and number of treatment
arms to be tested, identification strategies, sample
sizes, key outcomes, and budgets and timelines. The
i2i EOI is often the first attempt at situating the
IE within the framework of existing knowledge and
defining how it will contribute both in its immediate
and broader contexts. It is also often the first sig-
nal of policy influence of the IE research process:
75 percent of IEs contribute to rationalizing poli-
cy design by informing the design of a particular
Case Study: The Honduras Safer Municipalities ProjectThe Honduras Safer Municipalities Project participated in the IE 4 Peace workshop. Following the
workshop, the IE team and government continued to work together to refine the IE concept. The
initial concept developed at the workshop called for the evaluation of three separate interventions
targeting three different categories of at-risk youth. This was refined to focus on two categories
of youth—those still in school and those neither in school nor with steady jobs. It was proposed that
two versions of a labor-market training and insertion program be tested for each group. The variation
between the two versions related to the intensity of the cognitive behavioral therapy (CBT) component,
which was inspired by the team’s exposure to the Becoming a Man program at the workshop. This re-
search design was written into the IE concept note, which was reviewed and approved.
After the concept note review, the project underwent several challenges which could have led to the IE’s
cancellation. First, the project was restructured and its budget reduced, leading to the decision to focus
on the second category of youth—those neither in school nor with a steady job—as they are more likely
to be both victims and perpetrators of crime and violence. Second, the intended implementing partner
withdrew support for the IE, forcing the government to make the difficult choice between continuing
with the pre-established operational modality or with the IE.
The project chose the IE. This was not the easy road. In the short run, it almost certainly entailed great-
er costs. The continued implementation of this IE is testament to the strength of DIME’s demand-driv-
en approach to knowledge generation. The IE implementation is ongoing.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 8
intervention (or “treatment”), based on existing
knowledge, or by introducing variations in existing
interventions or entirely new interventions to be
tested. While these may have been discussed at the
workshop, their inclusion in EOIs signals a broader
consensus to go ahead.
Once seed funding is secured, IE and operational
teams work towards fully defining the research
design and documenting this in a concept note.
The concept note is subject to peer review for
both technical quality and policy relevance. Fol-
lowing quality clearance by i2i reviewers, a review
meeting is held to discuss the technical, operation-
al, and policy implications of the proposed work.
This meeting is chaired by the relevant World Bank
country management unit. This validates the policy
relevance of the proposed IE, informs a broader set
of stakeholders, and contributes to ensuring this is
factored in as part of the Bank’s support to a par-
ticular country.
The completion of the concept note review is the
final step in the IE design process and represents
a formal commitment by all parties—the govern-
ment, World Bank, external researchers, and devel-
opment partners—to conduct and complete the IE.
It is often followed by the deployment of a field co-
ordinator (if such a person is not in place already),
who serves as an in-country liaison for the research
team and whose role is to support the government
on all aspects of IE implementation; including op-
erational planning, supervision, data collection, and
dissemination.
2.3 Collecting Data and Setting-up Data InfrastructuresHigh-quality data is a hallmark of the i2i portfolio.
i2i research teams provide technical assistance for
data collection throughout the lifecycle of the im-
pact evaluation. At the initial IE workshop, teams
design a data strategy aligned with their project
cycle and discuss key points of influence. A typical
data strategy includes both in-depth surveys and
routine monitoring data.
The foundation of the impact evaluation analysis is
high-quality microdata from multi-module surveys
of a representative sample of the target popula-
tion. These surveys are completed at key influence
points during the IE lifecycle. Typically: before proj-
ect implementation (baseline survey), midway to
project completion (midline survey), and at project
FIGURE 4: Data in the IE Lifecycle
BaselineSurvey
ProjectImplementation
MidlineSurvey
Mid-CourseCorrections
RevisedImplementation
EndlineSurvey
FinalIE Results
IMPACT EVALUATION TO DEVELOPMENT IMPACT9
closure (endline survey). When developing a data
strategy, the researchers discuss with the project
team how to align data collection to positively im-
pact project design and implementation.
Essential to the i2i data strategy is to provide data
early and often throughout the project cycle. This is
in sharp contrast to a project-evaluation model in
which a team of evaluators arrives after a project is
completed to assess whether or not the project was
successful. The i2i model aims to provide real-time
feedback and actionable information on how to
improve. If results are not as expected at midline,
there is the opportunity for mid-course corrections
and additional learning-by-doing.
Effecting Decisions through Better DataBaseline Surveys: While not technically necessary
for randomized control trials (the majority of the
impact evaluations in the i2i portfolio), baselines
are an excellent opportunity to provide government
ministries with high-quality sector-specific data,
which is almost never otherwise available. If timely,
this data can provide valuable input to project de-
sign and implementation.
IE Example: The Impacts and Sustainability of Irrigation IE in Rwanda provides a useful exam-ple of how baselines can influence project design. A baseline survey, conducted on a sample of farmers cultivating within the irrigation areas, provided the project team with detailed data on farm practices in the targeted area. This was well-timed to have influence: the team was in the midst of designing interventions to comple-ment the irrigation infrastructure. For example, the project team had planned to collect fees by automatic deductions from sales to coopera-tives. The data revealed that less than 2 percent of farmers made any sales to the cooperative. This forced a change in strategy. In addition, the data provided representative statistics on veg-
etable cultivation, which the project sought to promote. Realizing how few farmers had experi-ence cultivating vegetables (5.3 percent) influ-enced the structure and intensity of agricultural extension.
Midline Surveys: Implemented midway through proj-
ect implementation, at a point when the project is
expected to have achieved initial gains. The midline
survey is a critical mechanism to improve project
implementation and assure that projects meet or
exceed their development objectives. Discussions of
findings of the midline survey hinge around poten-
tial mid-course corrections, and possible new exper-
imental implementation variations.
IE Example: Rwanda Land Husbandry, Water Har-vesting, and Hillside Irrigation Project IE. There are two primary agricultural seasons in Rwan-da, known as Season A (long rains) and Season B (short rains). A midline survey found that the project had met and, in fact, exceeded its ob-jectives in increasing agricultural productivity, commercialization, and farm incomes in Season A. However, they were significantly behind tar-get on Season B. As a result, the team increased their focus on that season, shifting agricultural extension efforts and developing complementary interventions, such as an experiment testing new savings products for farmers.
Endline Surveys: Completed at the end of project im-
plementation. The objective is to capture the full
lifecycle impact of the project and measure gains
from mid-course corrections by comparing indica-
tors to the midline. The research team presents pre-
liminary findings to the project team in a dissem-
ination mission and, after incorporating feedback,
prepares an impact evaluation report and policy
brief. A primary objective of the dissemination mis-
sion is to discuss policy implications of the impact
evaluation findings, particularly opportunities for
scaling up or down. The next chapter discusses pol-
icy implications in detail.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 10
Building a Data InfrastructureThe i2i data strategy aims to create a comprehen-
sive data infrastructure by integrating with the
project-monitoring system, utilizing existing ad-
ministrative data and testing new measurement
technologies. The end objective is to create a data
infrastructure that is informative, allows for timely
responsive action, and sustainable beyond the dura-
tion of the specific project being evaluated.
Monitoring & (Impact) Evaluation: Rather than having
parallel systems—the project’s M&E and IE data—
i2i IEs strive to fully integrate the two. Monitoring
systems are designed to use the same identifica-
tion codes as the impact evaluation, so that data
can be easily merged and compared, and key indi-
cators (for example, agricultural yield, firm profits,
and household income) are constructed in a consis-
tent manner. Each i2i IE team includes a field coor-
dinator, who is based in-country and typically sits
with government M&E staff. The field coordinator
is primarily responsible for technical assistance on
both in-depth surveys and monitoring data and fre-
quently conducts trainings on data management in
Excel and statistical software (for example, Stata,
SPSS) for government counterparts.
IE Example: The Kenya Patient Safety IE provides an excellent example of creating a data infra-structure, fully integrating monitoring and impact evaluation data, and filling an important data gap. The monitoring system includes: (i) data on planning and progress of the inspection pilots (for example, are inspections taking place?); (ii) inspec-tion results at the facility and aggregate levels for each pilot (for example, how are facilities performing in each intervention?), and (iii) third-party monitor-ing indicators to assess quality of intervention and protocol adherence (for example, what is the quality of the inspection delivered?). This customized solution then leads to the availability of timely and actionable information to identify challenges in the implementation and enhance accountabil-
ity to make mid-course corrections; without the intensive use of resources, expertise or equipment commonly absent in poor-resource contexts.
Administrative Data: Many of the i2i impact eval-
uations also incorporate existing administrative
data into their data infrastructure. Governments
typically have large amounts of existing data, but
is often in hard copy only or lacks identification
codes, inconsistently structured, or not centralized
at the national level. As a result, integrating this
data requires trips to field offices, digitization, and
painstaking efforts to merge on available variables.
However, the gains can be substantial, and it has
the positive externality of creating a useful data in-
frastructure sustainable beyond project completion.
IE Example: Building a Supportive Environment for Operation and Maintenance in the Tanzanian Rural Water Supply Subsector IE. To understand the extent of interactions between government employees (and particularly water technicians) and village citizens, the team obtained records from visitor logbooks in rural villages in the proj-ect area. After consultation with the village chief, surveyors took photos of each page of the log-book, which were then digitized. This provides a rich source of information on engagement of ru-ral communities with the government and NGOs, which had not been previously utilized or available beyond the village.
“Big” Data: The next frontier in building responsive
and sustainable data infrastructure is to establish
the most cost-effective ways to monitor project
outcomes by testing various indicators and mea-
surement technologies. Household surveys are
time-intensive and very expensive, which limits the
potential for large-scale or high-frequency data
collection. Big data has the potential to be a more
cost-effective alternative. Recent technology devel-
opments have dramatically increased data avail-
ability and processing capacity. The task at hand
is to apply these to program evaluation, and rigor-
IMPACT EVALUATION TO DEVELOPMENT IMPACT11
ously test how well big data can be used to mea-
sure key outcomes of interest by comparing to gold
standard methods and carefully comparing costs.
IE Example: the Nairobi smarTTrans IE will de-velop a validated measurement framework for driving and road safety and a set of instruments and crowdsourcing methodologies that can be de-ployed in Kenya and other low-income settings. The IE builds an ICT-based monitoring system, leveraging the high penetration of mobile phones and low-cost technologies. These include: 1) vehi-cle-based technology (sensors, GPS trackers); 2) app-based interfaces for owners and drivers to learn about road safety standards; 3) direct feed-back mechanisms for drivers on performance and ways to improve their driving; and 4) a hotspot and entertainment box combined with SMS and Android platforms for consumer feedback on driver behavior. The technologies will be integrat-ed into a big data infrastructure that will provide information to improve decisions by all actors in the urban transit system, from policymakers and regulators to private insurers, operators, drivers, and riders.
Ensuring High-Quality Data
Project teams in the i2i portfolio finance all data
collection. i2i provides technical assistance to en-
sure high-quality data at each stage of the process:
developing the terms of reference for a survey firm,
designing the survey instrument, participating in
FIGURE 5: Elements of a Big Data System
Data CollectionMethods
Storage Space &Analysis Tools
ExperiencedData Analysts
BIG DATAINFRASTRUTURE
FIGURE 6: i2i Technical Assistance During Data Collection
• Design representative, well-poweredsampling strategy
• Advise on survey firm TORs
• Design questionnaire, in consultation withproject team
• Pilot questionnaire• Program CAPI questionnaire
• Review field plan• Conduct surprise spotchecks in the field• Check data quality on a daily basis, establish
feedback loops with the project team tocorrect problems
• Review final dataset for completeness• Clean data• Analyze data, according to analysis plan
developed jointly with project team• Contribute to policy briefs and reports
Survey Preparation
• Review enumerator training manual
• Participate in enumerator training
Questionnaire Design
Enumerator Training
Fieldwork
Survey Completion
IMPACT EVALUATION TO DEVELOPMENT IMPACT 12
enumerator training, supervising field work, and
checking data quality and alerting the team about
any issues in real time. In this way, i2i achieves
high-quality data and builds capacity for high-qual-
ity data collection within the government ministry.
Figure 6 outlines the main areas for technical assis-
tance in a typical survey.
i2i surveys are typically computer-assisted person-
al interviews (CAPI), with exceptions only in case
of concerns of enumerator safety or extreme in-
frastructural limitations (for example, electricity,
internet, and transport). The i2i team typically pro-
grams the questionnaire to ensure that program-
ming meets the highest quality standards: CAPI
technology has the potential to greatly increase
data quality, but only if carefully programmed.
All i2i surveys incorporate: automated skip codes,
range restrictions, internal consistency checks,
pre-populated identification information for fol-
low-up rounds of panel surveys, and validation
before submission (for example, that all expected
fields are completed).
The field coordinators are the primary bulwark
against poor data quality. They work closely with
the project teams and survey firms on a day-to-day
basis, with support from the IE researchers. Recog-
nizing their critical role, i2i supports an annual field
coordinator workshop, in which all field coordinators
come together as a community of practice, to be
trained on managing high-quality surveys, learn the
latest developments in best practice protocols, and
build skills on relevant survey programming and da-
ta-analysis software packages.
2.4 Generating Evidence and Motivating ChangeDIME’s ambition is to use rigorous evidence to mo-
tivate policy change in the world’s poorest coun-
tries. At a minimum, this requires connecting les-
sons from our evaluations to new policy decisions.
However, DIME’s evaluation model aims to embed
learning into each element of the project cycle,
from defining policy, deciding whether to continue a
program or not, to the structure of the next phase
of learning. Given its continuous nature, our model
is a form of ‘real-time’ learning.
In traditional evaluation models (such as that
summarized in figure 7), the evaluation is done
after the program is completed. Policies inform
designs, which are then evaluated, and the re-
sults are used to decide whether the program
should continue or not. In the best cases, eval-
uation reports aim to distill wider learning from
the efforts of evaluation, but there is no strate-
gic linkage between that learning and the poli-
cy-formation process. This could even be true if
an impact evaluation is embedded in the project.
Figure 7 represents a model where the evaluation
team is seen as separate from the implement-
ers, independently receiving data and reporting
results in the final stages.
Real-time learning implies that we should aim to
undertake a circular model, where learning is an
integral part of each stage of policy development.
Figure 8 connects learning from previous project
cycles to new ones. However, DIME’s ambition is to
embed circularity and feedback loops in each stage
FIGURE 7: The Traditional Evaluation Model
POLICY DESIGN EVALUATE CONTINUE? LEARNING
IMPACT EVALUATION TO DEVELOPMENT IMPACT13
of the project cycle. From the anti-clockwise arrows
in figure 8, we see learning as feeding into policy de-
sign and also the decision as to whether to contin-
ue a program and how we should modify our plans
for the project evaluation. Similarly, discussions
around the continuation of the project are about
optimal design. Each stage of the project cycle can
be designed to feed back into any other.
Embedding such learning in each stage of the cycle
requires strong partnerships between us and our col-
leagues at the World Bank, other multilateral agen-
cies, and our government counterparts. Sections 2.1
and 2.2 explored how we build those partnerships
with our colleagues and clients. This section will
explore how we work with those partners at each
stage of the project cycle to effectively generate ev-
idence and motivate evidence-based change.
PolicyAs discussed in section 2.1, we aim to engage
with projects at an early stage of development.
Preferably, when the policies under which they
fall are still being developed. This allows us to en-
sure that policies reflect frontier evidence from
academic and policy research. This evidence
identifies both what is known, and what still
needs to be tested, so that evaluation can be
embedded into policy itself. 90 percent of DIME
projects have facilitated the embedding of fron-
tier research into government policy. This hap-
pens in both ad hoc and more systematic ways.
For example, DIME’s Fragile and Conflict States
group worked with a series of senior academics
and Bank staff to write white papers summariz-
ing frontier literature of the sector. These white
papers ensure ready access to frontier research
lessons and corresponding gaps for project
teams working in the concerned areas.
Sometimes, relevant evidence is not available in the
research literature. DIME works with government
counterparts and our colleagues to develop policies
that reflect these ambiguities. The change we hope
to inspire is a recognition in country policies that
the right path for a country is not currently obvious,
and so different interventions will be experiment-
ed with until there is sufficient evidence to choose
a single path, or expand multiple interventions to
those communities where they fit best.
We also work to generate evidence to inform policy.
In Tanzania, DIME works with the United Kingdom’s
Department for International Development (DFID)
to improve the maintenance of water infrastruc-
ture in rural areas. DFID’s approach to this project
is to pilot an ‘adaptive’ model. Rather than contract
DIME to undertake a single evaluation formulated
at the start of the project, DFID has written mul-
tiple stages of review into the contract so that
its policy on water maintenance in Tanzania can
be updated. Inspired by academic models of poli-
cy development such as ‘Problem-Driven Iterative
Adaptation’, this model ensures that DFID’s policy
constantly reflects the most up-to-date informa-
tion and insights from the field.
FIGURE 8: DIME’s Evaluation Model
POLICY
DESIGN
EVALUATECONTINUE?
LEARNING
IMPACT EVALUATION TO DEVELOPMENT IMPACT 14
DesignInterventions are designed, based on policy. While
policy sets the broad parameters of intervention,
there are typically large areas of ambiguity in how to
design a particular project or program. This is where
a circular, or real-time, learning model facilitates
evidence-based decision-making. Working with re-
searchers allows operational staff access to frontier
research on project design. Similarly, working closely
with operational staff allows researchers to identify
research that is ‘operationally-relevant’. Both sides
of the research-policy divide can best understand
the other’s perspectives when jointly designing a
project. That is what DIME strives to do.
For example, working closely with the Rwandan Gov-
ernment had given DIME’s agriculture team a repu-
tation for being easy to work with. The government
was keen to operationalize its agricultural strategy
and requested DIMEs support. DIME worked with
the government to design, introduce, and test inno-
vative farmer-feedback tools. Where it was not ob-
vious how to design the tools, DIME helped the gov-
ernment trial multiple modalities. DIME’s research
background also allowed for an additional twist to
be added to the project so that the evaluation could
learn about the underlying mechanisms driving the
results. It found that particular feedback tools in-
creased attendance in agricultural extension train-
ings and resulted in increased adoption of superior
farming techniques among non-users. The most
cost-effective feedback mechanism (a hotline) was
adopted and scaled up throughout Rwanda. In ad-
dition, the satisfaction data collected as part of the
evaluation convinced the government to continue
supporting a public-private partnership that had
supported the implementation. The ongoing policy
of the government reflects frontier research.
Evaluation
Embedding an evaluation into any part of the proj-
ect cycle allows for learning that can strengthen
the project and provide broader lessons for policy
around the world. Such evaluations can take many
forms, both across and within projects. In Gha-
na, DIME worked with the Office of the Head of
the Civil Service to survey all civil servants on the
bottlenecks to improved service delivery. As dis-
cussed in section 2.3, rigorous collection of care-
fully designed indicators can be sufficient evidence
to motivate change by itself. The results of the civil
servants survey showed that a series of obvious re-
forms could be implemented immediately. For ex-
ample, officials complained that organization heads
were not being monitored as was intended by the
Public Service Rules. The Head of the Civil Service
immediately fixed the monitoring system. Similarly,
in Kenya, DIME worked with the judiciary to build
a ‘Daily Court Returns Template’ to scientifically
gather and organize daily court output. This was
both legislated into national law and was a useful
data-collection tool for the IE.
DIME frequently embeds a preliminary evaluation in
its first year of engagement with a project. Using
the best available data, or that which can be col-
lected immediately, DIME researchers aim to iden-
tify whether there is an empirical basis for an evalu-
ation approach. In the Ghana case, there were areas
that all civil servants stated were working well, with
no need for reform. These didn’t seem the first
places to test for significant constraints to pro-
ductivity. There were others that required further
Many of our government and implementation partners require an answer to whether
an intervention should be continued or not.
IMPACT EVALUATION TO DEVELOPMENT IMPACT15
investigation, focusing the scope of our evaluation,
and changing the reform policy of the government.
Using a multitude of impact evaluation methods al-
lows DIME researchers to shine a light on areas of
reform that the government has been considering,
and those they had not yet conceptualized. These
discussions are then the basis to identify a series
of reform options that DIME typically evaluates us-
ing a randomized control trial. In 88 percent of our
Randomized Controlled Trial evaluations, we look to
go beyond a simple understanding of which flavor
of intervention works best to why it is most suc-
cessful. In Ghana, we are now using an RCT to build
a more effective training system for public officials,
something they highlighted as a major constraint in
the survey.
ContinuationMany of our government and implementation part-
ners require an answer to whether an intervention
should be continued or not. Public-sector funding is
constrained across the world, so allocating program
budgets efficiently is of importance to the effec-
tiveness of the state. DIME aims to provide inputs
to answer this question, but with a focus on how
to generate the largest gains from a program. By
isolating the best way to deliver an intervention, we
allow ourselves to reformulate its design and evalu-
ate it on its greatest strengths.
In Kenya, DIME worked on supporting regulatory re-
form in the health sector. It supported a review of
the health inspections regime for the country. Re-
sponding to the government’s question of whether
health inspections were worthwhile, DIME research-
ers supported the passing of an enhanced regula-
tory framework for health inspections and the de-
sign of a toolkit of instruments to measure patient
safety. The database created through these efforts
allowed the team to identify multiple inspection
regimes that could be trialed in an IE framework.
Thus, the project is now looking at how to maximize
the impact of patient monitoring.
LearningThis subsection of the report has outlined how
evaluation teams can organize projects to learn in
‘real-time’. There is often a point in an evaluation
cycle that is seen as an opportune time to reflect
substantively on the learning process and aggre-
gate lessons to date. In the traditional model, this
would be the point at which the evaluation report
was produced. In the DIME model, this can be at
various points throughout the lifetime of a single
evaluation or on completion of multiple comple-
FIGURE 9: IE on Real-Time Learning
IEs that provided:
Better Data Improved M&E57%
High-Quality88%
High-QualityEndline
96%
Access toKnowledge
Program LaunchWorkshops
63%
BaselineDiscussions
69%
IE ResultsDiscussions
92%
At Design At Implementation For Continuation. . .
mentary evaluations. DIMEs organization around
thematic groups allows it to strengthen the lessons
of any single evaluation by relating it to the learn-
ing from a range of others.
Working closely with the Nigerian Government on
its healthcare sector for almost a decade, DIME
undertook real-time learning in partnership with
the Ministry of Health. We introduced variations in
policy design informed by existing evidence on in-
terventions in the sector. Generating a wealth of
microdata, DIME supported the delivery of routine
program elements (such as working to improve the
timeliness of payments to frontline health work-
ers) and fed the resulting data into decision-mak-
ing across the health sector. However, to capital-
ize on the multitude of evaluations in the sector,
DIME organized an ‘Evidence and Action’ workshop
that presented senior stakeholders with the results
of all the evaluations in one go. This facilitated a
‘sector-wide approach’ to evaluation that enhanced
the learning of any single evaluation by allowing ev-
idence from one project to be cross-checked and
validated by another. DIME also benefited from this
exercise by allowing us to conceptualize the struc-
ture of learning that would best fit our next tranche
of work with the Nigerian Government.
Undertaking policy making in the developing world
can be a daunting experience. DIME has strived to
enable our partners to have access to better data,
knowledge, and learning at each stage of the proj-
ect cycle. Providing continuous feedback to imple-
menters that we work with also means that we are
learning in real time ourselves. Our model is chang-
ing and evolving with each new partnership and
evaluation that we undertake.
IMPACT EVALUATION TO DEVELOPMENT IMPACT17
3. Selection and Quality Assurance for Policy Relevance and Technical Rigor
As described in previous chapters, DIME’s goal is
to increase the use of impact evaluation evidence
in the design and implementation of public policy
across the developing world. It works with programs
at scale on game-changing issues to answer policy
questions that clients identify and to increase de-
velopment outcomes. The highest technical quality
and policy relevance is ensured at all stages of proj-
ect cycles, while maintaining a flexible environment
for adaptation based on real-time evidence and
changes in context and client needs.
How does DIME achieve this? It does so through an
innovative operating model with a bottom-up ap-
proach, transferring IE knowledge and tools to clients
and matching them with internationally renowned
technical experts to deliver the highest quality prod-
ucts of policy relevance (see chapters 2.1–2.4). Client
engagement from the early phases of the design
ensures relevance of policy questions, government
buy-in, feedback loops, and policy action through al-
most all phases of the project cycle. DIME projects
report policy influence at baseline, during implemen-
tation, and scaling up or down based on final results
(see chapter 8 on Cases of Policy Influence). This
approach, which is significantly different from con-
ventional development research, has defined DIME’s
model over the last 12 years.
But why this approach? Because it overcomes
critical challenges of traditional evaluation and re-
search. As chapter 2.4 discusses, to be done well,
FIGURE 10: DIME’s Operating Model
Syst
emat
ic o
f evi
denc
e
Inform policydesign Identify
knowledge gaps &Set sector priorities
Build capacitythrough IEworkshops
Open callsfor proposals
Traininggovernment
clients and othersin IE methods
Guide policyimplementation,make mid-course
corrections
Review &Select proposals
Provide financial& technical
support for IEs
Monitorimplementation
Match teamswith researchers
and subjectexperts
Provide policyfeedback to
inform adoptionand scale-up
Conduct analysis& Generate
results
Disseminatefindings
Improveeffectiveness of
development policy
Generate actionable results
and empowergovernments
Capacity building
1
5
2
6
3
4
8 97
IMPACT EVALUATION TO DEVELOPMENT IMPACT 18
IE requires strong technical expertise. This is why
researchers tend to secure most research funds
that come through various development channels.
However, due to a lack of client engagement in the
design of research, policy questions addressed and
results are often questionable in terms of their
policy relevance and there are usually no feedback
loops to generate policy action. DIME was built pre-
cisely to overcome this barrier and serve as a bridge
between researchers and policymakers. Its the-
matic programs start by defining broader research
priorities through close consultation with the re-
search community and development partners, in-
cluding World Bank’s global practices and regions
(see chapter 2.1). It then identifies operationally
relevant programs that enable impact evaluations
around critical policy issues. It engages operational
teams and government counterparts; starting with
a workshop to build capacity, raise awareness, and
form partnerships with technical experts, setting
the foundation for longer-term collaborations and
policy influence (see chapter 2.2-2.4).
Developing the Program in Energy and EnvironmentDIME’s Energy and Environment program began in June 2014, following an iterative approach
to balance the objective of addressing prioritized knowledge gaps with operational realities. The
process included direct engagements with the Bank’s Climate Change Cross-cutting Solutions
Area, Energy and Extractives Global Practice, Environment and Natural Resources Global Practice,
Water Global Practice within the World Bank, the Global Environment Facility (GEF), the Climate Invest-
ment Funds (CIF), the Energy Sector Management Assistance Program (ESMAP), and DFID (including
the evaluation department and climate-change teams).
These partners actively contributed to the identification of evidence gaps, mostly from an operational
standpoint, and a portfolio of potential projects aligned with those gaps. The main research partner-
ship was with the University of Chicago led by John List (Chairman of the Department of Economics).
Other collaborations on measurement and technology include partnerships with the University of Cali-
fornia at Berkeleyand the Centre for Effective Global Action (CEGA). These partners contributed to the
knowledge agenda by summarizing and discussing the current stock of academic literature and future
research priorities.
A kick-off workshop, held jointly with CEGA in August 2014 in Berkeley, brought together engineers, econ-
omists, and World Bank counterparts to explore leveraging new technologies to improve measurement
in energy and environment projects and research. A follow up measurement workshop, with a focus on
innovative measures for climate resilience, was held in June 2015. The main workshop, and official launch
of the program of work in Energy and Environment, was held in Lisbon in October 2014 bringing together
19 project teams (financed through DFID, the GEF, CIF, IDA, and IBRD) and 28 researchers from 11 aca-
demic institutions to refine research opportunities based on project interest and operational feasibility.
Representation from governments typically included high-level policymakers, project implementers,
and/or monitoring and evaluation specialists. For instance, from Bangladesh, participants were the
Project Director, the Project Deputy Director, and the Executive Engineer of the government’s Rural
Electrification Board of the Bangladesh Rural Electricity Transmission and Distribution project. From
IMPACT EVALUATION TO DEVELOPMENT IMPACT19
At the project level, the DIME model generates
evidence throughout the policy lifecycle. As chap-
ter 2.4 describes, at the design phase, DIME IEs
strengthen economic theory of interventions and
make improvements based on existing evidence.
During baseline and follow-up, data is used to stim-
ulate policy dialogue and sometimes to make mid-
course corrections. During implementation, the
IE strengthens client M&E systems and develops
high-quality survey instruments and data-collec-
tion protocols. During analysis and results, teams
fine tune country policies and programs based on
evidence, often motivating scale-up or down. Re-
sults are also shared more broadly with the inter-
national development community through pub-
lications, seminars, workshops, and face-to-face
interactions.
Applications for IE work are submitted to DIME
through calls for proposals, usually announced after
DIME workshops and targeting teams participating
in workshops as well as the development commu-
nity at large. All submissions, both at the EOI and
concept note stage, undergo rigorous technical and
policy relevance reviews. External technical experts,
identified for their IE and subject matter expertise,
score proposals on a set of technical criteria through
a blind review process. Internal Bank GP and regional
focal points score the proposals on policy relevance
and feasibility of implementation. All submissions,
review scores, and ranking of the submissions by
scores are submitted to the i2i technical commit-
tee. The technical committee, which comprises se-
nior and lead economists from the Bank’s Research
Group, makes selection decisions based on technical
the Argentina Renewable Energy in Rural Markets Project, representation included the coordinator gen-
eral, electrician engineer, and M&E specialist from the Energy Secretariat of the Ministry of Energy and
Mining. During the week-long workshop, project teams spent time with researchers to develop proposals
relevant to their specific projects. They presented their achievements on the final day of the workshop.
Two months after the workshop, in July 2015, researchers participating in the program were invited to
present initial designs and receive feedback during a session chaired by John List and supported by Mi-
chael Greenstone (Director of the Energy Policy Institute) at the University of Chicago. The first round
of projects presented concepts after initial scoping missions that led to the refinement and finalization
of concept notes. Nine projects developed through this process are currently ongoing, seven passed
the CN stage and are currently in the implementation phase. Client engagement continues through all
stages of the impact evaluations. Examples include joint analysis and discussion of the baseline data
to develop technical capacity in the government and inform intervention design.
FIGURE 11: i2i Review Process
IE teams submitproposals
Technical committeeapproval
Bank's review andmanagerial approval
Monitoring andreporting
External technicalreview
Policy relevancereview
IMPACT EVALUATION TO DEVELOPMENT IMPACT 20
and policy score, overall feasibility of implementa-
tion, and capacity to target important knowledge
gaps. The teams that pass the EOI stage receive
a preparation grant from i2i ($25,000) to devel-
op a full technical proposal. The teams that pass
the concept note stage receive an implementation
grant from i2i ($150,000 over three years).
After the concept notes receive i2i approval, they
undergo the Bank’s internal quality assurance pro-
cess, involving a separate review meeting, chaired
by country or Global Practice manager, and incor-
porating review feedback from at least two peer
reviewers, usually a subject matter expert and an
operations expert. This process of combining i2i’s
and the Bank’s review processes ensures technical
quality, buy in from the client, and ongoing rele-
vance to World Bank and country policy priorities.
In some cases the Bank’s regions rank a proposal
highly on policy relevance and there is a strong com-
mitment from the client to do the evaluation, but
the technical evaluation from the external review
does not meet i2i technical standards. Here, DIME
provides technical expertise to build capacity and
revise the design. For example, Colombia Mobile Vic-
tims Unit Impact Evaluation, selected through the
Fragility, Conflict and Violence funding window, re-
ceived very high policy relevance ranking but did not
pass the i2i technical review. With demand from the
Bank project team and clients, DIME assigned two
researchers to this team to think through the criti-
cal aspects of the design and revise the methodol-
ogy. The revised proposal was submitted to i2i for a
second review through external reviewers and then
went back to the technical committee, after which it
was approved and included in the i2i portfolio.
During implementation, i2i-supported IEs report an-
nually on progress and potential channels through
which they affect capacity and influence policy
decisions throughout the cycle of implementation
(see chapter 7.1 monitoring). On completion, final
product reports and working papers are submit-
ted to i2i for technical review, following a similar
FIGURE 12: Bank’s Review Process
IE teams submit proposals IE team revises andresubmits
Decision meeting chaired by manager
or country director
Peer review by at leasttwo internal reviewers
Wider circulation in the GPand region for feedback
FIGURE 13: Quality Assurance during Project Cycle
1Policy Design
2Baseline and
Follow-up
3Implementation
4Analysis and
results
• Concept Notetechnical andpolicy relevancereview
• Technical reviewof all surveyinstrumentsand protocols
• Annual in-depthprogress updatethrough MyIEmonitoring system
• Final product/IEreport and papertechnical review
IMPACT EVALUATION TO DEVELOPMENT IMPACT21
review process from the concept note stage. After
i2i clearance, teams proceed with the Bank’s review
of the final product, followed by delivery to client
and publication. Crucially, all data is to be made
available for public use through the World Bank’s
Microdata library.
Review Criteria for Expressions of Interest n Clarity of research questions and potential to contribute to evidence gaps
n Prioritized project components/interventions have logical pathways to intermediary and final
outcomes
n Credible identification strategy for each research question
n Potential for learning, for example, by including multiple treatment arms
n Clearly defined targeting and recruitment of participants, and adequate number of participants to
implement proposed analyses
n Feasibility of implementation (sample size, intervention, selection of beneficiaries, and country
context)
n Evidence of partner engagement and support
n Potential to influence the design and/or prioritization of current and future development
interventions
n Potential to influence policy design and/or scale-up.
Review Criteria for Concept Notes n Hypotheses and research questions clearly linked to the theory of change and relevant to
important research and/or policy questions
n Main outcomes of interest are relevant to answering research questions and are feasibly gathered
n Evaluation design and sampling strategy:
¡ identification strategy well explained and defines a credible counterfactual
¡ design presents no ethical issues, or if it does, mitigation measures are highlighted
¡ sufficient detail on sample size/power calculations provided for each of the primary research
questions, given available data.
n Details on data collection instruments; the data collection strategy is thought out and feasible and
includes information on ethical clearance
n IE management, research team, and implementing partners have sufficient capacity to carry out
the proposed research and proposed budget is realistic and represents research value-for-money.
Additional Requirements before i2i Funds are Transferred n Co-financing of at least USD 10,000 per year (for each year of i2i support)
n Trial registration of the research design prior to initiating a study
n Ethical approval of the IE if it involves human subjects
n Ethical training certification for researchers.
IMPACT EVALUATION TO DEVELOPMENT IMPACT23
4. Delivering on Our Promises
i2i has met and over-delivered its targets for FY17.
The i2i results framework in annex 1 shows prog-
ress towards all targets as agreed with the donor.
The rest of this chapter provides a description of
some of the main deliverables of IE products, ca-
pacity-building workshops, and policy influence.
PortfolioThe i2i portfolio has a total of 131 IEs across 53
countries, covering all regions and i2i thematic ar-
eas. Figure 14 below shows the organization of i2i
topics around thematic pillars. All targets for re-
gional and thematic area distribution have been
met. The portfolio was developed through close
collaboration with the Bank’s Global Practices and
calls for proposals during the first two years of i2i
implementation. In FY15, four calls for proposals
were completed for new IEs in i) Fragility, Conflict
and Violence, ii) Agriculture, iii) Energy and Environ-
ment, and iv) a separate call for ongoing IEs across
FIGURE 14: i2i Thematic Pillars
1
5
2
6
3 4SharedProsperity• Finance and
Private Sector• Agriculture• Infrastructure
Governance• Public Sector
Governance• Justice• Local Development
ClimateChange• Energy• Environment• Natural Resource
Management• Agriculture• Transport
HumanDevelopment• Education• Health• Social Protection
Gender• Human Capital• Economic
Opportunities• Voice/Agency
Fragile andConflictSituations• Economic/social
reintegration• Governance• Gender-based violence• Urban crime and violence
Cross-Cutting Themes
IMPACT EVALUATION TO DEVELOPMENT IMPACT 24
sectors (including proposals at various phases of
IE implementation). In FY16, three additional calls
were completed for new IEs in i) Governance, ii)
Trade and Competitiveness and iii) Transport and
ICT (see chapters 2.1, 2.2, and 2.4 for more back-
ground on thematic program development, work-
shops, and selection of proposals). Figures 15 and
16 show the distribution of the portfolio by region
and thematic areas.
In addition, targets for gender and fragile and
conflict effected situations have been met as well.
Gender and FCS are core themes of i2i work. As
figures 17 and 18 show, at least 17 percent of the
portfolio evaluates a gender-specific intervention
and 57 percent conducts disaggregated gender
analysis. 17 percent of the portfolio is in FCS coun-
tries, and 28 percent in the FCS sector.
In terms of design, 73 percent of the IE portfolio
adopts experimental methods, 14 percent uses both
FIGURE 15: Distribution of the i2i Portfolio by Region
IEs by RegionNumber (percentage) of i2i IEs
AfricaLatin America and
the Caribbean
South Asia
East Asia and PacificEurope and
Central AsiaMiddle East and
North Africa
Global
69 (53%)
24 (18%)
17 (13%)
9 (7%)
8 (6%)
3 (2%)
1 (1%)
The sum percentages may not be equal to 100% due to rounding off.
FIGURE 16: Distribution of the i2i Portfolio by World Bank Global Practice
IEs by Global PracticeNumber (percentage) of i2i IEs
Governance
Agriculture
Trade andCompetitivenessSocial Protection
and Labor
Transport and ICTs
Water
Finance and Markets
Energy andExtractives
Social, Urban, Rural,and Resilience
Health, Nutritionand Population
Education
Poverty
The sum percentages may not be equal to 100% due to rounding off.
4 (3%)
2 (2%)
Environment andNatural Resources
36 (27%)
20 (15%)
19 (15%)
12 (9%)
11 (8%)
7 (5%)
7 (5%)
5 (4%)
5 (4%)
2 (2%)
1 (1%)
FIGURE 17: Distribution of the i2i Portfolio by Gender
IEs Including a Gender AnalysisNumber (percentage) of i2i IEs
IEs Evaluating a Gender-SpecificIntervention
Number (percentage) of i2i IEs
IEs Falling under the GenderCross-Cutting Solution Area
Number (percentage) of i2i IEs
30 (23%)
98 (77%)73 (57%)
56 (43%)
107 (83%)
22 (17%)
Yes No
IMPACT EVALUATION TO DEVELOPMENT IMPACT25
experimental and non-experimental, while another
14 percent has non-experimental design. In terms
of multiple treatment arms, which test project in-
novations, around 65 percent include more than
one treatment arm.
Progress of portfolio implementation is also on
track and i2i meets its target for the number of
competed IEs. As of January 2016, 24 IEs reported
as completed and have final outputs such as work-
ing papers, publications, or project completion re-
ports. There are an additional seven IEs that report
having completed the analysis and working towards
producing the final output. Further, 43 percent of
the portfolio is in the implementation phase. These
have passed technical and policy relevance reviews
from i2i, completed World Bank concept note review,
and received country director and sector manager
FIGURE 18: i2i Portfolio in Fragility, Conflict, and Violence
IEs Fragile and Conflict-affected SettingsNumber (percentage) of i2i IEs
IEs Fragile and Conflict-affected CountriesNumber (percentage) of i2i IEs
36 (28%)
22 (17%)
108 (83%)
94 (72%)
Non-FCS Countries FCS Countries
Non-FCS Sector FCS Sector
FIGURE 19: Distribution of the i2i Portfolio by Evaluation Method
IEs by Evalution MethodNumber (percentage) of i2i IEs
18 (14%)
18 (14%)
95 (73%)
Experimental Non-experimentalBoth experimental and non-experimental
FIGURE 20: Distribution of the i2i Portfolio by Number of Treatment Arms
IEs by Number of Treatment Arms*Number (percentage) of i2i IEs
32 (36%)29 (32%)
21 (23%)
8 (9%)
>3321*Applicable after CN review
IMPACT EVALUATION TO DEVELOPMENT IMPACT 26
approval. The remaining 33 percent of the portfolio
is in preparation phase. Here, teams have passed
technical and policy relevance for expressions of in-
terest and received preparation grants from i2i to
develop full technical proposals. This usually entails
research and the Bank team travels to the country
to meet with the client and work together to final-
ize methodology and implementation plan.
Workshops for Capacity-Building and Dissemination In addition to supporting IE products, the i2i pro-
gram is committed to delivering four IE workshops
per year. As table 1 in section 2.2 shows, the i2i pro-
gram completed 14 workshops between FY15–FY17.
During the first two years of i2i implementation,
workshops focused on thematic program develop-
ment, capacity building, and portfolio selection. Au-
dience in these workshops comprised researchers
and subject experts from leading universities, gov-
ernment policymakers, operational staff from the
World Bank, and representatives from other Mul-
tilateral Development Banks and donors. Following
workshops, calls for proposals were organized to
invite teams to submit proposals for IEs in select
thematic areas. During this last FY, the nature of
workshops shifted from program development and
capacity building of government counterparts to
strengthening the research designs of the selected
portfolios in each thematic area and reviewing and
disseminating evidence. The target audience there-
fore shifted towards the research community, sub-
ject experts, and MDB operational staffs.
As the table below shows, all targets were met
for the number of workshops completed, people
trained, participation of organizations, and dis-
semination to policymakers. People trained in i2i
workshops and by i2i project teams form the i2i
network, which today includes close to 2000 rep-
resentatives and over 400 different organizations
(donors, MDBs, government agencies, academic in-
stitutions, and NGOs).
Policy Influencei2i IEs affect policy through all phases of project cy-
cle. Chapter 2.4 describes this model in more detail.
Data from November 2016 from the i2i Monitor-
ing System shows that 75 percent of the portfolio
reported having informed design of programs and
policies, based on a clear understanding of the un-
derlying theory of change and highlighted areas of
uncertainty and critical assumptions. 71 percent of
the portfolio reported that IE evidence from experi-
mental testing of alternative mechanisms was used
by governments or other stakeholders to determine
the most effective program alternatives or to in-
form policy decisions. And 59 percent of the portfo-
lio reported that IE evidence was used to motivate
scale-up or scale-down of policy.
TABLE 2: Workshops and Dissemination
Workshops and Dissemination
Target FY17 Actual Completed
Workshops (capacity building, knowledge sharing, program follow-up)
4 5 +
Number of people trained in i2i workshops or by i2i-supported teams
700 1944 +
Number of organizations 50 414 +
Dissemination events to policymakers 25 128 +
World Bank seminars 30 51 +
IMPACT EVALUATION TO DEVELOPMENT IMPACT27
It is worth noting that the i2i IEs included in this
survey are at different stages of their lifecycles.
Some of the policy indicators reported are highly
dependent on where in the cycle they are located.
For instance, only IEs that already have results are
asked questions related to whether the results were
discussed with the client or did the results motivate
scale-up/scale-down of a policy. We divided the IEs
according to their lifecycle into three phases: Phase
1 refers to the beginning of the IE, before implemen-
tation starts; Phase 2 refers to the time between
implementation of the intervention and before the
IE results are available; and Phase 3 refers to after
IE results are available.
Capacity Building and Dissemination Events
The following set of indicators aims to capture
the extent to which the IEs assisted clients or oth-
er counterparts in building capacity for policies or
programs, or for client staff in general. We present
three measures of engagement with the client to
develop skills or feed evidence into policy:
n Trainings provided: Over 660 people were
trained by IE teams on general monitoring, data
analysis, and other topics. This took place across
30 different training events.
n Discussion of baseline results and final re-sults with clients: About 69 percent of baseline
results were discussed with clients. Further, 92
percent of IE teams with final results discussed
them with clients. This represents, respectively,
1641 and 1515 people over 44 and 32 events.
n Task force type meetings: About 29 high-level
meetings were organized to align counterparts
and broader sets of stakeholders.
n Presentation to non-clients: IE results were
presented 52 times to non-clients, which rep-
resents a total of 2060 people.
Quality of Data
In this set of indicators, we aim to measure whether
the IE played a role in influencing the quality of the
data collected and used. We include three indica-
FIGURE 21: i2i IEs Add Value throughout the Project Cycle
Inform policy design75%
Guide mid-coursecorrections
61%
Inform adoptionand Scale-up
59%
FIGURE 22: Number of People Trained in i2i IEs FIGURE 23: Number of Events in i2i IEs
Capacity BuildingNumber of people trained in i2i IEs
Data Training
Other Training
211
455
Data Training
Other Training
Capacity BuildingNumber of events in i2i IEs
14
20
IMPACT EVALUATION TO DEVELOPMENT IMPACT 28
tors related to M&E and generation of data across
the IE, which in many occasions results in the first
comprehensive data on the topic in the country or
region where the IE takes place.
n Improved counterpart M&E*: Over 57 percent
of teams of IEs after concept note report that
the IE data requirements led to improvements in
monitoring and evaluation (M&E), data collection,
and/or reporting activities of the counterpart(s).
n High-quality baseline survey**: About 88
percent of i2i IEs provided high-quality baseline
surveys (including covariates with sufficient
sample sizes and representative of policy-
affected populations), thus creating evidence for
policymaking even before the IEs start.
n High-quality follow-up survey***: About 96
percent of i2i IEs provided high-quality follow-
up surveys (including covariates with sufficient
sample sizes and representative of policy-
affected populations).
Quality of Policy Decisions
This set of indicators aims at capturing whether
the impact evaluation has influenced policy deci-
sions in several ways, including at the beginning
through program design and later as the IE pro-
duces evidence through new data or IE results. We
present the following four indicators:
n Rationalized policy design*: About 75 percent
of IE teams report that the IE improved design
based on a clear understanding of the underly-
ing theory of change (causal links between the
intervention components and the outcomes) and
highlighted areas of uncertainty and critical as-
sumptions.
n Influenced others**: 45 percent of i2i IEs in-
fluenced the design or implementation of other
projects outside of the IE itself.
n Baseline informed policy design/implementa-tion**: Around 69 percent of IE teams report
that baseline data was used by governments and
other stakeholders to stimulate policy dialogue
and/or help identify problems and solutions.
FIGURE 26: Improved Data Quality (Percentage of i2i IEs)
High QualityFollow-up Survey** 96%
ImprovedCounterpart M&E*
High QualityBaseline Survey**
57%
88%
* : applicable after Concept Note review** : applicable after baseline results*** : applicable after final results
FIGURE 24: Number of Participants in i2i IEs FIGURE 25: Number of Events in i2i IEs
Presentation tonon-clients 2060
1641
1515
496
Presentation of baselineresults
Presentation ofIE results
Task Force type meeting
Dissemination EventsNumber of participants in i2i IEs
52
44
32
29
Presentation tonon-clients
Presentation of baselineresults
Presentation ofIE results
Task Force type meeting
Dissemination EventsNumber of events in i2i IEs
IMPACT EVALUATION TO DEVELOPMENT IMPACT29
FIGURE 27: Policy Indicators by Phase (Percentage of i2i IEs)
Rationalized design*
Baseline informeddesign**
Informed Adoption**
Scale-up or Down***
75%
69%
32%
59%
Phase 1 Phase 2 Phase 3
* : applicable after Concept Note review** : applicable after baseline results are available and discussed*** : applicable after final results are available and discussed
n Government or other stakeholders adopted causal mechanism(s) based on IE results***:
32 percent of IE teams report that IE evidence
from experimental testing of alternative mecha-
nisms was used by governments or other stake-
holders to determine the most effective program
alternatives or to inform policy decisions.
n IE results were used to motivate scale-up/scale-down of policy at national level***: Over
half of IE teams in Phase 4 report that the IE
produced evidence of sufficient (or insufficient)
effectiveness of the intervention in achieving de-
sired outcomes and were used by governments
and/or other agencies/stakeholders to motivate
scale-up (scale-down) of policy.
IMPACT EVALUATION TO DEVELOPMENT IMPACT31
5. How We Contribute to Development Issues
efficiency due to resource misallocation and/or
behavioral biases.
In this context, technology change encompasses any
shock in the production process that leads to higher
output, given the inputs available. That shock could
be caused by, for instance, better use of land, bet-
ter trained employees, better managerial practices,
reduction in red-tape costs, and change in organi-
zational culture. Technology change and, thus, pro-
ductivity growth goes hand in hand with technology
adoption. Shifts in the production frontier assume
factors of production are already used optimally.
The reality shows that this is more the exception
than the rule. DIME’s portfolio reflects that.
DIME’s Approach
The DIME team uses rigorous evaluation methods
to test different policies and interventions aimed
at increasing productivity and growth. Even though
several impact evaluations in DIME operate at a
micro level, the close partnership with both oper-
ational teams at the World Bank and government
counterparts help with scaling up good practices.
DIME’s endeavor then plays a crucial role in build-
ing knowledge, improving program designs inside
and outside the Bank, and the quality of policy
recommendations.
Currently, there are 31 IEs in 26 different countries
that are looking at issues related to productivity
growth. $1.7 billion has been allocated to this agen-
da through World Bank lending operations and $20
million has funded IE research.
5.1 Growth
Context Productivity is the key driving factor for long-
term and sustainable growth. The empirical re-
search, specifically at macro level, has mostly
emphasized the determinants of productivity
growth. Empirical research has mapped innova-
tion, human capital, and institutions as some of
the key determinants of the (total factor) pro-
ductivity (Syverson, 2011). However, much less
is known about how to increase productivity. In
other words, how to improve skills of the working
force? What reforms and regulations enhance
business environment? And, what incentives are
needed to spur innovation? The how to design ef-
fective policies question is key, given that many
developing countries struggle with low levels of
labor and total factor productivity (The World
Bank Enterprise Surveys, 1/2013).
In reality, to design effective interventions that in-
crease productivity, one needs to (1) identify ways
to increase efficiency of factors of production that
were idly used or misallocated and (2) find alter-
native combinations of factors of production that
result in higher growth potential.
Productivity growth can occur via both efficien-
cy gains (‘catch up’ effect)–changes in the pro-
duction process so as to help firms move closer
to the efficient production frontier–and shift in
the production frontier. In many cases, adoption
of new technologies is required to overcome in-
IMPACT EVALUATION TO DEVELOPMENT IMPACT 32
The current portfolio is structured in way that al-
lows the DIME teams to test different strategies
aimed at increasing productivity in both rural and
urban settings and among small and medium-sized
firms. The current DIME portfolio accommodates
these two sets of approaches dealing with produc-
tivity and growth.
For instance, in Bangladesh, Malawi and Mozam-
bique we are testing the impact of agricultural ex-
tension to leverage gender and social dimensions of
communication to disseminate a new technology.
In Brazil, two IEs are testing different ways of im-
proving management practices of small and medi-
um-sized firms. In one experiment (Sebrae-Parana),
small firms interacting on an electronic platform
will be randomly assigned to receive feedback in-
formation on their business practices. We will see
how they perform, compared to their peers, in the
adoption of best practices.
In the other Brazilian IE (Banco do Nordeste), me-
dium-sized firms randomly assigned to receive
feedback information, will be randomly split to
receive intensive on-the-job training and moni-
toring visits from consultants for six months to
help them adopt best business practices. Also in
Brazil, we tested the impact of a large finan-
cial literacy pilot program on knowledge and
adoption of improved financial decisions by high
school students. We found positive effects that
were magnified when parents were also provided
some training.
Other IEs in the portfolio are expected to affect pro-
ductivity through both efficiency gains and change
in the frontier. For example, in the Georgia IE, we are
exploring the rollout of broadband internet across
the country to test the impact of the broadband
expansion on firm performances, and whether an
intervention combining training on e-commerce
and demand shocks on first online orders increase
firms’ access to markets.
In Kenya, Mozambique, Nepal, and Rwanda we are
testing the impact of different irrigation interven-
tions on returns to on-farm investments and land
productivity. The interventions per se will also re-
duce exposure of small farmers to climate shocks
and thus incentivize investments in riskier (higher
returns) projects.
5.2 Shared ProsperityDIME’s research contributes to the theme of shared
prosperity in several distinct ways. Shared pros-
perity is a key goal of the World Bank, which has
twin objectives of ending poverty and raising the
incomes of the bottom 40 percent. The rationale
for this is based on the idea that we care not just
about the mean of a distribution, or the percent
below a certain threshold (such as the poverty line),
but also about the distribution across the popula-
tion. A given mean level of income can be consistent
with divergent levels of poverty depending on the
distribution of income within a society. Moreover,
issues of inequality have reemerged on the intellec-
tual agenda in recent years in both developing and
developed societies (Piketty 2014, Milanovic 2016).
In the development sphere, shared growth is now
seen as critical, not just for poverty reduction, but
also for the stability and legitimacy of institutions.
And inequalities are persistent, not just across in-
The DIME team uses rigorous evaluation methods
to test different policies and interventions aimed at increasing productivity and
growth.
IMPACT EVALUATION TO DEVELOPMENT IMPACT33
come groups, but between marginalized groups,
castes, ethnicities and across gender.
DIME’s contribution to this research agenda can be
organized conceptually around two fundamental re-
search questions:
1. Pervasive market failure suggests opportunities to
improve efficiency AND promote equity–but which
interventions work?
2. When are the objectives of promoting equity and
efficiency/growth complements and when are they
substitutes?
We can then further break this down into a series
of sub-research questions:
n Sub-research question 1: How can we reduce
physical (transportation) and soft (information,
networks) barriers for household and firms to
overcome social and spatial inequalities?
n Sub-research question 2: How can government
services be targeted to sustainably expand
access to marginalized populations?
n Sub-research question 3: How do social norms
contribute to gender inequality and what are the
growth and equity opportunities in overcoming
these inequalities?
Theme 1: Connecting People and Firms to MarketsThe importance of physical linkages to markets is
a central issue in development economics. Recent
contributions (Wantchekon and Stansig 2015)
have underlined the magnitude of this in develop-
ing countries. They estimate that transport costs
are the main drivers of poverty in Africa. Beyond
physical barriers, poor households and firms are
isolated from relevant information and face large
entry barriers to entrenched networks, exacerbating
inequality. An example of a policy response to this,
being evaluated by DIME, is the question of rural
road rehabilitation. While rural road projects are ex-
tremely common, there is relatively little evidence
about the returns to rural road construction. This
is at least in part because purposive targeting is
a major component of road project design, making
it difficult to construct a plausible counterfactual
scenario or control group.
DIME has two impact evaluations that will enable
progress on this critical issue. The first is the rural
feeder roads, transport costs, and local welfare in
Rwanda, which will use an Interrupted time series
identification strategy to estimate the impact of
road rehabilitation on import and export prices and
goods availability, land value, migration. The second
is a randomized rural road rehabilitation in Peru,
which may be the first experimental estimates of
the impact of rural roads. It will focus on income/
consumption gains and the use of public services
and gender-specific impacts.
Beyond rural roads, other projects in this area focus
on a) connecting firms to improved supply chains
(South Africa), b) connecting youth to job oppor-
tunities (Togo, Mauritania), c) connecting farmers
to markets (Kenya, other agricultural IEs), and d)
connecting workers in (urban) peripheries to urban
cores (Tanzania, Kenya, Colombia, Brazil, China, Ni-
geria, Nicaragua, Ethiopia).
Theme 2: Access to Government Infrastructure and ServicesA second theme relates to access to government
infrastructure and services. Most current evidence
focuses on the impacts of service provision (water,
sanitation, electricity, etc.), but there is relative-
ly limited focus on efficiency/equity issues. At the
same time, huge inefficiencies exist (for example,
$89 billion worth of electricity and 32.6 trillion liters
of water lost each year to theft and low-quality in-
frastructure). Private delivery of basic services may
be more efficient than public delivery (for example,
Galiani et al. 2005), but this raises equity concerns.
The marginal costs of expanding services to urban
and rural poor is often higher than willingness to
IMPACT EVALUATION TO DEVELOPMENT IMPACT 34
pay (WTP), and utilities face fiscal pressure and un-
certainty. Even when services are delivered to poor
areas, the transitory nature of urban slums means
that this may simply result in crowding-in higher
income populations, rather than providing targeted
support to the most needy. Therefore two key poli-
cy questions are: (i) how can we effectively expand
services to the poor and is public intervention jus-
tified?; and (ii), even when services are expanded
to poor areas, are the intended recipients the ones
who benefit?
Ongoing DIME projects in this area provide sub-
sidies to promote expansion of energy (Kenya,
Argentina), broadband (Georgia, Mauritania),
and water/sanitation (Kenya) services. Varying
subsidy levels allows each study to trace out the
demand curve for each service and the result-
ing benefits that accrue through access to the
service. This provides the information needed to
estimate the implied weighting of benefits/equal-
ity needed to justify intervention. Other work in
Kenyan slums takes this a step further. It mea-
sures changes in rent as well as in and out-migra-
tion resulting from provision of sewerage connec-
tions to households to estimate the effect of the
intervention on housing markets and resulting
gentrification.
Theme 3: Gender Equality and Social Norms A final area of focus is gender inequality and so-
cial norms. In this realm, taste-based discrimina-
tion and social norms limit the participation and
productivity of marginalized groups (gender, race,
ultra-poor, etc.). There is strong underlying theory
and empirical evidence across multiple sectors that
this leads to both inefficient and inequitable out-
comes (for example, Sen 1992, 2001; World Bank,
2015). DIME IEs focus particularly on gender and
ask: Where and how big are the market failures result-
ing from discrimination and what interventions work to
reduce them?
What is the impact of gender discrimination in
agricultural settings? One impact evaluation of
extension services in Malawi (Bin Yishay, Jones,
Kondylis, and Mobarak, 2016) shows that women
adopt new technologies better and retain knowl-
edge as well as (or better than) men, but are less
trusted as teachers. A second IE focuses on which
interventions work to reduce harassment of women
on public transport, in the context of women-only
train cars in Rio. This study estimates a) compli-
ance with the women-only law; and b) willingness to
pay for female-only cars and finds that willingness
to pay is close to zero, but that this is strongly (and
positively) associated with enforcement of the law.
The ongoing study also has planned experiments on
norm-shifting and efforts to increase compliance.
Other projects in this area focus on the impact of
providing legal aid for poor women (Jordan); female
leadership of SMEs (Haiti); transport impact on
women’s access to health and education services
(Peru); and the impact of factory jobs on young
women’s well-being (Ethiopia).
5.3 Risk and VulnerabilityIn addition to targeting poverty, we must con-
sider the extent to which the circumstances of
individuals, households, communities, and coun-
tries fluctuate over time. Indeed, variation in the
face of an otherwise sustainable, even comfort-
able livelihood can mean the difference between
well-being and destitution. Dercon and Shap-
iro (2007) estimate that 30–60 percent of the
world’s poor live in transitory poverty, meaning
that they may move in and out of poverty ac-
cording to fluctuations in their situation. For the
chronically poor, unmanaged negative shocks
have long-term negative consequences (Dercon
and Hoddinott, 2005).1
1 Deaton and Hoddinott (2004) show that adults exposed to drought in childhood experience permanent consequences in terms of stunting as well as educational and labor market outcomes.
IMPACT EVALUATION TO DEVELOPMENT IMPACT35
Sources of Risk Risk can come from a multitude of sources, some
unique to individuals and households (idiosyncrat-
ic) and others common to a whole area (covariate).
Negative shocks are likely to be most detrimental
to those living just within their means, since such
individuals can afford to lose the least. The 2014
World Development Report ‘Risk and Opportunity’
finds that countries with high incidences of pover-
ty are some of the least prepared to deal with the
risks that threaten so many livelihoods: Sub-Sa-
haran African countries are the least prepared to
manage risk, followed by southern and southeast
Asia and Latin America. For this reason, research
that seeks to improve risk mitigation and man-
agement in the developing world is vital for pover-
ty reduction.
Constraints to Risk ManagementSome barriers to risk mitigation and management
are within the control of individuals, while others
relate to broader issues such as missing markets,
absent institutions, and social or economic ex-
ternalities. In this section, we focus more on the
former issue: helping individuals to mitigate and
manage their own risks over time (see sections 5.4
on Governance and Accountability and section 5.5
on Global Public Goods and Externalities for de-
scriptions of work that seeks to tackle the latter
constraints). In particular, risk and vulnerability re-
search within DIME seeks to alleviate the following
three constraints:
n Cognitive and behavioral bias/failure: How
information asymmetries and behavioral biases
can be reduced/corrected for.
n Resource constraints: How to best unlock the
productive potential of poor and vulnerable
groups.
n Collective action failures: How to resolve
incentive and coordination failures between
multiple individuals.
Fifteen economists at DIME currently contribute
research to issues of risk and vulnerability through
36 impact evaluations in 23 countries across Afri-
ca, Asia, and Latin America. Indeed, the distribution
of DIME’s risk and vulnerability research is con-
FIGURE 28: Global Risk Preparedness
Source: WDR 2014, ‘Risk and Opportunity’.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 36
centrated in the geographies with lowest risk pre-
paredness, as diagnosed by the 2014 World Devel-
opment Report. The risk and vulnerability portfolio
represents around $33 million of research across
projects of a total value of $2.1 billion.
1. Informational Asymmetries, Behavioral and Cognitive BiasesInformational asymmetries lead to sub-optimal risk
management, since individuals must either base
their decisions on expected returns or expected
constraints (depending on the type of information
that is limited). In the event that expectations
deviate greatly from reality, or when the possible
options are very variable, this can lead to sub-opti-
mal individual decisions and socially inefficient out-
comes. On the other hand, individuals may also act
inefficiently despite seemingly having all the infor-
mation they need to choose the most efficient ac-
tion. This may be a result of poor interpretation of
available information or of a behavioral bias (such
as time inconsistent preferences), making the most
efficient solution less appealing at the time of deci-
sion-making (Duflo, 2006).
Interventions that seek to combat information
asymmetries or mitigate behavioral and cognitive
biases can help mitigate the risks faced by vulner-
able populations. DIME research targets both these
areas to answer questions such as: “How can we
harness mass media to provide information to em-
power individuals to make better decisions relating
to sexual behavior, human rights and democracy
promotion, and citizens’ rights to transitional jus-
tice in post-conflict contexts?” The program also
evaluates the effectiveness of interventions to mit-
igate behavioral biases such as time inconsistency
and present biases, asking questions such as: “How
expanding choice sets, such as by encouraging at-
risk youth to invest time in marketable skills or
through soft skills and cognitive behavioral therapy,
can reduce the propensity of such youth to partic-
ipate in crime?”
2. Resource ConstraintsNegative shocks can push households into poverty
traps (Sachs, 2005). For example, in the case where
an indivisible investment is needed to reach a cer-
tain level of productive capacity, households with
FIGURE 29: Distribution of Risk and Vulnerability Research: DIME
Source: DIME Internal.
IMPACT EVALUATION TO DEVELOPMENT IMPACT37
resources below this critical amount are unable
to transition from poverty. Resource injections to
move individuals over critical thresholds and social
safety-net schemes to prevent falling below these
are well-known approaches to resolving poverty
traps, though whether they do so in a sustainable
manner and how best to design and target such
schemes remain open questions. DIME’s work in
this area considers both approaches, asking: “How
to break poverty traps for the ultra-poor in the con-
text of high unemployment, to enable households
to support their own trajectory from poverty?” and:
“How best to design social safety-net programs to
prevent poverty traps, integrating productive safety
nets to foster long-run sustainability”.
3. Coordination Failures and Collective Action Problems
Coordination failures and collective action problems
may occur in contexts where the external conse-
quences of one’s actions do not factor in individu-
al decision-making or in the case where the action
chosen by other individuals is uncertain. For exam-
ple, in the case of disease prevention, an individual
deciding whether or not to have a particular vacci-
nation may not take into account the possible con-
tagious nature of an illness in the event of suffering
from it. By ignoring the effects their illness could
have on others, individuals underestimate the cost
of no vaccination and fewer are vaccinated than
might be socially optimal.
DIME’s risk and vulnerability research both evalu-
ates attempts to convey the marginal social cost
of actions with negative externalities and to help
individuals collaborate to reduce risks in the polit-
ical or social sphere. In the context of externali-
ties, research considers: “How informal institutions,
such as extended family networks, can be mobi-
lized to help individuals internalize the social cost
of disease?” When it comes to collective action
failures, DIME research asks, “To what extent fa-
cilitating information sharing, such as using public
recognition as a non-monetary incentive, can help
individuals to cooperate to reduce risk?”
5.4 Governance and AccountabilityA society’s institutions, or rules of the game, are
critical determinants of its development outcomes.
The design of a society’s governance and account-
ability structures leads to huge differences in devel-
opment.2 As such, large-scale development efforts
require an improved understanding of institutions
and the policy interventions that can change them.
Governance and accountability have taken center
stage in the search for institutions that underpin
development. For example, property rights have
frequently been argued as being the fundamen-
tal building block to an effective economy. Such
institutions span the public and private sectors,
and are present at each level of socio-economic
organizations.
At the theoretical level, a rich body of micro-the-
ory forged around contracting and principal-agent
models provides a framework for understanding
governance relationships. In its simplest formula-
tion, the principal-agent problem is about designing
contracts that can induce an agent (an employee, a
child, a subordinate division, or an aid contractor) to
perform a task as required by a principal. This could
be within a firm or government office or between a
community and their health workers. It is relatively
straightforward to design effective contracts when
effort and/or outcome are observable and there is
no uncertainty on how effort is transformed into
outcomes. In real world applications, problems arise
when neither effort nor outcomes are directly ob-
servable. Theoretical contributions have analyzed
these trickier situations at great length, including
2 Seminal contributions on this are those of North, Besley, and Person, and Acemoglu and Robinson.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 38
the seminal work of recent Nobel Prize winners
Bengt Holmstrom and Oliver Hart.
However, micro-empirical evidence has not kept up
with theory. The evidence base on many governance
institutions is limited. The approach of the DIME re-
search program in Governance and Accountability
is to start from the classical principal-agent frame-
work to produce evidence on the impact of gover-
nance and accountability mechanisms that are key
to development outcomes. As these mechanisms can
allow self-interested agents and principals to reach
a more cooperative equilibrium, our approach, ulti-
mately, also explores novel areas of collective action.
In particular, we focus on information, regulation,
monitoring, and enforcement mechanisms. Also, we
study governance and accountability problems with
an approach that recognizes their systemic nature.
Finally, we study the principal-agent model through
multiple lenses, which span theory, measurement,
and intervention.
Information, Regulation, Monitoring, and Enforcement
These mechanisms act as commonalities across
our research program. We study them in a variety
of contexts to assess whether they can be effective
in addressing one (or more than one) dimension of
the principal-agent problem. Shifting societal rules
and norms can affect the observability of effort
and outcomes and the type of agents that enter
into contracts with the public sector. Regulatory
reforms—that introduce new procurement process-
es (in Bangladesh, Brazil, Colombia, and The Philip-
pines) and introduce minimum patient and building
safety (in Kenya and Peru)—are studied to ascer-
tain whether they change the outcome (in terms
of quality and efficiency of the service delivered) of
the principal-agent game.
Providing information to key actors is predicted
to reduce information asymmetry, which is one of
the main issues within the principal-agent frame-
work. The research program is testing the impact of
providing information on court efficiency to judges,
registrars, clerks, and users (in Azerbaijan, Kenya,
and Senegal) as well as information on public works
performance to politicians and senior and junior bu-
reaucrats (in Pakistan). More direct mechanisms to
increase observability of effort and outcomes work
through monitoring and enforcement devices. Sev-
eral ways of tracking provider performance for bot-
tom-up accountability are being tested as part of the
research program (in Angola, Burkina Faso, Cambo-
dia, Democratic Republic of the Congo, and Nigeria).
Systemic-Wide Analysis Nature of Governance and Accountability
The systemic nature of collection action problems
requires a system-wide approach to analyze them.
Predicted solutions from theory (increase informa-
tion or monitoring) typically apply only to specific
elements of the system. There is growing evidence
on these elements of the system (for example, au-
dits and monitoring, rewards and warnings) but still
scarce evidence on the system and market equilib-
rium (price, quality, supply) effects of interventions.
A system-wide analysis is understanding and
aligning incentives in markets for road safety and
patient safety in Kenya. In particular, the Kenya
research program is studying, within the same
market, the impact of shifting formal and infor-
mal rules (regulation and enforcement of patient
safety and road safety), of providing information
(on clinic performance to clinic managers and pa-
tients and on driver safety to drivers, transport
company owners and riders), and of enabling bot-
tom-up and top-down accountability devices for
clinics and patients, and bus owners and drivers.
In addition to partial equilibrium impacts on ser-
vice provider performance, the research program
will study market outcomes in terms of prices and
supply indicators.
IMPACT EVALUATION TO DEVELOPMENT IMPACT39
Theory, Measurement, and InterventionOur research program explores the principal-agent
model through multiple lenses. These include: (i)
theory, which provides frontier predictions that can
be tested by our research program; (ii) a focus on
measurement to improve observability of efforts/
actions and performance and provide data for
monitoring/diagnostics; and (iii) an emphasis in de-
signing interventions/treatments that are embed-
ded in theory and test mechanisms.
We test theory predictions in Tanzania as we
study the role of characteristics of organization
structures on solving incomplete contracting in
infrastructure maintenance and the impact of
performance-based incentives to improve sub-
national service delivery under weak monitoring
capacity. Specific treatments underlying these
IEs stem from the same theoretical framework.
They include co-production organization in in-
frastructure maintenance (to solve incomplete
contracts) and performance based-grants to
subnational governments (to solve observabil-
ity of agent outcomes). On the measurement
side, our research program is producing globally
comparable micro-data on civil servants, devel-
oping systems to track subnational government
capacity, and court measurement systems to
track judiciary performance (Kenya, Senegal,
and Tanzania).
5.5 Global Public Good and ExternalitiesEconomies often fail to reach their full productive
potential because of coordination failures. In the
field of policy analysis, the Samuelson Rule states
that all production and consumption decisions
should be made so that the sum of benefits from
that good experienced by all people who consume
it equals the total cost borne by all actors who ex-
perience that good. Unfortunately, the optimal out-
come is often not achieved when the good in ques-
tion has the features of a public good, a common
good, or when either production or consumption of
a good creates externalities.
Public goods are goods whose consumption is
both non-rival and non-excludable. These features
create unique challenges for pricing and produc-
tion incentives, because those who pay the cost of
producing the good do not experience the benefit.
A classic example of a public good is broadcast
media. Once media such as an educational TV pro-
gram is produced by a government or foundation,
it is difficult to prevent other agencies from using
it (especially when copyrights are difficult to en-
force, which is the case in many developing coun-
tries) and anyone with a TV from viewing it for
free. But in order to incentivize the production of
high-quality educational TV content, co-financing
by other agencies or viewership fees need to be
collected.
Common goods are those whose consumption is
rival but non-excludable. An example of a common
good is water from an unregulated river. Any given
farmer can freely take water from a river to irrigate
crops, but if she does not take into account how
much her use affects other’s availability, water may
be overused.
Economies often fail to reach their full productive
potential because of coordination failures.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 40
Finally, goods with externalities are those where
the full benefits or the costs of consuming a good
are not fully borne by those who choose to consume
the good. Externalities may be positive or negative.
An individual’s decision to take actions to prevent
HIV infection reduces the risk of passing on infec-
tion to everyone around them, a positive external-
ity. Conversely, a decision to engage in criminal vi-
olence imposes negative externalities by diverting
resources toward policing and creating incentives
for vulnerable people to take costly actions to avoid
victimization.
Goods with the features of public goods, common
goods, or externalities appear in many contexts.
The coordination failures that cause them to be
produced and consumed inefficiently underlie many
of the common constraints that inhibit develop-
ment. Many contexts where non-rival goods or ex-
ternalities arise create justification for governance,
and can be addressed though properly structuring
the incentives of individual agents to align with so-
cial welfare as described in the previous sub-sec-
tion. But solutions to these problems can also lie in
addressing the constraints to market structures or
human behavior that interfere with efficient provi-
sion of goods. DIME is working to uncover the solu-
tions that best tackle these constraints.
Information as a Global Public GoodInformation is a unique type of public good that
has large positive, non-rival benefits, but is often
under-produced because producers are not able to
capture returns experienced by users and consum-
ers of information goods. Information goods may
be under supplied because of high production costs
and free-riding issues in their consumption, even
in the public sector. Examples are mobile apps or
edutainment programming, which can be costly to
produce. For these goods public investments are
difficult to be justified by any one government or
development agency without co-financing of other
agencies that may also benefit from these infor-
mation goods. As a result, despite the potential of
information goods in reaching millions of individuals
at low marginal costs—through mobile, TV, radio or
other communication and media outlets—they re-
main undersupplied.
In this theme, the impact evaluations address im-
portant knowledge gaps on effective ways to use
mobile technologies that monitor and promote effi-
cient use of resources and the cost of under-mon-
itoring; the relative effectiveness of different mass
media outlets (for example, TV, radio, print) to pro-
mote adoption of new technologies; and the role
that entertainment-education can play in address-
ing global challenges and epidemics, such as HIV/
AIDS and gender-based violence.
The knowledge agenda aims to unpack the causal
mechanisms from media exposure to individual and
community-level impacts on knowledge, attitudes,
and behaviors. In one example, an evaluation in Mo-
zambique measures the value of monitoring water
use within and across irrigation schemes to improve
efficiency. An evaluation in Nigeria has demonstrat-
ed the potential for an MTV-produced soap opera
to change social norms and behaviors. Section 6.8
provides greater details about DIME’s edutainment
research program.
Finally, goods with externalities are those where the full benefits or the costs of consuming a good are not
fully borne by those who choose to consume the good. Externalities may be positive
or negative.
IMPACT EVALUATION TO DEVELOPMENT IMPACT41
Managing the CommonsRecognizing that many crucial economic goods are
common pool resources, DIME evaluations demon-
strate and test the strategies for creating and
maintaining institutions that manage common pool
resources.
DIME’s work on this area seeks to test and demon-
strate which institutional structures and incentives
are effective in managing common goods. There are
many examples where individuals either over- or
under-perform an action relative to the social op-
timum because they do not take into account the
fact that their own use reduces the available re-
sources for others. DIME’s evaluations focus first on
the question of identifying which resources are sub-
ject to this problem. For example, for managing for-
ests in Burkina Faso, by varying institutional struc-
tures across communities, we measure how much
deforestation is due to failure to take account of
individual costs for others when making decisions
about collecting firewood.
In contexts where a given resource is known to be
subject to the problems arising from managing
common pool resources, such as irrigation, DIME
tests the question of why existing institutions es-
tablished to manage the problem fail. For example,
in Rwanda a project varies the user fee paid by ir-
rigation users to assess whether these fees are set
too high for sustainable participation in water-user
groups. Finally, DIME works to design and test in-
novations to institutions to manage the commons
to improve these institutions. For example, a project
in the Democratic Republic of the Congo is test-
ing accountability mechanisms for service providers
working in conflict-affected areas to assess wheth-
er these reforms change the private incentives to
participate in conflict.
Private Incentives for Adopting Pro-Social Activities and Conservation Technologies DIME research focuses on uncovering and mitigat-
ing the behavioral constraints that prevent people
from adopting technologies that would be privately
profitable to them, but also have non-rival attributes
that create large public externalities. This situation
is particularly apparent in the context of production
or consumption decisions that may be just prof-
itable or incentive-compatible for an individual or
firm to adopt, but also entail large public benefits.
For example, environmental conservation. Govern-
ments may need to provide incentives to individuals
or communities to adopt conservation technologies
with large social benefits. This can be the case even
when technologies may be profitable, from market
failures (for example imperfect information about
the technologies’ benefits, unclear property rights)
to financial constraints faced by households or
firms to psychological limitations that are increas-
ingly being studied by behavioral economists. For ex-
ample, present bias may prevent people from prop-
erly weighing the long-term benefits or savings of
adopting new technologies, even in the cases where
they can afford it and would be better off in the long
term by adopting these technologies.
The knowledge agenda in this theme covers opera-
tional research questions such as optimal subsidy
schemes for purchasing new technologies, such as
solar lamps in Argentina; and selecting the most
cost-effective methods to diffuse new information,
such as new cultivation strategies among farm-
ers in Bangladesh. A key question in this domain
is whether the policies that improve take-up of
pro-social technologies in the short term will lead
to sustained use and maintenance in the long term.
This question is being explored in Mexico and Ar-
gentina, where upkeep of pro-environment actions
is being assessed after the initial policies encourag-
ing take-up are withdrawn.
IMPACT EVALUATION TO DEVELOPMENT IMPACT43
6. Economies of Scale in Learning
6.1 Fragility, Conflict, and Violence
Context Around 2 billion people live in countries affected by
fragility, conflict, and violence. Poverty rates are 20
percent higher in countries affected by repeated cy-
cles of violence. By 2030, an estimated 46 percent
of the world’s poor will live in areas characterized
as fragile or conflict-affected. While trends show
that poverty is declining across much of the world,
countries affected by conflict are falling behind.
Development challenges in fragile, conflict, and vio-
lence (FCV) settings transcend national boundaries
through the displacement of populations, spread
of disease, reduced trade, and increased organized
crime and terrorism. Further, high levels of fragility
and violence exist in countries otherwise considered
relatively stable. Many countries in Latin America
and the Caribbean, for example, suffer levels of vio-
lence comparable to those in the most conflict-rav-
aged states.
The international community is committed to as-
sisting communities emerge from conflict, sustain
peace, and resume growth. The World Bank, bilat-
eral, and other multi-lateral donors invest billions
of dollars a year to help achieve peace and build
states. The evidence base for designing such pro-
grams is, however, sparse, especially with regards
to rigorous evaluations aiming to identify what
works, and how, to reduce fragility, conflict, and vi-
olence. This knowledge vacuum impedes our ability
to design effective interventions to promote pov-
erty reduction and welfare improvement in FCV
settings.
Rigorous evaluation of policies targeting FCV issues
is therefore of paramount importance, all the more
so as the volume of resources from the World Bank
and other development partners towards such
settings increases (under IDA18, for example, the
World Bank is set to significantly increase financ-
ing for FCV-related issues). Not only is developing
this evidence a priority, but experience to date
shows that, even with the amplified challenges of
working in FCV environments, rigorous evaluation in
such settings is possible. IEs have been conducted
in diverse FCV countries such as Afghanistan, DRC,
Colombia, Cote d’Ivoire, Sri Lanka, Rwanda, north-
ern Nigeria, and Liberia.
FCV Impact-Evaluation Program
In March 2014, DIME and partners inside and out-
side of the Bank launched the Evidence for Peace
(E4P) program. Its overall goal is to assess evidence
gaps in FCV responses and generate improved
knowledge about how to best support FCV clients
to deliver the results so critically needed for citizens
to gain confidence in the path out of conflict.3 E4P
was the first IE program to benefit from a launch
workshop under i2i, followed by the first open call
3 Internal World Bank partners include the Fragility, Conflict, and Violence (FCV) Group-CCSA and the Latin America’s Citizen Security Team. External partners include the Initiative for Impact Evaluation (3ie) and Innovations for Poverty Action (IPA). The original initiative included: (i) development of a scoping paper, based on an evidence ‘gap map’ that identifies the status of the evidence base to highlight priority questions for future research; (ii) design, implementation,
IMPACT EVALUATION TO DEVELOPMENT IMPACT 44
for proposals for impact evaluations addressing the
program’s four initial themes: jobs for resilience,
public-sector governance, urban crime and violence,
and gender-based violence.
Today, the program includes 36 IEs across 21 coun-
tries. The portfolio represents around $30 million
of research across projects of a total value of $2.1
billion. Further, a series of white papers synthesizing
the state of the evidence in each of the four target
themes and proposing priority IE research topics
will be completed and disseminated in 2017.
An important development during the last program
year has been the redefinition of the E4P focus
areas around broader development issues encom-
passing and building on the original four program
themes. Four key research areas have been defined:
(i) basic service delivery in weak states; (ii) job op-
portunities for at-risk youth; (iii) breaking poverty
traps and vulnerability; and (iv) the political econo-
my of post-conflict reconstruction. This redefinition
was conceived with a view to strengthen linkages
with other i2i areas and with World Bank Global
Practices, beyond the FCV group. Work under each
key research area is summarized below.
Basic Service Delivery in Weak States Strong institutions that provide quality services to
citizens are a necessary condition to support coun-
tries to move out of fragility, conflict, and violence.
Yet, these are the settings in which institutions are
likely to be the most eroded and dysfunctional. Here,
the program focuses on civil-service reforms as well
as the rebuilding of government capacity and ac-
countability systems, to improve our understanding
about what works to develop effective governance
structures where it is most needed and, perhaps,
the state is least capable.
Work towards service provision in weak states is
ongoing in Liberia, Zimbabwe, and the DRC. For ex-
ample, work in the DRC seeks to attract talented
individuals to work in the country’s civil service, in-
vestigating the effect of deployment outside one’s
home region on breaking the patronage systems
common in the government.
Improving Job Opportunities for At-Risk YouthYouth in FCV contexts are often left with few mar-
ketable skills and little opportunities to cultivate a
sustainable livelihood. They can become vulnerable
to involvement in conflict, illicit activities, or violent
crime. Work in this area investigates ways of break-
ing this cycle of poverty and violence through hard
and soft-skills training, psychosocial therapy, and
labor market-insertion programs.
Our program is currently seeking to expand the
opportunities for at-risk youth in Cote d’Ivoire,
Honduras, Liberia, and Nigeria. For example, work
in Honduras looks to break cycles of crime and vi-
olence through a temporary jobs program aiming
to provide at-risk youth with the hard and soft
skills needed to succeed in the labor market. A
focus on cognitive behavioral therapy as well as
traditional skills-based training aims to provide
recipients with the practical and emotional re-
sources to earn a sustained livelihood in a chal-
lenging setting.
Breaking Poverty Traps and Cycles of (Gendered) VulnerabilityIn FCV contexts, support for vulnerable groups is
often lacking and both their immediate and long-
run needs are overlooked and their productive ca-
pacity ignored. This can lead to a perpetuation of
poverty traps and cycles of vulnerability. Further,
evidence shows that poverty-induced vulnerabili-
and dissemination of impact evaluations funded within the Bank and through a new external funding window; (iii) development of a framework to improve the quality of analytical work on FCV and of tailored methodologies for evaluation/data collection in FCV settings; and (iv) hands-on training in impact evaluation and the creation of communities of practice for knowledge sharing.
IMPACT EVALUATION TO DEVELOPMENT IMPACT45
ties tend to disproportionately affect women and
children. Our research work in this area builds on
the ultra-poor literature to understand the poten-
tial of interventions geared toward breaking pov-
erty traps and addressing systemic vulnerabilities.
It also considers the effectiveness of social safe-
ty-net programs to support such individuals as well
as big-push interventions, which seek to provide a
productive livelihood and exit from poverty traps in
the long run.
Research is currently ongoing in Afghanistan, Co-
moros, the DRC, Egypt, and Tunisia. In Comoros,
research examines the implications for expenditure
and intra-household resource allocation of assign-
ing cash-for-work safety-net schemes to women.
Our work measures differences in investment in
children, an area of central importance for breaking
cycles of poverty in fragile settings. It also investi-
gates the potential of norms-shifting and targeted
interventions to eradicate child labor and address
gender-based violence.
Political Economy of Post-Conflict Reconstruction
E4P research in this area focuses on understand-
ing the drivers and perpetuators of conflict and on
evaluating strategies designed to address these.
Postwar societies are often confronted with a wide
range of issues—including information asymmetries
between elites and masses, low levels of inter-per-
sonal coordination, social dislocations, and security
and mobility constraints—that prevent a rapid re-
turn to stable social and political orders. Some of
these are root causes of the conflict in the first
place or conflict drivers that sustain FCV cycles.
Work in post-conflict reconstruction currently fo-
cuses on Liberia, Sierra Leone, and the DRC. For
example, research in Liberia seeks to overcome
informational asymmetries and collective action
problems in rural areas by providing groups of wom-
en with access and a safe space to listen to unbi-
ased political radio broadcasts by United Nations
peacebuilders. Results show that overcoming these
barriers can increase female political participation
in many forms.
Going ForwardOver the past two years, the FCV IE program
has contributed to generating valuable knowledge
on what works, and why, to resolve challenges in
key FCV areas. In addition, we have undertaken a
stock-taking exercise of the existing evidence in the
program’s initial four target areas in an attempt to
contribute global public goods over and above indi-
vidual studies. The white papers produced from this
exercise, alongside practical knowledge generated
over the past years through ongoing IEs, will help
us define an ambitious, integrated approach to fu-
ture research, which will maximize the possibility for
linkages and broad lessons.
The program has now entered a new phase of con-
solidation and expansion focusing on at least four
priorities:
n Engaging stakeholders on findings from the white papers: As already noted, the FCV IE
program has produced four draft white papers,
one for each of the original thematic areas,
which summarize the current body of evidence
and policy implications and highlight future
research directions. In the next phase, we are
planning a series of events to discuss and finalize
these papers and engage with stakeholders
from the World Bank, governments, academic
institutions, and other development partners to
share the lessons learned, to promote broader
dissemination of findings, to explore further
research synergies, and to launch a new series
of IEs. This process will begin with a workshop
in FY17.
n Strengthening analytical and IE work of our Gender/GBV and FCV portfolio: Our work on
gender/GBV and FCV remains limited. To expand
IMPACT EVALUATION TO DEVELOPMENT IMPACT 46
on this, we will seek to deepen the gender impact
of existing projects and to draw comparisons
across settings by adding similar interventions
in a number of contexts. In particular, we will
leverage existing interventions in the area of
‘Breaking Poverty Traps and Vulnerability’ to
study the potential for deepening the effect
of temporary safety-net programs through
business grants to females to provide productive
livelihood opportunities in the long term. We
will also research how best to tackle norms
surrounding GBV in the context of these
programs.4
n Launching new IE window(s) in emerging priority areas: As well as deepening the impact
of existing projects in current thematic focus
areas, we seek to expand our work into new
FCV-relevant areas. We will create a working
group in order to establish a formal process of
identifying knowledge gaps in priority areas. We
will then seek new collaborations to broaden
our research in the most considered way
possible. For example, we plan to work with the
FCV Group and its partners on a new initiative
towards understanding and addressing issues
related to forced displacement, both for the
displaced themselves and for destination
communities.
n Expanding collaborations with DFID country programs in overlapping priority areas: Thus
far we have had successfully collaborated with
DFID to carry out joint IEs on priority country
programs that fit with the current thematic
priorities. (We have completed IEs on the
political economy of post-conflict reforms in the
DRC and have ongoing IEs in Nigeria focusing
on employment of marginalized youth.) We plan
to strengthen and expand such collaborations
and, depending on demand, identify new priority
areas for further collaboration on IE work.
6.2 Agriculture
Urgent Need for Evidence in AgricultureThe Sustainable Development Goals (SDGs) note
that agriculture is the single largest employer in
the world. Globally, 40 percent of the population
earns its income from agriculture. The SDGs urge
the international community to make the invest-
ments needed to double agricultural incomes of
small-scale food producers.5 Astonishingly, little
evidence exists to rigorously inform the invest-
ments needed to meet this urgent goal. For ex-
ample, a 2015 systematic review on the effects of
training, innovation, and technology on smallhold-
er productivity in Africa identified only 19 studies
that met the scientific standard to be included in
the review, making it impossible to assess which
interventions yield the highest returns.6 The gap
between urgent need for action and the evidence
available to inform such action is therefore great-
er in agriculture than in many other sectors in
development.
Agricultural development is crucial, not only for pov-
erty reduction but for many other SDGs as well.
Ending hunger and improving nutrition for the 13
percent of people in the developing world, who are
hungry, requires restructuring the agricultural val-
4 For instance, we have a new multi-country initiative looking to target women who graduate from the livelihood interventions we are evaluating with some additional capital (through an unconditional cash transfer), with the view to promoting entrepreneurship and enhancing sustained livelihoods. We are also working with internal and external partners to incorporate GBV-prevention pilots into large social-protection programs we are already evaluating in countries such as DRC, Egypt, and Tunisia. Because such programs are typically implemented in poor communities with a high incidence of GBV, they figure as promising GBV interventions. Additionally, we are working on partnerships with NGOs such as Promundo, International Rescue Committee (IRC), Search for Common Ground or Women for Women International (WfWI) that are conducting innovative gender-empowerment and/or GBV-prevention programing to develop and implement joint IEs in critical areas.
5 http://www.un.org/sustainabledevelopment/hunger/.6 http://www.campbellcollaboration.org/lib/project/310/.
IMPACT EVALUATION TO DEVELOPMENT IMPACT47
ue-chain; from farmers who grow food to retailers
who sell it to consumers. As a sector contributing
both carbon emissions and capture and uniquely
susceptible to climate and extreme weather, agri-
cultural innovations can offer solutions to climate
change through both mitigation and adaptation.
DIME builds evidence on the innovations that best
address all of these challenges through its agricul-
ture portfolio.
Policy-Driven Evaluation Design
Many of DIME’s impact evaluations in agriculture
were launched following a workshop on Agriculture
Innovations held in June, 2014 in Kigali, Rwan-
da. Ahead of this event, the Africa Region of the
World Bank organized a high-level meeting of de-
cision-makers from ministries of finance and agri-
culture, researchers, and other policymakers to set
priorities for research. The June event then took the
resulting recommendations to a gathered set of
policymakers, project staff, and researchers to em-
bed research questions and designs into the project.
This model of involving policymakers from the ear-
liest stages of designing evaluations and building
the evaluation directly into projects ensures buy-in
from projects and immediate policy relevance of re-
search findings.
In November, 2016, DIME convened a conference on
Evidence for Agriculture to share findings from on-
going and completed evaluation in the agriculture
portfolio and identify emerging priorities for eval-
uations in the sector. This event engaged partici-
pants from fifteen institutions, including university
researchers, policymakers from governments and
multinationals, and donor agencies.
Active Impact Evaluations
The AADAPT portfolio includes more than 20 im-
pact evaluations in 12 countries across Africa,
South Asia, Latin America, and the Caribbean. The
evaluations are distributed across five knowledge
gaps identified as constraints to the design of ef-
fective agriculture policy. These topical areas of fo-
cus are highlighted in table 3.
Improving the State of Knowledge and ImplementationDIME produces rigorous evidence on under-studied
issues relevant to agricultural policy. This advances
knowledge that can be used to design policies to
improve productivity in the sector that provides the
largest source of income and jobs for the world’s
rural poor. DIME has already produced rigorous
research on how to adjust extension programs to
optimize knowledge diffusion, the relationship be-
tween land rights and technology adoption, and the
role of gender in learning about technology.
DIME’s model changes the way that agriculture
programs operate throughout every stage of the
impact evaluation, from establishing comprehen-
sive data-collection platforms to monitoring roads
in Rwanda, to changing the way that recipients
are selected for irrigation investments in Mozam-
bique, and providing conclusive evidence on the
most cost-effective arrangements for extension
programs in Malawi. DIME’s research from the ag-
riculture portfolio influences policy directly through
intensive interaction with partners from govern-
Many of DIME’s impact evaluations in agriculture were launched following a workshop on Agriculture
Innovations held in June, 2014 in Kigali, Rwanda.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 48
ments and multinationals. Further, it often appears
in working paper and top journals in the field of
development economics.
Going ForwardFollowing the stock-taking and knowledge-sharing
event in November 2016, the AADAPT program will
launch the next round of impact evaluations to in-
fluence both agricultural policy and the agricultural
research agenda. This process will include four sets
of activities.
Consolidating evidence into policy-relevant mes-saging: Almost all evaluations have now conduct-
ed a baseline survey and about half of those have
recently completed follow-up surveys. Another 25
percent of evaluations is expected to graduate to
follow-up stage in the next year. As the program is
maturing, we fill focus on consolidating policy les-
sons and ensuring dissemination of those findings
within the local government, across the portfolio,
and the wider development community. In Janu-
ary, 2017, Florence Kondylis gave a Policy Research
Talk at the World Bank on the topic of Retarget-
ing Investments in Agriculture. She distilled the
learning from the first eight years of DIME’s work
demonstrating strategies for enhancing agricul-
tural productivity. This talk provides the basis for
engaging policymakers in the World Bank and be-
yond on frontier issues in agricultural research and
innovation.
Focusing on emerging priority areas: In consul-
tation with the Agricultural Global Practice, the
AADAPT team have identified areas where prac-
titioners within the Bank feel that additional fo-
cus is needed. One example of a new approach is
understanding complementary investments and
TABLE 3: DIME’s Work in Key Learning Areas
Knowledge Gap Example Evaluation QuestionActive IEs in this
Area
Commercialization: What are the public investments needed to ensure that farmers have access to markets and receive fair prices for their products?
Can matching grants for inputs and equipment combined with training on business planning improve competiveness in markets?
Haiti, Brazil, Liberia
Financial Constraints: How do financial barriers and institutional constraints prevent farmers from making profitable investments? What are the simple interventions that can overcome these constraints?
Can financial education tailored to farmers increase savings and increase investments in inputs?
Rwanda, Benin, Haiti, Uganda,
Rural Infrastructure: Are large infrastructure investments always profitable? Beyond construction, how can we ensure sustainability of investments by building effective users groups to manage the infrastructure?
Is there a productivity trade-off when prioritizing smallholder participation in irrigation programs?
Rwanda, Mozambique, Kenya, Nepal
Information: Are farmers aware of the productivity gains to be realized from adopting new technologies and methods? If not, what are the most efficient ways to help them learn about these opportunities?
Which approaches to crop demonstration lead to the highest adoption of improved crop varieties?
Bangladesh, Mozambique, Malawi, Rwanda, Nepal
Natural Resource Management:How can we encourage rural communities to manage and protect natural resources such as forests, clean water, and soil, while supporting livelihoods that rely on these resources?
What is the role of Payment for Environmental Services (PES) in Reducing Emissions from Deforestation and forest Degradation (REDD+) in Burkina Faso?
Ghana, Burkina Faso, Brazil
IMPACT EVALUATION TO DEVELOPMENT IMPACT49
goals in agricultural programs, particularly those
related to nutrition, social protections, and climate
change. Ongoing evaluations in Nepal and Rwanda
seek to uncover how nutrition interventions and
social-protections interventions can enhance the
impact of agricultural productivity-enhancement
efforts. A second area is a push toward expanding
the commercialization portfolio. Newly-launching
projects in Senegal and the DRC will focus on val-
ue-chain interventions, warehousing, and market-
ing as channels to translate agricultural productiv-
ity into income.
Launching the next wave of agriculture IEs: ln the
fall of 2017, DIME will again convene policymakers
currently implementing impact evaluations to share
final findings and learning and to engage projects in
the early phases of implementation to launch the
next wave of evaluations.
6.3 Governance
The ieGovern ProgramGovernment organizations provide essential public
services in key areas such as health, education, and
infrastructure. The size of the public sector is espe-
cially large in developing countries. However, key de-
terminants of effective governments are still largely
unknown. For instance, which factor is most important
to ensure effective delivery of public goods? Or, which
mechanisms ensure a more transparent and account-
able public procurement process? These are still un-
answered policy research questions. Governance
reforms are often long term, complex, and difficult
to measure. Rigorous evidence on what works in the
sector is, therefore, in short supply. In fact, the gov-
ernance field represents less than 3 percent of reg-
istered impact evaluations.7
DIME and the Governance Global Practice launched
the ieGovern program in 2013 to produce rigorous
evidence to improve governance project results and
to push the frontier of available evidence on what
works in governance reform. To date, the program
has a portfolio of 31 IEs across the world that study
four main themes: (i) civil service reform, (ii) public
financial management (tax and procurement), (iii)
justice, and (iv) decentralization/subnational pub-
lic-sector management.
Over the last year, the program has reached matu-
rity: most IEs in the portfolio have passed concept
note stage and are currently being implemented.
A conceptual framework centered on mechanisms
of incentives, demand-side and top-down account-
ability, relaxing of constraints, and delivery mech-
anisms has been firmly established (see also the
Governance and Accountability section in chapter
5.4). Lastly, important initiatives related to ieGov-
ern have been launched: the Bureaucracy Lab and
the Research Flagship report of the Global Tax
Team (see more details below).
The Bureaucracy Lab (Civil-Service Reform)
IE research has mainly focused on studying perfor-
mance incentives for frontline staff—such as teach-
ers, nurses, and doctors—that address, for instance,
problems of absenteeism or underperformance. To
go beyond this, IE work in the civil service-reform
pillar focuses on research questions related to those
civil servants who work in core government minis-
tries, such as ministries of finance and education,
and bear the responsibility for designing a country’s
policies, collecting its taxes, and so on. Key policy
questions being studied include how to motivate
public-sector workers to perform better with differ-
ent (monetary and mission-based) incentives (Libe-
ria and Pakistan), how to improve the governance
of maintenance of public infrastructure (Tanzania),
how streamlined information flows within the pub-7 3ie Impact Evaluation Repository (http://www.3ieimpact.org/
evidence/impact-evaluations/), accessed 1/14/2016
IMPACT EVALUATION TO DEVELOPMENT IMPACT 50
lic sector can improve project performance (Pa-
kistan), and how a public-private partnership can
facilitate access to public services for marginalized
groups (India).
IE work on civil-service reform fits into a broader
research program called The Bureaucracy Lab, which
is an initiative co-led by DIME and the Governance
Global Practice. The Lab is creating improved ad-
ministrative data on the characteristics of public
officials and their organizations to inform the op-
erational design of public-sector organizations. In
addition, the Lab is undertaking experimental work
with large-scale surveys of civil servants to generate
an evidence base on how to survey civil servants ef-
fectively. The Bureaucracy Lab is also working with
academic anthropologists and sociologists to create
a detailed picture of civil services across the world.
Each of these elements uses the ieGovern program
as a platform for research, while providing inputs
that feed back into the design of the evaluations.
Tax and Procurement
Public Financial Management (PFM) reforms are
a core of support in client countries, by the World
Bank and other donors for a long time. Yet only few
IEs exist on the effectiveness of different PFM sys-
tems. The IE research work under ieGovern tried to
fill this gap with several IEs in the PFM subsectors
of tax and procurement. Research questions being
addressed include the impact of the adoption of
e-procurement systems on competition and mar-
ket entry of new firms, prices and value for money
of government purchases (Bangladesh and Brazil),
how centrally coordinated framework agreements
affect the procurement process and quality of
services procured (Colombia), and how behavioral
nudges and facilitation measures can affect will-
ingness to pay taxes and tax compliance (Tanzania
and Colombia).
The ieGovern portfolio of tax research has helped
spark the creation of a broader research program
embedded in the Governance Global Practice fo-
cused on Innovations in Tax Compliance. The objective
of the research program is to influence the design
of World Bank tax operations through the develop-
ment of a multifaceted approach to improving tax
compliance. This approach explores strategies that
are both i) technically appropriate and ii) lever the
Bank’s broader governance operations to engage
citizens and progressively build trust, reciprocity,
and support for tax compliance.
To do this, the project will develop a framework
that holistically looks at enforcement, facilitation,
and trust as key mechanisms to improving tax
compliance. The work recognizes that technocrat-
ic reform focused on enforcement and facilitation
remains essential, but more substantial and long-
term improvements are ultimately likely to depend
on building a relationship of mutual trust between
government and taxpayers. The project will serve
as a convening force for research partnerships
both inside the Bank and with outside academic/
research institutions, including International Centre
for Tax and Development (ICTD), Institute for Fiscal
Studies (IFS), and Chr. Michelsen Institute (CMI).
JusticeAn efficient, fair, and accessible justice system is
consequential for peace and security, encourages
investment and growth, and is fundamental to no-
tions of citizenship and trust in government. Yet,
there is little empirical research in justice-system
reform, in large part because data is not easily avail-
able. In recent years, however, governments around
the world have embraced electronic case-manage-
ment systems and used innovative technologies to
expand access to justice. Leveraging the Bank’s re-
lationship with governments, the ieGovern work on
justice is uniquely positioned to take the lead in jus-
tice research. It is evolving into a broad research pro-
gram to both establish a global data infrastructure
for the justice sector and develop a global program
for understanding the economics of justice reform.
IMPACT EVALUATION TO DEVELOPMENT IMPACT51
The ieGovern program is currently focused on (i)
strengthening administrative data collection and
developing case-management systems capable of
producing high-quality data (Kenya and Senegal),
(ii) developing an empirically validated measure-
ment framework for justice research, (iii) using
high-frequency case data to understand the impact
of justice reforms (Croatia and Senegal), and (iv)
setting up the foundation for future experimenta-
tion in justice (Azerbaijan and Kenya). Going for-
ward, the program aims to establish a global depos-
itory of administrative and survey data on justice
and identify additional priority countries in which to
experimentally and iteratively test the impacts of
new justice reforms.
Subnational PSM/ Decentralization
Transferring power and responsibilities to local enti-
ties is a very popular reform for many countries, in-
cluding in OECD, middle-income, and poor countries.
However, the evidence base of how decentralization
reforms fare in practice has not kept up with the
number of reforms. Unanswered research questions
include how to measure and incentivize the per-
formance of local governments, how to deal with
potential elite capture at the local level, and how
to ensure local governments have sufficient capac-
ity to handle increasing responsibilities and collect
their own revenues.
Our research programs explore several dimensions
of the decentralization puzzle. In Cambodia, an IE is
testing how to harness social-accountability inter-
ventions to improve service delivery of local govern-
ments. The impact of demand-side actors such as
community officers and community-based organiza-
tions in making local governments more accountable
is being studied in Burkina Faso and Solomon Islands.
Transfers of resources from central government to
local governments based on their institutional per-
formances are being tested in Tanzania to assess
whether this program-for-results type of incentive
scheme is effective in improving local service delivery.
Highlights of Future ActivitiesSeveral key items will be launched or produced in
the next year. As part of the Bureaucracy Lab, we
plan to publish a global data-set on public sector
wage bill, public-private employment comparisons,
and public sector compression ratios. In addition,
surveys of civil servants in Liberia, Tanzania, and
Guatemala will be completed. Overviews of the lit-
erature on the anthropology of bureaucracy and a
review of civil-servant surveys are also planned.
The justice program aims to establish a global
depository of administrative and survey data on
justice and identify additional priority countries in
which to experimentally and iteratively test the im-
pacts of new justice reforms.
Two new seminar series are in the pipeline: one fea-
turing task team leaders and IE research teams
that highlights how the i) design of an impact eval-
uation and ii) embedding it within the project is pro-
moting adaptive learning at various stages of the
project lifecycle and another on aspects of bureau-
cracy in the developing world.
An efficient, fair, and accessible justice system is consequential
for peace and security, encourages investment and
growth, and is fundamental to notions of citizenship and trust
in government.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 52
6.4 Climate ChangeWhat is the right balance between meeting the
World Bank’s objective of eradicating poverty while
limiting environmental consequences? The Sustain-
able Energy for All initiative aims to achieve uni-
versal access to sustainable energy, but, this would
mean providing electricity access to over one billion
new people. Globally, approximately two-thirds of
greenhouse gas emissions already come from en-
ergy extraction and use (IPCC 2014). Agriculture is
the largest sector in many developing country econ-
omies, but deforestation contributes between 10
and 17 percent of annual carbon emissions (Samii
et al., 2014).
While we are on target to eradicate extreme pover-
ty by 2030, we are also on target to increasing the
Earth’s temperature to irreversible levels that are
anticipated to have far-reaching long-term conse-
quences on economic growth, vulnerability, and the
environment. While the problem of climate change
is fundamentally a global collective action chal-
lenge, there are important program-level activities
and insights that can help us mitigate its effects
and strengthen resilience.
The initiation of an Energy and Environment (E&E)
program was motivated by the dearth of rigorous
impact-evaluation evidence in these sectors and
the influential role they play in poverty alleviation
and climate change. In 2011, the World Bank had 12
ongoing or completed impact evaluations in both
energy and environment (compared to over 100 in
education), despite the fact that these programs
consist of almost 20 percent of the World Bank’s
lending portfolio.
E&E Impact Evaluation Program
The E&E program was launched in Lisbon in Octo-
ber 2014. It brought together 19 project teams (fi-
nanced through DFID, the GEF, CIF, IDA and IBRD)
and 28 researchers from 11 academic institutions
to refine research opportunities based on project
interest and operational feasibility. This was com-
plemented by a parallel set of workshops focused
on measurement opportunities in the sector.
The first workshop, held jointly with CEGA in Au-
gust 2014 in Berkeley, brought together engineers,
economists and World Bank counterparts to ex-
plore leveraging new technologies to improve mea-
surement in energy and environment projects and
research. A follow-up Berkeley measurement work-
shop, focused on innovative measures for climate
resilience, was held in June 2015. To strengthen the
economic theory underpinning each IE, a research
workshop was then held in Chicago under the lead-
ership of John List and Michael Greenstone. A set
of project teams were given a chance to present
their current design and receive critical feedback
from leading academics within the E&E research
team to ensure the work is able to maximize its
contribution to the global knowledge agenda.
The original set of 12 impact evaluations was se-
lected from a funding window in Jan 2015 that laid
out the research program focusing on two pillars:
(i) environmentally sustainable electricity supply,
access, and efficiency; and (ii) natural resource and
sustainable land-management issues, with a focus
on incentive schemes, governance, and vulnerabili-
ty. Transport, industrial pollution, and urban devel-
opment relate directly to energy and environment
and present major development challenges with
important economic and environmental implica-
tions. However, these topics are addressed by other
programs.
The research agenda has benefited from direct en-
gagements with the World Bank’s Climate Change
Cross-Cutting Solutions Area, Energy and Ex-
tractives Global Practice, Environment and Natural
Resources Global Practice, Water Global Practice
within the World Bank, the Global Environment Fa-
cility (GEF), the Climate Investment Funds (CIF), the
Energy Sector Management Assistance Program
IMPACT EVALUATION TO DEVELOPMENT IMPACT53
(ESMAP), and DFID (including the evaluation de-
partment and climate-change teams). Work under
the main themes is summarized below.
Topic 1: Energy Access, Reliability, and EfficiencyThere is a growing body of evidence on the impacts
of energy access on health, education, and produc-
tivity. But, important questions remain about the
cost-efficiency of investments to balance the cov-
erage (extensive margin) and per-connection avail-
ability (intensive margin) of providing electricity.
The average annual energy consumption of elec-
trified households in Kenya is 20 times less than
the average American household. Therefore, under-
standing the demand and impact of different tiers
of access from solar lanterns (Tier 1) through to full
grid access (Tier 5) becomes an important concern
to help governments efficiently allocate resources.
There is an important trade off—while lower tiers of
access may provide less opportunities for economic
growth (for example, being unable to power large
appliances and machines), the benefit of lower in-
vestment costs and easier expansion may outweigh
this concern.
We explore this question programmatically by
looking at the demand for, and impacts of differ-
ent levels of energy access in Senegal (solar lan-
terns), Argentina (solar home-systems) and Kenya
(grid connections). We offer different subsidies in
the various projects to elicit demand curves for so-
lar-energy products and produce evidence that will
help policymakers learn how to set efficient subsidy
levels to balance expanding access with fiscal sus-
tainability for service providers.
Moving beyond access, there is emerging evidence
that focuses on the importance of providing reli-
able electricity for industrial development and the
effects of rural electrification on household welfare
finding strong negative effects on firm revenues
and producer surplus. The program is currently
working with two large infrastructure investment
projects in Nepal and Bangladesh—both of which
face acute energy constraints—to understand the
impacts of improving reliable energy access by re-
habilitating and expanding transmission lines and
upgrading grid substations.
Topic 2: Incentivizing Sustainable Land Use and Natural-Resource ManagementThe overuse of natural resources can be a result
of externalities, unclear property, or high discount-
ing of the future. Since natural-resource manage-
ment has both local and global implications, find-
ing the right interventions and policy approaches
to address these issues presents a challenge. For
instance, a systematic review on the effective-
ness of one of the most common policy interven-
tions used to overcome coordination failures—the
creation of decentralized forest management
groups—found limited evidence of reduced defor-
estation rates and could not reject the possibili-
ty that these programs have negative economic
consequences (Samii et. al., 2014). The objective of
this program is to generate knowledge on effective
ways to address the causes of unsustainable use
of natural resources.
One common intervention to address the external-
ities associated with sustainable forest and land
management is Payment for Ecosystem Services
(PES). The program includes four PES projects that
offer financial incentives to landholders to reduce
deforestation and promote sustainable land man-
agement in Uganda, Burkina Faso, Ghana, and Mex-
ico. Here, we explore the role PES incentives play in
reducing deforestation, but also whether alternative
livelihood options may help ensure the economic
well-being of beneficiaries and increase the sustain-
ability of these programs.
We also explore the dynamics associated with in-
centivizing long-term behavior change. For instance,
IMPACT EVALUATION TO DEVELOPMENT IMPACT 54
in Uganda we explore the impact of PES after in-
centives are removed. Does deforestation remain
low, return to pre-intervention rates, or increase to
catch up with total deforestation in control areas?
Each scenario has plausible justifications, but re-
sults in very different interpretations around the
overall role that PES schemes can play to mitigate
our impact on the climate.
Going ForwardSince the program began two years ago, the focus
on using rigorous evidence in the energy and envi-
ronment sectors has only modestly increased. It still
requires a more concerted effort to catch up to oth-
er evidence-led sectors like education and health.
The mapping of evidence to development projects
is currently skewed in favor of subtopics that are
more amenable to impact evaluation. While the im-
pacts of energy access have been a preoccupation
in current economic literature, the reality is that the
vast majority of development funds are directed to-
wards generation and supply.
Tackling questions on the drivers of energy avail-
ability to connected customers will be the primary
focus of the energy agenda, moving forward. This is
more aligned with the major development challeng-
es in the sector. The program aims to work with
utilities and other service providers to explore the
interplay between pricing, service delivery guaran-
tees, billing and payment schemes and enforce-
ment. This will help identify the bottlenecks and
associated solutions to optimally utilize electricity
infrastructure and provide reliable energy to house-
holds and industry.
For environment topics, we aim to expand the fo-
cus area beyond financial incentives, to also include
co-management practices and regulatory influenc-
es to better represent the major development tools
available to practitioners and policymakers.
6.5 Financial and Private-Sector DevelopmentProductivity is the key driving-factor for long-term
sustainable growth. Empirical research, specifically
at macro level, has mostly emphasized the deter-
minants of firms’ productivity. Much less is known
about how to increase firms’ productivity (Syver-
son, 2011). In reality, this challenge has to do with
(1) identifying ways to increase efficiency of factors
of production that are used idly, and (2) find in-
novative ways of combining factors of production
so as to increase growth potential. This question is
paramount, given that most developing countries
struggle with low levels of labor and total factor
productivity (The World Bank Enterprise Surveys,
1/2013).
The FrameworkThe Trade and Competitiveness (T&C) agenda in
DIME is structured to test and identify effective
ways of increasing firms’ productivity through both
efficiency gains and shifts in the production fron-
tier. Efficiency gains are understood as changes in
the production process to help firms move closer to
the efficient production frontier. The two assump-
tions underlying suboptimal allocation of inputs
are: (1) market imperfections and/or behavioral bi-
ases—such as misperception of returns associated
with a given business practice, lack of motivation
to adopt better production process (Gibbons and
Henderson 2012; Nguyen and Nguyen 2016); and
(2) organizational barriers that prevent firms from
adopting new technologies (Atkin et al., forthcom-
ing at QJE) and using inputs optimally. In this light,
adoption of new technology is key to firms’ (and
economic) growth.
In this context, technology change encompasses
any shock in the production process that leads to
higher output given the inputs available. That shock
could be caused by, for instance, better trained em-
ployees, use of better managerial practices, use of
IMPACT EVALUATION TO DEVELOPMENT IMPACT55
cheap credit lines in credit-constrained firms, and
business registration to access public services that
are only made available to formal firms. Technology
change and, thus, productivity growth go hand in
hand with technology adoption.
Shifts in the production frontier occur only when
factors of production are already used optimal-
ly, and could result from improvements in busi-
ness regulations, innovation, and infrastructure. It
is worth mentioning that interventions aimed at
moving out the production frontier are more dif-
ficult to evaluate with randomized controlled trials
for two main reasons. First, regulation policies usu-
ally involve changes in the current legislation and
such decisions are usually taken at higher levels and
are not necessarily theoretically grounded. Second,
those policies are highly likely to have general equi-
librium effects (spillovers).
The T&C Program
Some Background
Even though the T&C agenda in DIME accommo-
dates both efficiency gains and growth to increase
firms’ productivity, the projects in the pipeline tend
to be more concentrated on testing policies aimed
at raising firms’ productivity through the former. It
hasn’t always been like that.
In 2010, DIME and the T&C Impact Program (now
called ComPEL) at IFC co-organized the first IE
workshop that focused on interventions to improve
firm capabilities, such as matching grants and
training programs. Several training programs were
rigorously evaluated since then, but, almost all eval-
uations found null effects on jobs creation and firm
productivity (McKenzie and Woodruff 2012). Nine
matching-grants programs could not even be eval-
uated, because very few firms took up the program
(Campos et al. 2013). We learnt that supply-side
interventions should pay closer attention to con-
straints on the demand side, which can hinder in-
tervention participation and, thus, adoption of new
technologies.
Three years later, in 2012, DIME and the T&C Im-
pact Program co-organized a second T&C IE work-
shop. The objective was geared towards generat-
ing rigorous evidence on how to help/nudge small
informal firms to formalize their business. The
prevalent view was that reducing business regis-
tration costs (formalization) and simplifying tax
were necessary conditions to the growth of small
firms. The overwhelming evidence suggested that
most small firms do not want to formalize their
business, even when registration costs are fully
subsidized.
Even among firms that do formalize, the impact on
performance indicators (for example, sales and rev-
enues) are either small or null (Bruhn and McKenzie
2014). The evidence generated by the rigorous eval-
uations led to a change in IFC’s approach towards
small informal firms. It no longer sees formalization
as stepping stone for growth of informal firms. The
challenge now is how to make informal firms more
productive.
Current Portfolio
Improvements in the investment climate are still
central to the Bank’s agenda, but the projects in
T&C are now looking towards better understand-
ing issues that are under firms’ control and beyond
Find innovative ways of combining factors of
production so as to increase growth potential.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 56
firms’ control, to better attack the constraints
firms face to grow.
A good example on how to better tackle intervention
participation while targeting supply is that of our IE
in Brazil. Medium-sized firms will be randomly split
to receive information, information plus training,
and a placebo intervention. As evidence shows that
take-up of training programs is low, this IE aims at
measuring participation incentives. One group will
be offered 40 percent subsidy on training costs in
addition to information, while the second will be of-
fered 80 percent subsidy plus information. Unlike
most training programs, the training will be provid-
ed at firm location and consultants will pay weekly
visits to monitor the adoption of good practices. In
addition to testing the impact of different subsidy
rates on training take-up, this design will help the
implementing institution (a regional development
bank) identify how much firms are willing to pay for
a training program, but also if firms misperceive the
actual returns.
Another example of how we can face firm growth
constraints, is of another training-oriented IE, this
time in Georgia. In this case, the training will focus
on e-commerce and be offered to a random sample
of small firms. This is to ensure that they have the
basic skills to compete in the market and gain ac-
cess to markets. However, to also address demand
constraints, half of the treatment firms will then
be randomly assigned to receive additional step-
by-step trainings on product branding, marketing,
and various other online services. Access will be
granted through a self-selection process, where
firms will have to reach a specific goal at the end of
each level (for example,. receive ten online orders),
before continuing to the next step. This competi-
tion type process aims at increasing the odds for
firms to access a broader consumer market. The
idea is to develop some record on service quality
and measure if firms are more likely to receive new
customer orders.
Overall, the current T&C portfolio accounts for 26
IEs, distributed across 22 countries—nine in the
preparation phase, 11 ongoing, and six completed.
The total estimated budget is $14 million, of which
25 percent is funded through i2i. Over half the pro-
gram (16 IEs) evaluates World Bank projects, repre-
senting a total of $243.7 million in loans. In terms
of outputs, the T&C teams have produced six re-
ports, eight working papers, and two publications.
Going Forward The lessons generated so far have substantially
shaped the current T&C Global Practice (GP) agen-
da. A more holistic approach has been put in place
since the last IE workshop, held in Istanbul in May
2015. It has been maturing since then. The ComPEL
and DIME teams are pushing projects that were se-
lected for funding support (September 2015 call)
to take into account some of the cross-cutting
themes that were identified as major issues in pre-
vious IEs such as: (i) potential spillovers occurring
in the market; (ii) low take-up rate of input-based
interventions; and (iii) the importance of improving
intervention targeting.
Those cross-cutting issues gained momentum in
the T&C program and played a critical role in the
selection of projects that will attend the next IE
workshop in Mexico City between February 27 and
March 2, 2017. Three priority areas were recently
identified by the T&C GP as strategic for knowledge
generation through rigorous impact evaluations:
1. Firms’ access to markets and spillovers.
2. Identification and support to high-growth small
and medium-sized enterprises (SMEs).
3. Regulatory efficiency.
There is an increasing focus on how to improve
firms’ linkages to both consumers and larger firms
in the global value-chain. Interventions will then
have to look carefully at both supply (for example,
training and matching grants) and demand-side/or
IMPACT EVALUATION TO DEVELOPMENT IMPACT57
institutional constraints (for example, technology
adoption and diffusion, access to markets—busi-
ness-to-business and business-to-consumers, and
regulatory environment) to (i) maximize the chanc-
es for intervention success at least on implemen-
tation grounds, and (ii) increase the odds of policy
effectiveness.
6.6 Transportation and ICTDIME’s ieConnect impact-evaluation program seeks
to generate rigorous evidence on the impact of
large-scale development investments in the trans-
portation and ICT sectors. Transportation and as-
sociated infrastructure investments are critical in-
vestment sectors for developing countries. A large
percentage of lending by the World Bank and of de-
velopment finance and grants from other donors is
aimed at this sector. But, there is relatively limited
evidence of their impact through rigorous experi-
mental or quasi-experimental evaluation.
The goal of the ieConnect program is to generate
impact-evaluation evidence on transport policy and
investments, including indirect benefits, at suffi-
cient scale to substantially improve the evidence
base for policy making in selected evidence gap
areas. DIME’s ieConnect program, which was de-
veloped in collaboration with the Bank’s Transport
and ICT Global Practice and DFID, aims to fill this
gap by linking projects with research teams and en-
abling them to develop innovative and rigorous im-
pact-evaluation designs. The focus is on identifying
and estimating the impact of transport and ICT
investments themselves, as well as developing and
testing ancillary interventions that can maximize
their impact.
The program targets the subthemes of roads and
transport corridors, urban mobility, road safety, and
information and communications technology (ICT)
infrastructure. There are currently 11 impact evalu-
ations in progress or under preparation. Following
a planned expansion of the program for ieConnect
phase 2, there will eventually be 22 experiments in
the portfolio.
The ieConnect program was launched in Rio de
Janeiro at the ieConnect for Impact workshop in
June 2015. A call for proposals was made in Sep-
tember 2015 and projects were selected on a com-
petitive basis in December 2015. Work began in
earnest in 2016 on concept note development and
IE implementation. Over the course of the year,
there was important progress on the projects that
emerged from this original workshop and call for
proposals.
A key milestone occurred in June 2016. An in-
tensive technical quality/IE design-strengthening
and capacity-building workshop for East Africa
transport IEs was held in Nairobi. It was attend-
ed by a number of transport IE teams (Rwanda
rural roads IE, Ethiopia expressway IE, Nairobi
SmarTTrans IE, Nairobi BRT IE, and the Dar es
Salaam BRT IE), external academic experts, civ-
il-society representatives, Kenyan government
counterparts, as well as a group of World Bank
transport-corridor projects, which could become
candidates for ieConnect IEs as the program is
expanded in the future. The workshop enabled
strengthening of selected IE designs via targeted
technical assistance from other IE teams and ac-
ademic experts, initial preparatory work on poten-
tial future transport IEs, and continued capacity
building of government and project counterparts
on impact-evaluation methods.
The original IE work from the Rio workshop was
supported by seed funding from DIME’s main i2i
program, but the program was initiated with the un-
derstanding that the scale and data requirements
of transport IEs would necessitate the creation of
a separate, transport-specific sectoral IE program.
This expansion of the program, beginning in 2017,
will be possible thanks to DFID’s ieConnect phase
IMPACT EVALUATION TO DEVELOPMENT IMPACT 58
two grant, which was developed over the first half
of the year and submitted in July 2016. The grant
was finalized over the second half of 2016, setting
the stage for the kick off of ieConnect phase two
in early 2017.
Beyond these programmatic developments, the fol-
lowing are IE design and implementation highlights
from ieConnect’s work in 2016:
n In Rio de Janeiro, the IE team studying the im-
pact of women-only gender-segregated metro
cars has generated preliminary results, using ex-
perimental variation in pricing to show low will-
ingness-to-pay for the female-only cars, which
may be explained by low compliance with gen-
der-segregation policy by men.
n In Rwanda, the rural roads impact evaluation
team has completed baseline data collection in
all study communities, and has begun end-line
data collection in a subset of communities. In
addition, ongoing high-frequency data collec-
tion on prices and availability of goods in local
markets is in process. This will ultimately allow
estimation of the impact of roads on compre-
hensive measures of household welfare.
n In Peru, a concept note has been approved for
what is, to our knowledge, the first randomized
impact evaluation of a rural roads-rehabilitation
program. The IE focuses on economic impacts of
road rehabilitation and also includes a substan-
tial gender focus, examining the link between
road rehabilitation and women’s access to edu-
cation and health services.
n In Dar es Salaam, the Bus Rapid Transit System
impact evaluation has fielded a baseline survey
in early 2015, prior to the system’s Line 1 open-
ing in May 2016. An end-line survey for phase 1
of the project is planned for 2017. The IE plans
to cover successive BRT line as they open in the
coming years.
n In Nairobi, the concept note was approved for
the SmarTTranS impact evaluation, which will
test the effect of shifting information and in-
centives for safe driving in Nairobi’s informal bus
transit (matatu) system. The project team has
begun developing a state-of-the-art technolo-
gy and big-data system for real-time monitor-
ing and an empirically-validated measurement
framework for road safety.
n In Ethiopia, the Expressway Development Sup-
port IE (still in design phase) is working to ex-
pand its scope beyond evaluating the impact of
expressway construction from Mojo to Hawassa.
The IE team is seeking to study the joint effect
of these transportation improvements with the
impact of major industrial investments such as
the Hawassa Industrial Park, which will employ
thousands of young workers in light manufac-
turing in rural Ethiopia.
n Several other projects are also still in the design
phase, due to either significant redesign (Mau-
ritania) or changes on the client side (Bogota,
Rio de Janeiro). Others, such as the Tunisia ICT
IE, have been dropped because associated World
Bank lending operations were cancelled.
Finally, an important focus of effort for the team
in 2016 has also been preparations to scale up ie-
Connect’s work program in 2017 for phase two.
Additional phase two resources will be used to a)
continue pushing existing IEs to concept-note com-
pletion and IE implementation, and b) to develop
follow-up/additional experiments in phase 1 IE
sites, as well as to develop several completely new
projects. We are also working to expand the team
to deliver this expanded work program by hiring
economists (2), a data specialist (1), an operations
officer (1), a research analyst (1), and an adminis-
trative assistant.
Areas of focus in phase 2 will expand to include
more gender-specific interventions and gender-dis-
aggregated data collection, additional work on frag-
ile and conflict (FCV) or conflict-affected settings,
additional work on transport corridors, greater re-
IMPACT EVALUATION TO DEVELOPMENT IMPACT59
gional focus on South Asia (outside of India), and
increased collaboration with other MDBs including
the Asian Development Bank and the African De-
velopment Bank.
6.7 GenderGender equality for development is a core theme
of i2i work and cuts across all thematic areas.
We identify two areas of growth for the current
i2i gender analytical agenda: design and test gen-
der strategies in areas where gender-specific con-
straints have been identified; and build the evidence
on gender-specific market failures in underserved
areas of impact-evaluation practice. The themat-
ic coverage of the i2i gender program seeks to fill
the gaps in four areas identified in the 2012 World
Development Report, “Gender Equality and Develop-
ment”: (i) human capital, (ii) economic productivity,
(iii) access to finance, and (iv) empowerment. Over-
all, over half of the current i2i portfolio is planning a
gender-disaggregated analysis, while 17 percent of
i2i-supported IEs are testing interventions tailored
to address gender issues.
i2i supports rigorous evaluations of policy actions
that look to relax supply-side constraints (for ex-
ample, improving service delivery for clean water,
sanitation, and maternal care) as well as market
and institutional constraints (for example, reducing
systematic differences in earnings). In underserved
IE research areas, the i2i program places emphasis
on documenting gender constraints in the context
of infrastructure investments and governance, with
special focus on transport, electoral participation,
and women’s labor market participation in FCV
settings. The research agenda evolves with i2i’s
portfolio, fueling iterative learning. i2i operational-
izes this vision by providing technical and financial
assistance to policymakers to identify relevant gen-
der issues, designing appropriate policy action, and
testing their impact to motivate scale-up, scale-
down and, new testing.
Addressing Human Capital Gender Gaps through the LifecycleGender gaps in human capital are well-document-
ed, but there is little evidence on how best to close
those gaps. The i2i research agenda focuses on how
to design interventions that address women’s ac-
cess to and use of health services and education,
and reduce women’s vulnerability to shocks that
disrupt human-capital acquisition. An IE of a voca-
tional training program in Malawi (Cho et al, 2015)
found that family obligations limited participation
and resulting skills development for young women.
Another IE testing the impact of a business literacy
course for female micro-entrepreneurs with rela-
tively low education in five different states in Mex-
ico, finds significant improvements of managerial
skills (Iacovone et al., forthcoming). As a result, nine
new states in Mexico have submitted proposals to
expand the program to their states.
New IEs in Nigeria are testing supply and de-
mand-side interventions to increase women’s
access and use of medical antenatal and birth
services, and community-level interventions to in-
crease uptake of malaria-prevention technologies
and increase accessibility of anti-malarial drugs.
One specific study in Nigeria, measuring the impact
of entertainment education through soap operas
on attitudes and behaviors about safe sex and HIV
testing, finds positive impact on both outcomes
(Orozco et al., forthcoming). The study also shows
Gender equality for development is a core
theme of i2i work and cuts across all thematic areas.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 60
that, given the popularity of soap operas among
poorer and less educated households, they can be
used to positively alter attitudes and behaviors
of millions of individuals at very low costs around
many development issues.
Economic OpportunitiesWomen’s access to economic opportunities is undermined by their lower access to production inputs. Female farmers have less access to in-formation, as agricultural extension networks are dominated by men. In Malawi (BenYishay et
al, 2016) and Mozambique (Florence Kondylis et
al., 2014), i2i IEs showed that women can make effective extension partners: they are at least as good as men at encouraging adoption of im-proved technologies.
Throughout Sub-Saharan Africa, women are dis-
proportionately limited in their land ownership and
transfer rights. An IE in Benin showed that land
demarcation increases soil fertility investment in
female-managed landholdings, shifts household
decision-making, and reduces spousal conflict
(Goldstein et al, 2015). An IE in Kenya tested a
recent policy innovation, known as “microfranchis-
ing”, which provides unemployed participants with
a proven business model and the specific capital
and business linkages based on the hypothesis that
many unemployed youth would like to be generating
income, but lack both experience to be competitive
and the financial and human capital (Owen Ozier et
al., forthcoming). Early results from the study found
that, for young women, the program increased
self-employment. This is an important finding con-
sidering around 55 percent of urban women in Ken-
ya aged 15 to 25 are unemployed.
Another IE in Afghanistan is measuring the impact
of a program aimed at lifting the poorest out of ex-
treme poverty by providing a way to transition into
sustainable and profitable economic activities and
linking them with microfinance programs (Aidan
Coville et al., forthcoming). It applies the program in
a setting where female labor force participation is
among the lowest in the world (15 percent) and has
a strong focus on supporting female-headed house-
holds, tackling multiple constraints simultaneously
to provide households with a big push out of ex-
treme poverty.
Further, large parts of populations in developing coun-
tries do not have access to essential social services.
In Comoros Island, for example, an i2i IE is testing
the effects of temporary employment cash-for-work
program on social and economic outcomes of poor
households, and whether they vary according to gen-
der within household (Mvukiyehe et al., forthcoming).
Access to FinanceAccess to productive assets constrains women’s economic opportunities, whether women farm-ers or micro-entrepreneurs. An i2i IE in Rwanda
tested introduced targeted and pre-commitment
savings accounts. Initial findings show that wom-
en are more likely to earmark their savings to buy
durable goods, relative to men who invest in ag-
ricultural inputs, suggesting that intra-household
bargaining over resources plays an important role
in women’s investment decisions (Jones et al,
forthcoming).
In India, an IE showed that women that partici-
pated in a women’s empowerment and rural live-
lihoods program had improved access to loans, ac-
cumulated assets, and invested in education, which
Women’s access to economic opportunities is undermined by their
lower access to production inputs.
IMPACT EVALUATION TO DEVELOPMENT IMPACT61
further made them feel more empowered (Shah et
al., forthcoming). Early results from an ongoing IE
in Benin examining several incentive mechanisms
to attempt to get business owners to formalize
found that male business-owners formalized much
more than female business-owners (McKenzie
et al., forthcoming). Ongoing analysis is exploring
the reasons for lower formalization amongst fe-
male-owned firms.
In Dominican Republic, an ongoing IE is studying
the impact of financial literacy and job skills, es-
pecially benefiting women, on household-finances
management, savings, credit, use of formal-sector
financial products, ability to search for, obtain, and
retain formal employment, management of small
businesses, new businesses opened, and income
levels (Xavier Gine et al., forthcoming). An ongoing
IE in Malawi on identification and fingerprinting,
a topic that is at the heart of the development
agenda, is testing whether requiring fingerprint au-
thentication for transactions alleviates access to
credit more for females and improves repayment
more for those that borrow (Gine et al., forthcom-
ing). This is also important because it would make
it impossible for male relatives to seize control of
women’s assets on the death of the husband, as is
common in Malawi.
Promoting Women’s Empowerment and Agency for Economic Development
A growing body of evidence shows that placing wom-
en in the center of the development agenda can in-
crease efficiency in the management of institutions
and resources. Also, female leaders can have benefi-
cial impacts on social norms. The i2i research agen-
da focuses on using gender empowerment to com-
bat domestic violence, testing interventions such as
cash transfers and active labor-market policies to
economically empower women, and role of law and
justice in achieving gender equality, among others.
An ongoing IE in Azerbaijan, for example, tests the
extent to which free legal aid leads to greater le-
gal empowerment, improved dispute resolution, and
higher welfare from reclaimed income and benefits,
more stable household settings, productivity gains,
and a gradual move away from discriminatory
norms and practices (Bilal Siddiqi et al., forthcom-
ing). In Pakistan, an IE is evaluating the impact of
women-inclusion mandates and ratification in vil-
lage-level grant management, which imposes an
inclusion mandate that 50 percent of individuals
organized in a village have to be women (Gine et
al., forthcoming). As results come in, we will know
whether having more women in these village-level
bodies changes the composition of projects that
are funded and leads to a better overall allocation
of resources. Another IE in India is testing whether
privately-run kiosks offering access to government
services under the Right to Public Services Act al-
low women greater access to basic services, and
whether this changes their attitude (Daniel Rogger
et al., forthcoming).
Gender in Underserved Research AreasA notable opportunity for the i2i portfolio is to make
a dent in understanding gender issues in under-
served areas. Recent progress on this front includes
the transport sector, and economic and electoral
participation in fragile settings.
Transport
Reducing transaction costs by improving trans-
port infrastructure has the potential to change the
way women access markets. In Ethiopia, a large
expressway construction is combined with the
development of a large industrial zone. Since the
large majority of employment in the industrial zone
will be of young women, this will be an opportunity
to study the effect of a large labor market shock
(60,000+ jobs over a period of several years) on
young women’s economic and social outcomes in
IMPACT EVALUATION TO DEVELOPMENT IMPACT 62
the vicinity of the zone. A complementary interven-
tion will be set up to experimentally study the role
of skills, information, and access to employment
opportunities.
In Peru, an intervention to promote women’s ac-
cess to health services and education is being eval-
uated in the context of a rural road-rehabilitation
project. In Brazil, a new IE on gender-segregated
public transport tests the extent to which gender
segregation is beneficial for women (Kondylis et
al., forthcoming). Harassment in public transport,
and sometimes even risk of rape, limits women’s
movements, activities, and employment in many
developing countries. Results from this study are
expected to inform policies going forward on public
transport-systems in cities worldwide.
Electoral Participation in FCV Settings
Despite recent policy efforts to increase women’s
participation and representation in politics, sig-
nificant gender gaps remain. Less than 10 per-
cent of the world’s countries have a female head
of state and fewer than 30 countries have reached
the target of 30 percent female representation in
parliament. Further, women continue to have low-
er electoral participation rates than men and their
voting choices are often influenced by powerbro-
kers or household heads (Giné and Mansuri 2011;
Tripp 2001; Geisler 1995).8 Gender gaps in political
participation are especially pronounced in war-torn
settings, where women tend to disproportionately
bear the consequences of conflict (Buvinic et al.
2013; Sow 2012; Rehn and Sirleaf 2002). While
there are individual country cases where women’s
representation in governing bodies have increased
in the aftermath of civil war, such representation
has not necessary translated into their efficacy in
voicing policy preference or interest (O’Connell 2011;
Tadros 2011; Hogg 2009).
What explains gender gaps in political partici-
pation? A growing number of DIME IEs investi-
gate the effects of information-provision inter-
ventions and a variety of delivery mechanisms
designed to remove or circumvent these con-
straints. The underlying premise of these in-
terventions is that since information provision
can occur relatively quickly and at lower cost,
interventions designed to provide information
can potentially address the lack of awareness,
thereby promoting political participation (Gine
and Mansuri 2010; Kumar 2001).
Consistent with this intuition, a DIME impact
evaluation in Liberia investigates the positive
effects on the political attitudes and voting be-
haviors of rural women when they are provided
access to United Nations elections-related radio
programs. The results point to significant effects
of the intervention on women’s political participa-
tion, both on national and local levels. Worrying-
ly, though, the study finds no evidence of effects
on women’s political efficacy and empowerment
outside of the electoral context, suggesting the
need to complement such brief interventions with
more sustained interventions that tackle slow-to-
change constraints (on supply and demand sides)
that might be embedded in prevailing social struc-
tures and norms.
DIME is evaluating a number of interventions that
do just that. In Zimbabwe, for example, a DIME IE
tests the effects of an intervention designed to
reform village-level governance via horizontal pres-
sure on gender inclusion and empowerment. Like-
wise, in Liberia, another DIME IE investigates the
extent to which a nine-month civic education in-
tervention that provides men and women a forum
for monthly deliberation on governance, rights, and
gender-equality issues, helps narrow gender gaps in
political participation.
8 Evidence from recent public opinion surveys across African countries suggests that 36 percent of female respondents are not interested in politics and that 39 percent never discuss politics. 22 percent and 24 percent were reported for men, respectively (Afrobarometer, 2008, reported in Bleck and Michelitch 2011).
IMPACT EVALUATION TO DEVELOPMENT IMPACT63
Economic Participation in FCV Settings
Women’s access to productive assets and agency
over the household’s economic decisions is even
lower in fragile contexts. Yet, women’s access to re-
sources is particularly impactful on human-capital
investments that can help poor children in tough
places get out of poverty (Duflo 2003). i2i is sup-
porting a number of studies that aim to provide
economic opportunities to poor women in fragile
states.
In DRC and Indonesia, i2i is supporting the eval-
uation of unconditional ‘business grants’ to wom-
en for the creation of sustainable livelihoods and
for long-term poverty alleviation. A social-network
treatment will also be tested, in which participants
join a series of workshops from female mentors
that focus on building links between individual busi-
ness-owners. The relative impact and complemen-
tarities across these interventions will be captured
by the experimental design. In Tunisia, an IE is
testing the effect of capital injections to comple-
ment a more traditional income-support program
that supports the unemployed through short-term
employment opportunities. Focusing on vulnerable
women, the impact of this additional intervention
on long-run consumption and labor-market out-
comes will be compared to the outcomes of those
who merely participate in short-term labor-inten-
sive works (Mvukiyehe et al., forthcoming).
In Liberia, activities will be centered on developing
diagnostic studies, designing materials and mi-
cro-interventions, and testing these materials and
micro-interventions as part of the Liberia Youth
Opportunities Project (LYOP). The diagnostic stud-
ies will focus on understanding the concrete actions,
behaviors, and decisions that influence women’s
access to male-dominated trades and explore the
underlying preferences, information, and assump-
tions (conscious or otherwise). The goal is to step
back and research actual needs and obstacles on
the ground before jumping to conclusions about
solutions. It will pay special attention to considering
prevention and mitigation strategies for potential
unintended consequences of supporting females in
non-traditional or male-dominated fields.
Once the diagnoses have been carried out, the next
step will be to design and develop materials and
interventions to address the identified constraints
and safely support vulnerable females in these sec-
tors. A critical last step will be test and refine the
FIGURE 30: i2i IEs with Gender Component
Yes No
IEs Including a Gender AnalysisNumber (percentage) i2i IEs
73 (57%)
56 (43%)
IEs Evaluating a Gender-Specific InterventionNumber (percentage) i2i IEs
107 (83%)
22 (17%)
IEs Falling under the GenderCross-Cutting Solution AreaNumber (percentage) i2i IEs
98 (77%)
30 (23%)
IMPACT EVALUATION TO DEVELOPMENT IMPACT 64
material and interventions before taking them to
scale in the project, including in the impact evalua-
tions each project intends to undertake.
Portfolio overviewBased on current donor funding, the i2i portfo-
lio consists of 131 IEs spanning 53 countries and
covering all of the i2i thematic areas. At least 17
percent of the portfolio evaluates a gender-specific
intervention and 57 percent conducts disaggregat-
ed gender analysis.
6.8 Edutainment
Background Every year, the World Bank and client governments
invest millions in behavior-change campaigns across
almost all development sectors. However, many of
these campaigns are unconvincing, lack inspiring
narratives, and are communicated through out-
moded and uninteresting outlets such as billboards
and leaflets. Systematic reviews of these cam-
paigns, from risky sexual behavior to handwashing,
consistently show little or no effect on behavior, es-
pecially in the long term.
There is an unprecedented opportunity to use en-
tertainment media to change the lives of billions of
people, especially in urban areas.9 Entertainment
education or edutainment can be a game-changer
for development. Unlike traditional behavior-change
campaigns that convey abstract concepts and can
become repetitive quickly, educational narratives
are easier to follow and remember than abstract
information. Characters in mass media have the
power to be role models, inspire audiences to en-
gage in new thinking about “what is possible”, and
change the perception of what is “normal” and so-
cially acceptable behavior.
The 2015 and 2016 World Development Reports
respectively highlighted the untapped potential of
entertainment education and mass media in de-
velopment practice. However, the evidence base re-
garding the effectiveness of entertainment media
remains thin, especially to advise the scale up of
9 While access to TV and radio is almost universal in developing countries, consumption of internet entertainment is prevalent in urban areas. According to the 2016 World Development Report, last year there were 3.2 billion Internet users in the world and 8.8 billion Youtube videos were watched every day.
10 http://blogs.worldbank.org/voices/newest-weapon-against-hivaids-africa-mtv — https://www.youtube.com/watch?v=0mCv5U5LRG4 https://www.bloomberg.com/news/videos/2015-01-20/using-data-entertainment-to-combat-hiv-stigma — http://www.hollywoodreporter.com/news/mtv-shuga-viewers-twice-as-900752.
MTV Shuga: A Dramatic EvaluationProduced by the MTV Staying Alive Foundation, Oscar-winner Lupita Nyong’o starred in the first
two seasons of the show. MTV Shuga is a television drama that targets African youth. The show
is broadcast in over 70 countries, reaching over 500 million people worldwide. In the eight-month
follow-up survey, the DIME study finds that the treatment group was twice as likely to get test-
ed, reported fewer concurrent sexual partnerships, and reduced attitudes and behaviors related to
gender-based violence. Among female viewers, chlamydia infections were halved. These are substantial
impacts, especially in light of the limited effects found in other HIV behavior-change trials. The study de-
sign and preliminary results have been discussed in TEDx talks, Bloomberg TV, The Hollywood Reporter,
and WB blogs, among other media outlets.10
IMPACT EVALUATION TO DEVELOPMENT IMPACT65
entertainment media as a development tool across
different sectors. There is a lot to learn about the
best way to maximize the impact and minimize
unintended consequences of entertainment me-
dia, a powerful tool that is largely untapped for
development. DIME is starting to expand this evi-
dence base with ongoing experimental evaluations
that explore the relative effectiveness of radio
spots versus printed narratives to promote adop-
tion of solar lanterns in rural Senegal; the use of
a Nollywood (the Nigerian film industry) movie to
promote financial savings, and of the MTV Shuga
drama to reduce risky sex and gender-based vio-
lence in Nigeria.
A Multi-Sectoral Program The Entertainment-Education program was
launched in May 2016 to explore the use of en-
tertainment-education and, more generally, how
mass media behavior-change campaigns can be
designed to change perceptions of social norms,
achieve adoption, and sustain healthier behaviors.
A multi-sectorial tool, the knowledge agenda of the
first phase, focuses on edutainment applications to
promote social-norms shifting and behavior change
in reproductive health, gender equality, early years
education, water and hygiene, and violence preven-
tion, including gender-based violence. Thus, it aims
to contribute to the achievement of Sustainable De-
velopment Goal 3 - Good Health and Well-being, 4
- Quality Education, 5 - Gender Equality, 16 - Peace,
Justice, and Strong Institutions, and, more broadly,
of SDGs 1 - No poverty.
Impact evaluation studies being implemented or
planned explore topics like: the use of a Nollywood
movie to promote financial savings among entre-
preneurs (SDG 1); the impacts of the MTV Shuga
drama on risky sexual behavior and gender-based
violence (SDG 3, SDG 5, and SDG 16); the use of so-
cial-norms campaigns to encourage families to en-
roll girls in primary school (SDG 4 and SDG 5); the
relative effectiveness of radio spots versus printed
narratives to promote adoption of solar lanterns in
rural areas (SDG 7); the impacts of including enter-
tainment education in in-school life-skills programs
to reduce bullying and to prevent drug and alco-
hol consumption among young people (SDG 3 and
SDG 16).
This new program is being rolled out in the major
entertainment hubs of the world and is supported
by different Bank units and leading media hous-
es and research centers of edutainment from the
“Hollywoods” of the world, including MTV Staying
Alive Foundation, USC Hollywood Health & Society,
UCLA Global Media Center, Children’s Film Society
of India, the Asian Center for Entertainment Edu-
cation, and the TV networks Televisa (Mexico) and
Rede Globo (Brazil).
The impact evaluations of the first phase also ad-
dress important questions regarding the indirect
or spillover effects of mass media on communi-
ty members that may have heard about the pro-
gram messages through their friends; as well as
the role that social networks have in disseminating
and magnifying potential impacts. Finally, the im-
pact evaluations also study how best to reinforce
edutainment messages through new interactive
technologies, from mobile messaging to social me-
dia outlets to videogames.
DIME “Narrating Behavior Change” Workshop The official launch of the “Narrating Behavior
Change” program took place during a DIME im-
pact-evaluation workshop, jointly conducted with
the Inter-American Development Bank. The event
brought together 22 project teams from Lat-
in America, Sub-Saharan Africa, and South Asia,
and producers and researchers from leading media
organizations and universities to design the next
generation of impact evaluations of entertainment
media and behavior-change campaigns. The work-
shop outlined the evidence base and knowledge pri-
IMPACT EVALUATION TO DEVELOPMENT IMPACT 66
orities and through clinics, allowed project teams to
work with researchers to develop interventions and
evaluation proposals relevant to their projects.
Before-after comparisons show the workshop im-
proved knowledge of participants. In a seven-point
scale, the average score increased from 4.17 to 4.46
or 6.9 percent. As expected, individuals who scored
low during the pre-test, benefited the most from
the workshop: the scores for those that scored un-
der 4 in the baseline test increased from 3.12 to
3.75 or 20.4 percent. Over 90 percent of partic-
ipants reported being satisfied with the technical
content and to have learned what works and what
doesn’t to measure the impact of a program. The
positive feedback was also reflected by positive
emails from participants:
“The workshop was a huge learning curve for
me and I have come back to India enlightened
and feeling super confident about the next
stage of our work here, Vinta Nanda (India), Managing Director, Asian Center for Entertain-
ment Education and CEO, The Third Eye.
“Let the magic that we all witnessed last week
transform the world with all these projects,
ideas, connections, and impact evaluations,”
Lorena Guillé-Laris (Mexico), Director, Cinepolis
Foundation.
“I learned a lot and now I have the challenge to
share all ideas with my colleagues here at Ro-
berto Marinho Foundation. Today, it is so rare
to participate in a meeting of so high a level,
with so many interesting people from differ-
ent places and backgrounds. Congratulations!”
Monica Pinto (Brazil), Development Manager, Ro-
berto Marinho Foundation.
“Thank you for a wonderful gathering of the
most interesting leaders in EE and for pro-
voking stimulating discussions in plenary
and small teams throughout the week. I hope
many new EE collaborations will take place as
a result, and I look forward to working with
you all!” Sandra de Castro Buffington (USA), Di-
rector, UCLA Global Media Center.
Going ForwardDespite being launched a year ago, the Edutain-ment program has generated important knowl-edge in the field of mass-media entertainment.
DIME has or will soon have three published papers of
edu-tainment interventions to i) promote financial
literacy and savings among entrepreneurs in Lagos;
ii) reduce risky sexual behavior and gender-based
violence among youth in Nigeria; and iii) adopt solar
panels in rural Senegal. In addition to studying the
effectiveness of edutainment across sectors, these
advanced evaluations study the effectiveness of
different mass-media outlets (that is, movies, TV
series, radio spots, and printed material). Study re-
sults have been presented in academic, policymak-
er, and producer circles. As mentioned above, the
results have received media coverage beyond devel-
opment outlets.
The program team is raising funds to open a window that will support a new generation of edutainment research required to introduce edutainment into development mainstream. The
window will focus on innovations that can poten-
tially promote and sustain behavior change among
the largest number of individuals. Thematically, the
window would support research projects in the fol-
lowing sub-themes: Sex in the city, Stopping vio-
lence, Empowering men and women, Keeping clean,
and Playful learning. Table 5 provides a list of ongo-
ing and likely edutainment projects. Over the next
two years, the program will use its existing part-
nerships with development partners and leading
media houses to expand its research in the major
entertainment hubs of Brazil, Mexico, Nigeria, and
India, countries with large populations that produce
for their respective regions. This should facilitate
translating research evidence into development and
industry strategies for global impact.
IMPACT EVALUATION TO DEVELOPMENT IMPACT67
TABLE 4: List of Ongoing and Likely Edu-tainment Projects
Sub-theme Country Region Development ChallengeIntervention Evaluated IE Objective
General behavior change
Nigeria AFR The growth of small enterprises in developing countries is often restricted by poor financial and debt management of entrepreneurs.
Promoting financial literacy and savings of entrepreneurs through the Nollywood movie “Story of Gold” in Lagos.
Evaluate immediate and medium-term effects of movies on financial literacy, savings, and responsible borrowing.
General behavior change
Senegal AFR Many rural communities in Africa lack access to the electric grid. Solar panels provide clean light and energy.
Promoting adoption of solar lanterns through radio spots and printed narratives in rural Senegal.
Contrast the relative effectiveness of radio “narrative” spots and print media in awareness and adoption of solar lanterns.
Sex in the city
Brazil LAC Bahia has one of the highest teenage pregnancy rates in Brazil.
Combining live skills and edu-tainment to reduce teenage pregnancies in Bahia.
Measure the impact of a comprehensive life-skills program and the additional impact of edutainment.
Sex in the city
Kenya AFR Stigma against people living with HIV remains a challenge, contributing to lower HIV testing.
Mixing soccer, social media, and community screenings to reduce stigma and promote HIV testing.
Evaluate impacts of behavior change campaigns during national amateur soccer league on attitudes, HIV testing, and risky sexual behaviors.
Sex in the city; Stopping violence
Nigeria AFR Two million people get infected with HIV every year, disproportionally affecting African youth. Gender violence is prevalent worldwide.
Stopping HIV and gender-based violence through the television drama “MTV Shuga”.
Evaluate impacts of TV dramas on knowledge, attitudes, and behaviors of risky sex and gender-based violence.
Stopping violence
Mexico LAC Mexico has the highest bullying rates in OECD countries. Most bullying takes place in schools.
Stopping bullying in middle schools with edu-tainment and social media approaches.
Measure the programs’ impacts on bystanders self-efficacy to intervene behaviors towards bullying.
Stopping violence
Brazil LAC The government adopted new regulations against sexual harassment, common in public spaces.
Stopping sexual harassment and GBV in transport and public spaces through social-norm campaigns.
Fill evidence gap about how to combine different media outlets to reduce sexual harassment in public spaces.
Empowering men and women
Niger AFR Child marriage and fertility rates in Niger are among the highest in the world, restricting economic and human development.
Stopping child marriages and reducing fertility rates through “husband” clubs and behavior-change radio campaigns.
Fill evidence gap of gender social-norm campaigns targeted at men and the use of radio campaigns to change attitudes and behaviors around fertility.
—continued
IMPACT EVALUATION TO DEVELOPMENT IMPACT 68
TABLE 4: List of Ongoing and Likely Edu-tainment Projects
Sub-theme Country Region Development ChallengeIntervention Evaluated IE Objective
Empowering men and women
India SAR Many young women have low aspirations and are forced to work under exploitative conditions in spinning mills in Tamil Nadu.
Changing aspirations and stopping bonded labor through community screenings in Tamil Nadu.
Evaluate the programs’ impacts on aspirational alternatives of young women and parents and attitudes towards bonded labor.
Empowering men and women
Nigeria AFR Eight million girls in northern Nigeria are not in primary school due to social norms against female education.
Getting girls in primary school and reducing child social-norm campaigns and conditional cash transfers (CCT) in northern Nigeria.
Contrast relative effectiveness and social-norms campaigns on school enrollment, retention and completion rates, and child marriage rates.
Keeping clean
Ghana AFR Due to social norms and bullying, many teenage girls don’t go to school during their periods, affecting their self-esteem and educational achievement.
Behavior change campaigns for menstrual hygiene management and educational engagement in middle schools in Accra.
Fill evidence gap on how to maximize water infrastructure investments with behavior change.
Playful learning
Colombia LAC Access to quality school-readiness programs is limited in Colombia and developing countries.
Improving school readiness and healthy habits of young children through digital entertainment in Medellin.
Study the effectiveness of Fun Academy’s digital platform, which brings together teachers, parents, and students through homework and play.
IMPACT EVALUATION TO DEVELOPMENT IMPACT69
7. Innovation in Data Quality and Monitoring of Policy Influence
7.1 Monitoring System and ResultsMind MyIE is a monitoring system that aims to
give an overview of all i2i and/or DIME-support-
ed impact evaluations (IE). It is a user-friend-
ly cost-effective and low-maintenance data-
base-management system and M&E software
to track status and collect information for each
impact evaluation. It gathers data on all IEs: pro-
file and status, evaluation design, data-collection
aspects, monitoring and quality indicators, coun-
terpart details, influence on programs and policies
and, finally, produced documentation. It provides
over two hundred indicators or characteristics as-
sociated with each IE.
An Innovative PlatformThe MyIE Monitoring System collects over two hun-
dred indicators for each registered IE. In this sense,
it serves as a monitoring and evaluation instrument,
facilitating production of reports for intra-unit (DE-
CIE/DIME) monitoring purposes, inter-unit com-
munication (for example, World Bank Group Global
Practices), and outside reporting such as for donor
results framework.
More than just a reporting tool, the system is unique
in its approach and serves as a knowledge-generat-
ing platform. By collecting data on how IEs feed into
project design, support capacity building, influence
outside projects, and/or motivate scale up or down
of an intervention, the MyIE system was developed
with the objective to better inform policy decisions,
while focusing on research with “impact”. With
this view, the series of produced summary statis-
tics were created with a problem-based approach,
helping to understand the challenges and lessons
learned from the DIME/i2i portfolio. The system’s
outputs can be delivered in various dimensions,
supplying information on levels such as regions,
timeline, budget, evaluation designs, project lifecy-
cles, and themes. Finally, its added value is also ac-
FIGURE 31: Mind MyIE Framework
INPUT
Indicators on:
Evaluation Design
Policy Influence
Capacity Building
MIND MyIESYSTEM
OUTPUT
Reporting System
Knowledge Platform
Personal Dashboard
IMPACT EVALUATION TO DEVELOPMENT IMPACT 70
counted for in its efficiency and transparency gains,
the system delivering automatic reports with just
one click.
Features and ContentThe system asks of Task-Team Leaders (TTLs), or
other authorized respondents, to self-report during
the entire lifecycle of their respective portfolios. The
data is collected annually for all ongoing IEs and
currently translates into 172 i2i and/or DIME IEs,
of which 76 percent are DFID/i2i-funded. The lat-
ter thus encompasses the 131 IEs presented in this
annual report. A complete list of all collected indi-
cators can be found in the Indicators Section. These
are divided into eight sections: IE profile, Evaluation
Design, Data Collection, IE Monitoring and Quality
Indicators, DIME Involvement, Counterpart Details,
IE Influence on Program/Policy, Documentation and
Research Outputs.
Through the IE questionnaire tab (“Manage my Proj-
ects” in the system), users have access to a Reports
Section. It consists of graphs and tables for each of
the following:
n Map of IE distribution across the world
n IEs across GPs and sub-themes
n IEs across regions
n IEs across IDA countries
n IEs by lifecycle
n IEs by duration
n IEs involving gender components or analysis
n IEs in fragile and conflict-affected settings
n IEs and main counterparts
n IE budget distribution
n IE secured funding
n IEs across evaluation methods
n IEs across number of treatment arms
n IE data-collection rounds and response rates
n IEs that have ethical clearance and/or study
registry
n IEs that influence project design or
implementation
n IEs that generate evidence used to support proj-
ect adoption, scale-up, scale-down, continuation,
or cancelation decisions.
n IEs that have contributed to improving the cov-
erage, quality, delivery, output/outcomes, or cost
of the program or other interventions outside
the IE
n Number of previously-ongoing impact evalu-
ations completed and reported (for example,
as working papers or policy briefs published
online)
FIGURE 32: Mind MyIE Phases
Design Phases
CURRENT:
Enables Task Team Leaders to enter a large set of indicators on their IE portfolio
MEDIUM TERM:
Will also serve as a resource platform, giving users access to their personal dashboards
LONG TERM:
Incorporate all the World Bank’s IEs to the system
1 2 3
IMPACT EVALUATION TO DEVELOPMENT IMPACT71
Moreover, users currently have the option to enroll
in a MyIE competition, which is divided into three
categories: Best Concept Note Technical Rating,
Most Innovative IE, and Best Pictures.
What’s Next?In the short term, the MyIE monitoring system will
add to the resource aspect of the platform by giv-
ing users access to their own personal dashboards.
This will reflect the same indicators that are in the
Reports Section, but on a portfolio level. The disag-
gregated data will be presented relative to the over-
all average statistics, giving users an indication as
to where they stand in relation to other portfolios.
Additionally, a databank is currently in production.
This will allow users to directly download data from
the system, conditional on the latter being public.
Finally, in the long term, the system will have a
public-access feature, where unregistered users will
be able to obtain aggregated descriptive statistics
and access to documentation for completed evalu-
ations. Moreover, the final objective is for MyIE to
serve as a recording system for all World Bank-re-
lated impact evaluations.
7.2 Review of DataA new i2i data initiative, “DIME Analytics”, pro-
vides data-quality assurance to all i2i impact eval-
uations. DIME has spent the past seven years col-
lecting multi-module household surveys across all
regions, perfecting methods and protocols. Since
2009, DIME has supervised more than 208 sur-
veys.11 In the past year (2016), DECIE staff super-
vised 37 surveys (15 baselines and 22 follow-up
surveys), representing over $7.5 million in cli-
ent-executed funds. However, there is no platform
for sharing knowledge and standardizing protocols
across research teams. Moreover, the knowledge
is staying in-house, failing to reach researchers
and policymakers beyond the DIME team and ac-
ademic and policy partners. To bridge this gap, we
created DIME Analytics, which has the following
objectives:
1. Create a publicly-accessible DIME Wiki to ensure
that protocols, guidelines, and training materials
are available to all and remain up-to-date in a
rapidly-changing field.
2. Develop and implement a Survey Review process
to ensure that all i2i-associated surveys are fol-
lowing established best practices.
3. Advance the knowledge frontier on Survey Meth-
ods by identifying and capitalizing on opportuni-
ties to test new technologies for data collection
and cross-cutting research opportunities.
The DIME Wiki and the Survey Review go hand-in-
hand, as standardizing data quality across the i2i
portfolio requires both better resources for manag-
ing a high-quality survey and an accountability sys-
tem to ensure that best practices are implemented
and final data gets publicized.
DIME Wiki: a public good, targeted at all researchers
and M&E specialists at the World Bank, donor insti-
tutions, NGOs, and governments. It creates unique
value by providing a ‘one-stop shop’ for resources on
all phases of the survey process, from questionnaire
design to publication of the final dataset (see figure
33). It features both content developed by DIME and
links to existing resources.12 The Wiki structure of
interlinked articles perfectly suits the type of body
of knowledge we want to disseminate as there is no
starting point common to all users. It also has the
advantage of being familiar to future users around
the world thanks to Wikipedia. Articles with links
11 Data from the myIE monitoring system, this only refers to surveys associated with the i2i trust fund, and is a lower bound estimate for DECIE as a whole.
12 For example, methodological research from the Living Standards Measurement Survey team at the World Bank or impact evaluation resources from Innovations for Poverty Action or the Abu Jamal Poverty Action Lab.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 72
to detailed further explanations of concepts men-
tioned will make this resource equally accessible to
new and advanced users. Most importantly, unlike
a paper manual, it is a dynamic, searchable, interac-
tive platform with a community of users.
Survey Review: We are currently piloting the review
process for six IEs with surveys planned in the
first half of 2017. The review process is a mecha-
nism to identify common weaknesses, link teams
with existing resources on survey best practices,
and create accountability to follow best practices
and generate the best data. It is not a one-time
review, rather a dynamic and ongoing engagement
with the IE team, with review of key materials trig-
gered at specific milestones in the survey-prepa-
ration process. The review will start a minimum
of four months ahead of intended data collection.
Materials will be approved at each key milestone
in the survey-preparation process: sampling strat-
egy, Terms of Reference for survey firm, survey in-
strument(s), fieldwork supervision, and data-qual-
ity checks.
Survey Methods: The review process will allow us to
identify opportunities for methodological research,
both in terms of testing new technologies for data
collection (for example, satellite data, unmanned
aerial vehicles, crowdsourcing) and rigorously test-
ing options for survey planning and design. Among
the survey resources under development are ‘gold
standard’ modules, based on the most recent meth-
odological research and high-quality programming.
Offering off-the-shelf gold standard survey modules
will enhance data comparability. This opens up new
possibilities for portfolio management, comparing
outcomes across space and time, and re-defining
their development targets and project design based
on solid evidence.
We anticipate that the DIME analytics will bene-
fit the i2i portfolio, but will also create high-value
public goods, with particular benefit for govern-
ment counterparts in the line ministries collabo-
rating on DIME IEs. We experience high demand
for training in data generation and analysis from
government counterparts in line ministries. Our
workshops are an excellent introduction to im-
pact evaluation and research methods, but teams
need further resources to implement high-quali-
ty data collection. Offering survey methods and
resources through an open, interactive platform
with accessible off-the-shelf solutions to the
most common survey design and implementation
roadmap will contribute to filling these gaps in
coverage.
FIGURE 33: Impact Evaluation Data Lifecycle
Sampling Questionnairedesign
Piloting Questionnaire programming
Survey firm procurement Field plan Enumerator training Data quality monitoring
Data management &cleaning
Data analysis Review of analytical code Publication to microdatacatalogue
IMPACT EVALUATION TO DEVELOPMENT IMPACT73
Off-the-Shelf Solutions for High-Quality Surveys
The survey resources that will be developed by our team will include the following:
Manage High-Quality Surveys: Field Coordinator Training Workshop n Knowledge aggregation process: We have developed a well-regarded and highly in-demand training on
managing successful impact evaluations. Every year, the workshop grows significantly in terms of
both participants and content. The most recent workshop (June 2016) included 30 sessions, 16 of
which were interactive lab sessions, and 50 participants (the course filled to maximum capacity well
before the start). Much of the training is focused on data collection and analysis, and applies to all
IE surveys.
n Impact on quality of future surveys: Participants learn best practices in sampling for project surveys,
procuring survey firms, supervising data collection, and cleaning and analyzing household survey
data. They will be introduced to the Survey Wiki and all survey resources developed by our team.
Best Practice Data Collection Protocols n Knowledge aggregation process: DECIE has supervised more than 208 surveys since 2009, and
Analytics staff have personally been involved with more than 50 surveys at DIME. We will rely on
that experience to document agreed-upon best practice protocols and share with our network of
field coordinators for inputs and comments. The revised version will be a dynamic document, as it
will live on the Wiki, where all users can suggest additions or revisions (in a process moderated by
our team).
n Impact on quality of future surveys (our output): We will create guidelines for each key step of the survey
process, with associated checklists for field coordinators, covering topics such as questionnaire
design, piloting, ethical clearances, survey-firm procurement, high-frequency data checks, fieldwork
supervision, and more. This will provide a learning resource for anyone preparing to collect data, and
eliminate the need for researchers to train every new team member. Checklists will make sure that
even experienced staff do not forget key steps during the busy survey-preparation process.
Research Transparency n Knowledge aggregation process: Pilot a system for internal peer review of data-analysis coding.
Develop a series of trainings on doing reproducible research in collaboration with the Berkeley
Institute for Transparency in the Social Sciences (BITSS). Create standard guidelines for
anonymizing survey data, and set protocol for publishing data in the microdata catalog.
n Impact on quality of future surveys (our output): All peer-reviewed analytical outputs will also undergo
a code review in which all code is re-run to ensure that results replicate. All survey data will be
anonymized and submitted to the microdata catalogue in a timely fashion in accordance with the
newly-set protocols.
—continued
IMPACT EVALUATION TO DEVELOPMENT IMPACT 74
Software Codes and Guidelines n Knowledge aggregation process: Compile requests for coding assistance from DECIE’s network of 50+
field coordinators and research assistants, and organize into a searchable library. Requests currently
come to the data coordinator on the team, who is well placed to categorize common concerns and
develop code snippets with optimal solutions.
n Impact on quality of future surveys (our output): Improve coding from programming of the survey
instrument to data cleaning and analysis, by providing off-the-shelf solutions and a reference library
for programmers. Research assistants follow optimal coding conventions, and have easy-to-access
resources for help with common problems with working with survey data. Coding solutions are
standardized across the team, increasing consistency and transparency.
Gold Standard Survey Modules n Knowledge aggregation process: We will first review a subset of the 200+ existing questionnaires to
identify commonalities, focusing on questionnaires identified as best practice by the implementing
team. Then, we will review the survey design literature for existing research on best practices. We
will summarize findings in a two-page design note for each module, and prepare an example gold-
standard module, programmed to be used in CAPI surveys. Coding will also reflect best practice. The
modules and accompanying design notes will be available to all teams as off-the-shelf solutions,
which they can mix-and-match to create high-quality surveys.
n Impact on quality of future surveys (our output): Teams have access to carefully developed standardized
survey modules. This will improve the quality and consistency of the data, by increasing quality
of survey content (especially in contexts in which extensive piloting is not feasible) and reducing
programming errors. It will also increase the comparability of data across surveys, creating new
research opportunities.
IMPACT EVALUATION TO DEVELOPMENT IMPACT75
8. Cases of Policy Influence
Introductioni2i works with clients to incorporate a dynamic
agenda that generates lessons during all stages
of the IE. This includes using existing evidence to
guide policy design in real time to incorporate evi-
dence from baseline analysis, adopting tested caus-
al mechanisms, and finally, to making decisions on
scale-up or down of interventions. This type of
influence or advocacy can be critical to delivering
an instrument of sustainable development into the
hands of the policymaker.
i2i has provided support in varying dimensions by:
(i) allowing for a more programmatic approach to
evidence-based policymaking; (ii) capacity building
around a broad set of stakeholders through poli-
cy-relevant research agendas; (iii) expanding the
reach of underdeveloped areas in impact evaluation
(IE) and evidence-based policy. These are all critical
elements to facilitate impact and influence policy
at different stages of a project and the policymak-
ing process.
In particular, i2i has funded a web-based monitor-
ing system (MyIE) that will complement current
advocacy and campaigning efforts by reporting on
periodic progress of IEs through the life of i2i as
more IEs are completed at different phases of their
lifecycle. These indicators showcase the diversity of
influence at all levels and well beyond the typical IE
results and impact.
TABLE 5: Selection of Policy Influence Indicators Collected in Mind MyIE System
Section 7: IE Influence on Program/Policy*
IE actions led to improved counterpart M&E *
IE data requirements led to improvements in monitoring and evaluation (M&E), data collection and/or reporting activities of the counterpart(s).
IE team provided training for data analysis **
IE team delivered skills training to local institutions and/or staff for general monitoring and other data analysis independently (through discussions, technical assistance, workshops, and other training channels).
IE team provided other types of training**
IE team delivered training to local institutions and/or staff on other topics than data analysis.
Rationalized policy design * IE improved design based on clear understanding of the underlying theory of change (causal links between the intervention components and the outcomes) and highlighted areas of uncertainty and critical assumptions.
IE baseline results discussed with clients **
If baseline results were discussed with client (if IE has baseline).
Baseline informed policy design/implementation **
IE baseline data was used by governments and other stakeholders to stimulate policy dialogue and/or help identify problems and solutions.
IE final results were discussed with clients ***
IE analysis and results were discussed with the client to understand their policy relevance and application.
—continued
IMPACT EVALUATION TO DEVELOPMENT IMPACT 76
Below we present four case studies that exemplify
how i2i-funded IEs have influenced program design,
supporting regulation, and scale up.
Catalyzing Regulatory Reform in KenyaThe Kenya Patient Safety Impact Evaluation (KeP-
SIE) recently started implementation of the inter-
vention. But, there are already two significant con-
tributions to policy during the pre-implementation
phase, which includes design, baseline, and prepara-
tion for implementation: (1) an enhanced regulatory
framework for health inspections, and (2) strength-
ened institutional capacity through inspection pro-
tocols and an inspection-monitoring system.
KePSIE is a unique partnership between the Ken-
yan Government and the World Bank Group to de-
sign, implement, and evaluate an inspection system
for public and private health facilities in Kenya. This
system aims to improve patient-safety standards
(practices that reduce the probability of preventable
harm to patients and healthcare workers during the
process of healthcare, such as availability of hand
hygiene supplies and new syringes), which consti-
tute a public health problem in Kenya and across
the world.
Estimates suggest that globally, approximately
42.7 million adverse events result every year from
unsafe medical care in inpatient services. Most are
in low and middle-income countries.13 According to
the project’s baseline data, 97 percent of health
facilities in the three KePSIE study counties—Ka-
kamega, Kilifi, and Meru—are non-compliant with
minimum patient-safety standards. To assess
the extent to which governance and accountabili-
ty mechanisms can improve patient safety, KeP-
SIE experimentally allocates all private and public
health facilities in these three counties to one of
three arms: (1) high-intensity inspections with en-
forcement of warnings and sanctions for non-com-
pliant facilities; (2) high-intensity inspections
with enforcement of warnings and sanctions for
non-compliant facilities coupled with public disclo-
sure of inspection results; and (3) “business-as-usu-
al” low-probability inspections (the control group).
Enhanced Regulatory Framework: At the outset
of KePSIE, stakeholders recognized the progress
achieved with previous reforms but identified the
following challenges: (i) unclear and discretionary
rules of the game; (ii) lack of incentives to improve
patient safety at different levels of compliance with
the standards (for example, unclear (virtually inexis-
tent) sanctions and weak enforcement, except for
extreme cases of malpractice); and (iii) inadequate
13 Jha AK, Larizgoitia I, Audera-Lopez C, Prasopa-Plaizier N, Waters H, and Bates DW. The Global Burden of Unsafe Medical Care: Analytic Modelling of Observational Studies. BMJ. 2013; (10): 809-15.
TABLE 5: Selection of Policy Influence Indicators Collected in Mind MyIE System
Section 7: IE Influence on Program/Policy*
Adopted causal mechanism(s) based on IE results ***
IE evidence from experimental testing of alternative mechanisms was used by governments or other stakeholders to determine most effective program alternatives or to inform policy decisions.
IE results used to motivate scale-up/scale-down of policy ***
IE results reported success (or insufficient) impact of the intervention in achieving desired outcomes and were used by governments and/or other agencies/stakeholders to motivate scale-up (scale-down) of policy.
Presentations to (non-client) policymakers and academics of IE results ***
Details on presentations given to policymakers on the IE results.
* applicable after Concept Note review. ** applicable after baseline results are available and discussed. *** applicable after final results are available and discussed
IMPACT EVALUATION TO DEVELOPMENT IMPACT77
government capacity to inspect and monitor a suf-
ficiently large number of facilities (around 4 percent
of facilities are inspected in a given year). As a re-
sponse to this assessment, KePSIE’s first output
was a new regulatory framework—gazetted under
Cap 242, Legal Notice No. 46 on March 21, 2016—
to conduct inspections in both private and public
facilities (only private facilities were inspected prior
to the reform).
This new framework includes the following ele-
ments: (A) a refined Joint Health-Inspection Check-
list (JHIC); (B) a scoring system that allows facili-
ties to be categorized according to the level of risk
presented to patients; and (C) scores that trigger
warnings and sanctions to be enforced according to
a facility’s level of risk.
Building on previous reform efforts that led to the
first JHIC in 2012, this constitutes one of the most
comprehensive efforts to monitor patient safety in
the region so far. To give some context, of 45 coun-
tries in the Africa region with de jure inspection
regimes, only five (South Africa, Mauritius, Namib-
ia, Equatorial New Guinea, and Seychelles) actual-
ly carry out any type of inspections, and that too
mostly for private health facilities (IFC, 2011).
Strengthened Institutional Capacity: At the out-
set, there was no job description for inspectors, no
training materials or protocols, no monitoring sys-
tems, and no institutional arrangement to link the
Ministry of Health, the regulatory bodies participat-
ing, the three county governments, and the inspec-
tors for an intervention such as the one we were
designing. The team developed all these materials,
standards, and protocols as the basis for the imple-
mentation and the monitoring function, reflected
in an operations manual and a training manual for
inspectors.
Currently, there is no administrative system to
check for inspections results in the country. All are
paper-based records. A system was put in place to
conduct inspections by tablets and manage prog-
ress and monitoring through a web-based monitor-
ing system that reports progress, performance, and
challenges in real time. This customized solution
then leads to the availability of timely and action-
able information to identify challenges in the im-
plementation and enhance accountability to make
mid-course corrections, without the intensive use of
resources, expertise, or equipment, commonly ab-
sent in poor-resource contexts.
Supporting an Adaptive Learning Agenda in Tanzania
As part of the broader engagement with DFID,
DIME supports the testing of an “adaptive learn-
ing” approach to DFID’s Payment by Results (PbR)
program, which started in 2015 and will run up to
2019. The £150 million PbR is DFID’s flagship pro-
gram in Tanzania, aiming to improve rural water
access. One of the key drivers of low water cover-
age in rural Tanzania is a disproportionate focus on
building new schemes, at the cost of maintaining
existing ones. Of the approximate 74,000 rural wa-
ter points in Tanzania, an estimated 46 percent are
non-functional at any point in time. To the extent
that those can be fixed at a reasonable cost, in-
creased efforts on maintenance could dramatically
increase coverage, improving welfare in various ar-
eas ranging from health to agricultural productivity.
PbR’s aim is to shift the focus and resources of gov-
ernment into maintenance, by providing local gov-
ernment authorities (LGAs) with bonus pay-outs to
reward increased functionality at the district level.
As part of this effort, DIME’s role is to provide DFID
with evidence on how to tackle the water mainte-
nance challenge.
In the spirit of adaptive learning, which places the
focus on the outcome rather than the particular in-
tervention modality, the evaluation has a wide re-
mit—it explores the general bottlenecks of that en-
vironment, going beyond estimating PbR’s impact
IMPACT EVALUATION TO DEVELOPMENT IMPACT 78
itself. So far, DIME has provided support in three
ways. First, the team worked with DFID to analyze
the program’s design (for example, incentive struc-
tures, funding flows) and incorporate available evi-
dence into programmatic decisions. Second, DIME
conducted formative research to generate a “thick
description” of the program’s context and challeng-
es. This included both quantitative and qualitative
research. Finally, DIME is currently designing a pro-
cess evaluation and an experimental evaluation to
tackle some of the challenges identified in the for-
mative research.
Although DIME’s research is at an early stage and
the experimental work has not started, the team’s
work has directly and indirectly influenced policy
design and implementation.
Understanding the bottlenecks to water main-tenance: The first year of the partnership (2015-
2016) was dedicated to building a “thick de-
scription”, to better understand the challenges
surrounding rural-water maintenance. DIME as-
sembled a multi-disciplinary team (including econ-
omists, engineers, and an anthropologist) and
conducted a combination of quantitative analy-
ses using secondary data and qualitative research
through an in-depth anthropological study of two
districts. On completion, DIME provided partners
with a set of five notes shedding light on various
drivers of water-point sustainability. Those drivers
included: specific characteristics of water points
(for example, age, extraction method, location); in-
stitutional characteristics (local government staff-
ing, capacity); and political cycles. Some of those
findings are briefly presented here.
Generating continuous feedback to support adaptive learning: A key characteristic of DIME’s
partnership with DFID is the “adaptive learning”
aspect of the PbR. The flexible operational model
means that findings can be incorporated directly
into DFID’s ongoing program, not only into future
ones. After formative research in Year 1, the focus
of Year 2 is to provide DFID with a process evalua-
tion, as the PbR starts disbursing its first payouts.
Project expectations will be compared with actual
implementation, possibly highlighting bottlenecks,
which will be analytically reviewed. The evidence will
be used to update the implementation strategy for
the rest of the program.
Using experimental research to calibrate PbR’s impact through a review of incentives and in-complete contracts: Combining field visits with the
learning generated from the formative research, the
evaluation team, together with DFID and govern-
ment counterparts, identified an important bottle-
neck to PbR’s impact. Tanzania’s Water Supply and
Irrigation Act states that minor repairs of water
infrastructure are the responsibility of communi-
ties, whereas major repairs are the responsibility of
LGAs. However, this distinction is necessarily am-
biguous, and not effectively enforced on the ground.
This creates an incomplete contract between com-
munities and local government, which typically
leads to shirking of responsibilities.
While the government is moving towards full de-
centralization, increasingly emphasizing all-encom-
passing roles for communities, these communities
still require substantial support from government.
Therefore, it is unclear whether LGAs will have the
incentive or institutional structure to translate PbR
payouts into improved functionality. It is suspected
that they may require a more direct approach to
overcoming the inherent problem of incomplete con-
tracts, and the associated vagueness in responsibili-
ties. The agreed impact evaluation will thus aim to
bridge this gap by strengthening the co-production
arrangements between communities and govern-
ment. In particular, the impact evaluation will roll
out a program that brings community water lead-
ers to the LGA offices on a regular basis to receive
training on maintenance-related issues and medi-
ation to resolve existing maintenance concerns.
Perhaps, more importantly, this will create an op-
portunity for government and communities to dis-
IMPACT EVALUATION TO DEVELOPMENT IMPACT79
cuss maintenance challenges face-to-face, and how
they can collaborate to solve them. The objective
is to help LGAs develop a clearer strategy for how
PbR (and other government) funds could be used to
address sustainability challenges in their commu-
nities. Through this experiment, DIME is combining
state-of-the-art contracts theory and evaluation
methods to produce a tailored solution, which aims
to magnify the impact of a large program on rural
water access.
Using IE to Shift the Policymaking Culture in Nigeria
DIME has worked in the Nigerian health sector since
2007. Beginning with a single evaluation focused
on expanding malaria-related public-health services
using the private sector and community volunteers,
DIME developed collaborative relationships to gen-
erate and use IE evidence to inform policymaking.
This was possible through partnerships with the
Federal Ministry of Health and its agencies, includ-
ing the National Primary Health Care Development
Agency, the National Malaria Elimination Program,
the National Health Insurance Scheme, and the Na-
tional Agency for the Control of AIDS.
These relationships served as the foundation for
a series of IEs with the common objective: to pro-
mote and support innovative thinking and gener-
ate evidence to improve health outcomes in Nige-
ria. In working with our partners, DIME promoted
and implemented IE research to address multiple
constraints in the health sector. At the same time,
DIME used this process to promote a shift in the
policymaking culture towards evidence and re-
sults-informed decisions. With this, we positioned
ourselves as a knowledge broker and provider of
high-quality research services and policy advice.
In 2012, DIME entered into a formal partnership
with the Federal Ministry of Health and the Bill &
Melinda Gates Foundation to use IE in strategic
government programs aiming to reduce maternal
and child mortality. Through this partnership, DIME
conducted IEs on expanding the availability and ac-
cessibility of primary healthcare services, improving
the quality of these services, and motivating front-
line health workers. In a further partnership with the
Gates Foundation, DIME tested whether embedding
culturally relevant messaging in mass media could
alter norms and behaviors related to sexual and re-
productive health and gender-based violence, areas
that are notoriously difficult to change.
DIME IEs include a series of research and analyt-
ical products that provide input at all stages of
policy design and program management. Examples
of such inputs provided through our Nigeria health
work include:
Program design DIME research introduced variations in program design to learn which is most effective, answering questions such as “What is the added value of providing ongoing monitoring and coaching to public primary health facilities—in addition to information on baseline quality—in terms of improving the quality of care?”
Program delivery DIME supported the delivery of routine program elements, such as working to improve the timeliness of payments to frontline health workers (subject to budget constraints).
Documenting experience DIME helped to document the experience of interventions that did not make it to scale (“learning from failure”), such as a conditional cash transfers for maternal health and a community-monitoring intervention to reduce drug stock-outs at primary health facilities. This will help avoid the re-learning of the same lessons in the future.
—continued
IMPACT EVALUATION TO DEVELOPMENT IMPACT 80
This is a snapshot of the type of policy influence
achieved through a sector-wide approach to us-
ing IE to deliver better health outcomes. Achieving
such results across programs and agencies re-
quires staying power to progress beyond capacity
constraints, unpredictable institutional environ-
ments, and external shocks. Notably, this work
provides a bridge for policy continuity: while this
research was largely launched under President
Goodluck Jonathan, discussions on the implica-
tions for future policy are now being held with the
administration of current President Muhammadu
Buhari. This would not have been possible with-
out DIME’s extended engagement in the Nigerian
health sector, and the building over time of a net-
work of persons committed to changing the sta-
tus quo. It is only through long-term investment
that change takes effect. While others may enter
for the quick win, DIME and Nigeria have engaged
for the long haul.
How Partnerships Can Have Sector-Wide Influence in Rwanda DIME’s work with the Government of Rwanda be-
gan in 2011 under the Global Agriculture and Food
Security Program (GAFSP) and is an example of
how governments can take on a sector-wide ap-
proach to impact evaluation (IE). What started as
one GAFSP-financed evaluation, evolved into a large
portfolio of IEs in the agricultural sector, driven by
the keen interest of the Rwandan Ministry of Ag-
riculture and Animal Resources (MINAGRI) to sys-
tematically learn from robust evidence.
This program of IEs contributes to building science
of delivery in a number of areas: investment in
large infrastructure (terracing, irrigation, and feed-
er roads), rural finance, accountability in extension
service delivery, as well as understanding mecha-
nisms for operation and maintenance of rural roads
and irrigation projects. The scale and lifecycle varies
across each of the research areas, and has led to
studying several unique but interlinked questions in
a close partnership where research feeds into pro-
gram design and operations.
For long-term large-scale projects like the Land
Husbandry, Water Harvesting, and Hillside Irriga-
tion (LWH) program, the evaluation aims to for-
mally document the impact of the intervention in
project sites, using as a comparison group similar
pre-identified watersheds that will not receive LWH
project activities. In addition, there is an active col-
laboration between DIME and MINAGRI to strate-
gically test delivery mechanisms on a number of
sub-interventions that aim to directly affect the
results of the broader LWH program.
Capacity building DIME provided IE training to hundreds of policymakers and government technical staff and established the Nigeria Health IE Community of Practice, a forum for exchange of health research and impact evaluation-related capacity building (comprising government, civil society, and the private sector) to stimulate the exchange of ideas and cross-programmatic learning.
Policy debate and design DIME engaged high-level policymakers, including the Minister of State for Health, in a debate on the implications of recent IE findings and provided inputs into the draft National Health Policy (NHP).
Data generation and documentation
DIME IEs have collected a wealth of microdata in a country that, despite its size, has limited rigorous health research, that too in areas that are infamously data poor. This data serves as an input into DIME’s own analyses, but beyond these there is great potential for its use to understand and inform additional aspects of policy design.
IMPACT EVALUATION TO DEVELOPMENT IMPACT81
One example of this included running a random-
ized control trial (RCT) with 80 self-help groups in
Karongi district in 2012-13 aimed to introduce and
rigorously test two new types of savings products:
a targeted savings account and a commitment
savings account. An important component of LWH
was transitioning from subsistence to commercial
agriculture. DIME worked with the rural-finance
specialist to create and rigorously test innovative
financial products designed to help farmers man-
age their finances to provide for their families and
have money available at the beginning of each sea-
son to purchase agricultural inputs.
The new savings products found promising results
and, on this basis, the LWH team decided to test
similar products at scale and worked with DIME to
implement an RCT with five agricultural cooperatives
in Rwamagana district in 2013-14. However, program
scale up did not translate into similar positive im-
pacts when rolled out beyond the careful manage-
ment of the project team. Based on the evidence of
the lack of impact in the larger program in Rwama-
gana, the LWH team decided not to extend the pro-
gram to other districts. This presents an important
lesson. While many IEs run proof-of-concept evalu-
ations to advise national scale up, testing the effec-
tiveness of a program on a larger scale in a more re-
al-world environment is critical to understanding the
true potential impact of program scale-up.
In addition to rural finance, testing the modalities to
improve extension services are central to LWH’s aims
of capacity-building and technology diffusion. In LWH
project areas, farmers purchase agricultural services
(inputs and extension) from One Acre Fund (OAF).
DIME worked with OAF and MINAGRI to design, intro-
duce, and test innovative farmer-feedback tools. The
government was interested in monitoring the exten-
sion services provided by OAF, and farmer satisfaction
with the service. In addition, the research team was
interested in whether feedback tools could actually
increase demand and improve low participation rates.
Together, the team set up a large field experiment, in
which two types of feedback tools were randomly as-
signed to groups of OAF clients. The team also tested
the cost-effectiveness of different feedback modal-
ities. The most cost-effective feedback mechanism
piloted (a hotline) was adopted and scaled up by OAF
throughout Rwanda the following season. In addition,
the satisfaction data drawn from the feedback tools
themselves helped convince MINAGRI to continue its
partnership with OAF and scale it up to new LWH sites.
Over the course of the partnership between DIME
and MINAGRI, there has been consistent efforts to
build local capacity surrounding the design and de-
ployment of impact evaluations. The DIME team
has led four technical workshops for officials within
the ministry, ranging across IE design and methods
to the use of statistical packages. In addition, par-
ticipating in LWH missions from very early on, the
research team was able to contribute to and inform
specific elements of program delivery, including mo-
tivating the government’s interest in piloting, test-
ing, and scaling up different complementary inter-
ventions—the rural finance experiment in Kayonza
and Rwamagana being prime examples.
Moving forward, through its partnership with local
counterparts on the Rural Feeder Roads project,
DIME is putting together a large-scale data system—
digitizing existing records and working closely with
government stakeholders to set up the infrastructure
to gather information in ways that will alter the way
in which the government seeks to pose policy ques-
tions and deliver relevant results. Finally, by testing
the science of delivery around the sustainable oper-
ation and maintenance of irrigation systems, DIME
and MINAGRI are displaying how the iterative pro-
cess of program implementation, evaluation, and pol-
icy retooling can be leveraged to deliver significantly
improved results. This presents a sector-wide ap-
proach to learning through individual project cycles
and linking the learning across projects to build com-
prehensive evidence-led sector strategies.
IMPACT EVALUATION TO DEVELOPMENT IMPACT83
9. Communicating Lessons and Results
Strategy
The i2i dissemination strategy last year focused
on raising global, regional, and local country-level
communications coverage of i2i-related activities
among key audiences in the development commu-
nity, particularly country clients, governments, do-
nors and shareholders, researchers, and operational
staff at the World Bank Group and other IFIs.
In parallel, the strategy emphasized outreach via a
variety of existing and new technologies including
the web, social media channels, blogs and other av-
enues more easily accessible and where efficiency
and impact could be magnified.
Stocktaking 2016
Research economists continued to work with proj-
ect clients and government officials to document
evidence during several points of project lifecycles.
Outputs produced at this level include 20 baseline
reports and 16 IE reports. Baseline reports and IE
reports stimulate policy dialogue and support the
adoption of casual mechanisms based on results.
These can influence the scale-up (or scale-down) of
a policy at the national level.
A number of DIME workshops and smaller confer-
ences and events were held during the past year.
As projects and programs mature and produce
results, our outreach focus has shifted. Where, in
previous years, a lot of emphasis was on launch-
ing new programs, for example through our large-
scale global workshops, DIME has expanded this to
sharing evidence generated at different stages of
projects. This is done both by bringing together de-
velopment practitioners, academics, and donors in
conferences, as well as in country events with local
policymakers at different levels.
A number of DIME workshops and smaller confer-
ences and events were also held during the past
year. This included (1) a policymaker summit in
Nigeria summarizing and presenting several years’
work in the health sector of the country, (2) an im-
pact-evaluation workshop in Mexico on Entertain-
ment Education aligning researchers, practitioners,
and producers from entertainment hubs including
Hollywood, Nollywood, and Bollywood to design and
evaluate behavior-change campaigns under a new
DIME program, (3) an evidence day in Kenya bring-
ing together all IE work in the country, and final-
ly, (4) a number of workshops and conferences in
Washington, DC in various thematic areas including
child labor and agriculture. In all, 25 DIME seminars,
workshops, or trainings took place over the course
of the year.
The team also continued to garner wider policy
outreach through (a) policy briefs, (b) World Bank
working papers, and (c) journal submissions. This
past year (March 2016 to February 2017), seven
working papers were published under the catego-
ry of impact evaluation at the World Bank Group.
The team produced 26 policy briefs and seven blog
posts, providing an opportunity for researchers to
proffer advice or solutions stemming from research.
IMPACT EVALUATION TO DEVELOPMENT IMPACT 84
Below are several examples of how dissemination
takes place in practice at different stages of a proj-
ect or program.
Narrating Behavior ChangeThe Entertainment Education Program component
of DIME highlights how the World Bank can use its
comparative advantage in research and convening
power to bring a variety of stakeholders to intro-
duce and mainstream a development practice. Two
World Bank Flagship publications (the 2015 and
2016 World Development Reports) highlighted the
untapped potential of entertainment education and
mass media in development practice. In 2016, DIME
launched a research program on entertainment
education. The program is supported by leading
researchers, development partners—for example,
Gates, DFID, and IADB—and global media power-
houses—for example, MTV Staying Alive Founda-
tion, USC Hollywood Health and Society, UCLA
Global Media Center, Rede Globo, Brazil, and Third
Eye, India. It uses DIME’s existing partnerships with
traditional and non-traditional partners to expand
the research agenda in the entertainment hubs of
Brazil, Mexico, Nigeria, and India. The generated
evidence base should support the systematic and
effective scale up of edutainment in development,
from radio to TV to videogames to mobile apps. The
Bank’s External and Corporate Relations (ECR) unit
supported dissemination activities of the new DIME
program. We received media coverage from glob-
al media outlets such as TED talks, Bloomberg TV,
MTV, and The Hollywood Reporter, among others.
To this end the unit held an inaugural DIME work-
shop in Entertainment Education in Mexico City. The
new program brought together 22 project teams
from Latin America, Sub-Saharan Africa, and South
Asia with producers and researchers from leading
media organizations and universities to design the
next generation of entertainment research. Further
outreach ensued from an MTV Shuga launch with
the MTV Staying Alive Foundation in New York and
also through individual outreach efforts conducted
by the program leader at various venues including
USAID events, Morelia Film Festival, and the Rocke-
feller Foundation.
Educating the Masses in Financial Literacy in BrazilBetween 2010 and 2011, DIME supported the larg-
est experimental evaluation of a financial-literacy
program for high school students. The pilot program
was sponsored by the Brazilian Committee of Fi-
nancial Education (CONEF) and took place in six
Brazilian states and covered approximately 900
schools and 20,000 high-school students. The in-
tervention was intensive and generated positive
results on learning, attitudes, and behavioral out-
comes. The results of the IE were disseminated in
policy talks and academic seminars, and were cov-
ered extensively by the main newspapers and mag-
azines in the country.
This evaluation had important policy implications as
the Brazilian Ministry of Education decided to scale
up the program nationally. Based on this successful
experience, CONEF decided to test a new pilot tar-
geting primary-school students under the assump-
tion that impacts should be stronger, the earlier
students are introduced to financial concepts. The
The Entertainment Education Program component of DIME
highlights how the World Bank can use its comparative
advantage in research and convening power.
IMPACT EVALUATION TO DEVELOPMENT IMPACT85
new pilot was implemented in 2015 in two states,
Santa Catarina in the south and Manaus in the
north. The pilot included 100 schools and covered
about 18,000 students aged six through 15.
This is the largest RCT of a pilot program targeting
this age group and the first conducted in a develop-
ing country to find positive effects of a financial ed-
ucation program on student learning and attitudes.
The results were disseminated in an event in May
2016 that brought together donors, academics, and
policymakers and were covered by two of the main
newspapers in the country.
Mozambique: Localizing Knowledge and Globalizing Outreach Promoting sustainable irrigation and drainage is es-
sential for smallholder farmers to be resilient to the
intensifying climate variations. DIME has developed
a long-term research agenda aimed at understand-
ing (1) how to best leverage irrigation investments
to increase resilience of small farmers and (2) how
to build local institutions to ensure sustainability of
these schemes. In Mozambique, the team developed
a series of studies aimed at shedding light on these
critical research questions.
The team is involved with several stakeholders in
the Mozambique irrigation sector and, since incep-
tion of the project, has sought to maximize learning
across projects and stakeholders. Extensive moni-
toring systems have been set up with government
counterparts and, last year, comprehensive agricul-
ture baseline surveys were collected for both proj-
ects. The baseline results provided useful insights
into the way the beneficiaries are selected as well
as potential ways to improve water efficiency by
altering watering practices.
In November 2016 DIME organized a brown bag-
lunch event in the World Bank country office, bring-
ing together representatives from the Ministry of
Agriculture and Food Security, including regional
representatives, Ministry of Land, Environment, and
Rural Development, the National Irrigation Insti-
tute, World Bank, AfDB Operations, and DFID. The
session covered baseline results data systems em-
ployed for each project, challenges, lessons learned,
and next steps.
The results were shared with other irrigation-re-
lated projects in the global agriculture portfolio at
the World Bank during a recent DIME Agriculture
conference in Washington, DC as well as at Johns
Hopkins University. A working paper and policy brief
results are currently being drafted.
Kenya Evidence DayThe World Bank’s active lending support to Ken-
ya amounts to around $6.8 billion, of which, around
$700 million (or 12 percent) is associated with im-
pact evaluations. Despite the fact that Kenya is
one of the largest producers of impact evaluations
in the world, evidence from the country often does
not reach intended audiences. To this end, the DIME
group brought its convening power and organized
a one-day event bringing together policymakers,
practitioners, and researchers for a high-level dis-
cussion on how evidence can be used to generate
relevant insights from conception to completion of
projects. The day was also utilized to showcase a
series of ongoing impact evaluations that provide
examples of how to do this in practice.
The workshop focused on how IE can and has been
used to improve government programs along two
dimensions: (1) strengthening accountability sys-
tems to improve service delivery; and (2) maximiz-
ing the impact of large-scale investments in public
infrastructure.
The event included a combination of (i) presenta-
tions of the latest evidence and learnings from six
ongoing impact evaluations in the country; and (ii)
panel discussions on what more can be done to im-
prove accountability systems and maximize pub-
lic-infrastructure returns. The event led to a stra-
tegic approach to combining fragmented research
and a set of comprehensive lessons to inform activ-
ities across operations, research, and policy.
Reconciliation in Sierra Leone
A recent paper on reconciliation in Sierra Leone ti-
tled “Reconciling after civil conflict increases social
capital but decreases individual well-being” was ini-
tially published in Science Magazine, a partnership
effort led by Innovations for Poverty Action. This
highlights how a topic of global interest can quickly
and easily find extensive dissemination coverage at
all levels of outreach.
The article created a media blitz and was men-
tioned significantly in a number of news outlets
around the world including in the Washington Post,
Le Monde, and Reuters. The paper also had heavy
social-media outreach and subsequent mentions by
policymakers, researchers, and journalists.