Impact Evaluation to Development Impactpubdocs.worldbank.org/en/741221447880138458/i2iAnnualReport.pdf · capacity constraints; coordination failures and transaction costs in establishing

Annual ReportMarch 2014–2015

Impact Evaluation to Development Impact

IMPACTDISCONTINUITY

ENER

GY

EVALUATIONA

GR

ICU

LTU

RE

ENVIRONMENTTRANSPORT A

NA

LYSI

S

DIFFERENCE-IN-DIFFERENCE

PROPENSITY SCORE MATCHING

RES

EAR

CH

CLI

MA

TETRIALRANDOM GOVERNANCE

CONFLICT

ICT

SOCIAL FRA

GIL

ITY

TR

AD

E

MARKETS

CO

MPE

TIT

IVEN

ESS

FINANCE

VIO

LEN

CE

EVEN

T

HEALTH

EDU

CA

TIO

N

CONTROL

MA

TCH

REGRESSION

EDU

-TA

INM

ENT

Departmentfor InternationalDevelopment from the British people

Transforming DevelopmenT Through impacT evaluaTion


– iii –

Overview

Impact Evaluation to Development Impact (i2i) is a World Bank multi-donor trust fund program launched in March 2014 with support from the U.K. Department for International Development (DFID). i2i’s mandate is to generate high-quality and operationally relevant impact evaluation (IE) research to transform development policy, reduce extreme poverty, and secure shared prosperity. Its objective is to increase the use of IE in the design and implementation of public policy, strengthen country institutions for evidence-based policy making, and generate knowledge in strategic development areas. The program aims to serve as a global platform with the purpose of building donor partnerships in expanding the use of IE across the developing world. i2i is managed and delivered by the Development Impact Evaluation Unit (DECIE) in the Research Group of the World Bank and adopts a Bank-wide governance structure that comprises the Bank’s Regions and global practices.

i2i achieves its objectives in the following ways:

♦ Developing and running experiments in collaboration with government partners to inform their policy decisions

♦ Building agencies’ capacity to do this systematically

♦ Drawing lessons and sharing them face-to-face with global audiences

i2i is aimed at funding IEs across all World Bank sectors with a special focus on under-evaluated areas. It covers 12 operational sectors of the Bank across four pillars and two cross-cutting themes. The pillars are shared prosperity, governance, climate change, and human development. The cross-cutting themes are gender, and fragility, conflict, and violence. Based on current donor financing, support is not available for the human development sector, including education, health, and social protection. As new donors join, i2i aims to support this sector.

i2i adopts the Bank’s DIME operational model. DIME is a global program managed by DECIE in the World Bank, which has developed institutional structures to ensure that its IE products are relevant, and can influence the decision-making process. Internally, DIME works with a cross-institutional council composed of chief economists and directors from operational vice presidencies who provide strategic guidance; with working groups in each global practice setting learning priorities and selecting IE cases; and with each operational team for the selected cases to shape project design and structured experiments to guide project implementation toward greater effectiveness. Externally, DIME engages clients from the beginning to set research questions, adapt policy implementation, and agree on entry points to affect policy decisions using experimental results. This model, adopted

ExEcutivE Summary

i2i aims to generate evidence on which programs and policies work by targeting important, relevant, and demand-driven policy questions and empowering policy makers to promote evidence-based policy making in developing countries.


– iv –

by i2i, transfers knowledge and tools needed to support evidence-informed policy making to country institutions through a medium-term learning-by-doing approach. i2i also focuses on building capacity among other intermediaries of development policy to help move the global development agenda forward. As a result, all research products are jointly owned.

i2i aims to generate evidence on which programs and policies work by targeting important, relevant, and demand-driven policy questions and empowering policy makers to promote evidence-based policy making in developing countries. Most important, by engaging the government counterparts throughout the IE cycle, from identification of policy questions, to the design and implementation of the IE, i2i puts governments in the driver’s seat to test innovations and scale up solutions to their most pressing policy issues. Through this approach of identifying pathways from policy to results, i2i’s purpose is to shape the way development work is done at scale and in a large number of contexts and practices.

Deliverables during Year OneBased on current donor funding from DFID, i2i has committed to supporting 125 IEs (65 new and 60 ongoing) during the period March 2014–March 2019. This report describes the activities during the first year of i2i implementation (March 2014–March 2015). During this year, i2i has supported the launch of four programs, building capacity for IE design and implementation and developing portfolios of IEs in the areas of (i) fragility, conflict, and violence; (ii) agriculture; (iii) energy and environment; and (iv) governance. It has supported 40 new and 44 ongoing IEs, overdelivering its targets for the year by around tenfold.

As a result, the current i2i portfolio of IEs spans across 42 countries. For new IEs, the portfolio has the following distribution across the i2i pillars and cross-cutting themes: fragility, conflict, and violence (14 IEs); agriculture (13 IEs); and energy and environment (13 IEs). With further funding windows in the process and in planning, i2i will support an additional 25 new IEs by


– v –

September 2015 in the areas of governance; trade and competitiveness; finance and markets; and transport and information and communication technology (ICT). For ongoing IEs, the portfolio has the following distribution across the i2i pillars and cross-cutting themes: shared prosperity (19 IEs), climate change (7 IEs); governance (11 IEs); gender (5 IEs); and fragility, conflict, and violence (2 IEs). Across the entire portfolio of new and ongoing IEs, 16 IEs (19 percent of the portfolio) include a gender-specific intervention and an additional 36 IEs plan to conduct disaggregated gender analysis.

i2i has completed four capacity-building workshops during its first year of implementation in the areas of fragility, conflict, and violence; agriculture; energy and environment; and governance. i2i workshops serve three main purposes. First, they train government counterparts and other partners in IE methods to give them tools for active participation in the development and implementation of their IEs. Second, they expose participants to the latest available evidence in each focus area, with a view to incorporating relevant evidence in their own program and policy design. And third, each country team is paired with one or more IE researchers to develop a prospective IE that will answer research questions of direct relevance to their specific program. Most important, i2i workshops equalize opportunities to access financial support across the development community by training participants to design technically strong IE proposals.

has completed four capacity-building workshops during its first year of implementation in the areas of fragility, conflict, and violence; agriculture; energy and environment; and governance.


– vi –

behavior of students resulted in a policy decision by the Ministry of Education to scale up the program nationally. An evaluation in Mozambique showed that training and placing women in extension delivery positions in agriculture benefits the broader population of women in terms of technology awareness and adoption. The results are being considered in the design of a new agriculture extension policy in Mozambique. In Nigeria, evidence and learning from a set of coordinated evaluations of maternal and child health interventions are producing evidence to reduce both maternal and neonatal mortality, saving lives of mothers and babies. The results are informing health policy discussions with the incoming Nigerian government.

Results to date are encouraging, but there remain important challenges to realizing the full potential of IE research, including research capacity constraints; coordination failures and transaction costs in establishing researcher–policy maker relationships; and limited understanding and ability to integrate IE into the implementation of policies and programs at scale.

The i2i operating model is designed to overcome these difficulties and to maximize both the private and public good value of IE. This is done through (i) a programmatic approach to using IE knowledge for more effective policy; (ii) matching policy makers with researchers from the outset; (iii) building project-specific research teams to implement IEs in collaboration with government agencies; (iv) emphasizing technical quality control and policy relevance; and (v) ensuring a project-specific aspect to global dissemination and policy outreach.

In so doing, i2i works to improve the state of knowledge and to improve the effectiveness of policy around the world. All i2i-supported IEs build on ongoing and completed work to create virtuous cycles of learning and policy impact.

Policy RelevanceGovernments and donors are increasingly recog-nizing the value of IE as a tool for more effective policy making. Prospective multi-arm IEs designed to reliably identify cause-effect relationships based on counterfactual analysis can guide decisions over which policies and programs are better able to achieve desired objectives, and how to best design these programs for maximum impact.

A study using data from Bank projects approved between 2005 and 2011 finds that projects with IEs are better at implementing development activities and do so in a timelier manner. Using disbursements against agreed activities as an objective measure of implementation, the authors estimate that IE increases average cumulative disbursements by two-fifths (40.8 percent) and reduces the planned-to-actual disbursements gap by one-half (54 percent). The results suggest that IE is a powerful tool to move projects from design to implementation. In so doing, projects are more likely to achieve their objectives, supporting the idea that project financing and IE research are complementary services provided by the World Bank.

Furthermore, i2i IE-specific case studies show that IE significantly contributes to generating policy-relevant evidence and affecting development policies. For example, results from a large-scale evaluation of a school-based financial education program in Brazil that found impact at improving financial knowledge, attitudes, and

Results to date are encouraging, but there remain important

challenges to realizing the full potential of IE research, including

research capacity constraints.]


– vii –

About This ReportThe i2i Annual Report serves as a management tool for the i2i program and as an annual implementation update for donors. The report summarizes the progress made by the different i2i program components and main financial flows during its first year of implementation. It is submitted to the donor, DFID, within three months of the end date of the first year of implementation. This report covers the reporting period from March 2014 to March 2015. The next reporting period will cover March 2015–March 2016, which coincides with the DFID fiscal year.

Contact InformationFor further information on this report, or any inquiries about the i2i program, please contact the following i2i team members:

Arianna Legovini, DIME and i2i program manager; [email protected]

Florentina Mulaj, i2i program coordinator; [email protected]

Anushka Thewarapperuma, i2i communications and knowledge management; [email protected]

Eric Mvukiyehe, i2i program lead for fragility, conflict, and violence; [email protected]

Florence Kondylis, i2i program lead for agriculture, rural infrastructure, and gender; [email protected]

Aidan Coville, i2i program lead for climate change, trade and competitiveness, and finance and markets; [email protected]

Vincenzo Di Maro, i2i program lead for governance and transport; [email protected]


– vii –


Abbreviations and Acronyms AA Administrative Agreement Bank World Bank BM&FBOVESPA Brazilian Stock Market CDD Community-driven development CFI Client Feedback Instrument DECIE Development Impact Evaluation Unit DFID Department for International Development, U.K. ENEF Estratégia Nacional de Educação Financeira EOI Expressions of Interest ESW Economic and Sector Work FCV Fragility, Conflict, and Violence FMIS Financial Management Information Systems FMoH Federal Ministry of Health GBP U.K. Pounds GDP Gross Domestic Product GoB Government of Brazil GRM Grant Reporting and Monitoring HWWS Handwashing with soap i2i Impact Evaluation to Development Impact IE Impact Evaluation IFC International Finance Corporation IO International Organization LIPW Labor Intensive Public Works MCH Maternal and Child Health MDGs Millennium Development Goals MSMEs Micro-, small, and medium enterprises MSS Midwives Service Scheme NGO Nongovernmental Organization PFM Public Financial Management SBA Skilled Birth Attendant SLM Sustainable Land Management SMEs Small and Medium Enterprises TA Technical Assistance TE External Training TSSM Total Sanitation and Sanitation Marketing TTLs Task Team Leaders USD U.S. Dollars VPUs Vice Presidential Units


– ix –

tablE of contEntS

Executive Summary iii

about this report vii

contact information vii

i. background and context 1

ii. Snapshot of the i2i first-year Deliverables 5

iii. management and Governance 9

iv. Program components and operating model 11

v. research Pillars 19

a. Shared Prosperity 19

agriculture 19

financial and Private Sector Development 20

infrastructure 21

b. Governance 22

civil Service reform 23

Justice 24

Public financial management 24

Subnational Public Sector management/Decentralization 24

c. climate change 24

Energy 25

natural resources management 25

D. Human Development 25

Health 26

Education 26

Social Protection 27

E. Gender 27

Human capital Gender Gaps through the life cycle 27

Economic opportunities 28

access to finance 28

Promoting Women’s Empowerment and agency for Economic Development 28

f. fragility, conflict, and violence 29

Jobs for at-risk youth as a tool for resilience 29

Public Sector Governance 30

Sexual and Gender-based violence 30

urban crime and violence 30

vi. Portfolio overview and results framework 31


– x –

vii. monitoring & Quality assurance 39

viii. Knowledge Generation, capture, and Dissemination 43

appendixes 47

a: i2i iE concept note template 49

b: i2i iE concept note budget template 63

c: i2i results framework 67

D: i2i Workshop Participation 71

E: i2i monitoring indicators 77

f: iE case Studies 85

case Studies: client Engagement and Policy influence 85

case Study 1: brazil financial literacy 85

case Study 2: taking Pilot Evaluations to Scale in tanzania 86

case Study 3: minding the Gender Gap in mozambique: identifying Gender-Specific barriers and integrating Gender into Development operations through impact Evaluation 88

case Study 4: the Evolution of Programs Designed to increase utilization of Skilled birth attendance in nigeria: Executive Summary 89

G: i2i Supported impact Evaluations 93

H. Policy influence of World bank impact Evaluations 105

impact Evaluation Helps Deliver Development Projects 105

background: Development aid and World bank impact Evaluation 105

investigating the role of impact Evaluation on Project Performance 106

results: How impact Evaluation Helps Projects Get Down to business 106

conclusion: Project financing and iE research are complementary 107

i: client feedback on iE Products 109

background 109

iE Survey results 109


– 1 –

i. bacKGrounD anD contExt

i2i’s mandate is to generate high-quality and operationally relevant impact evaluation research to transform development policy, help

reduce extreme poverty, and secure shared prosperity. Its objective is to increase the use of IE in the design and implementation of public policy, strengthen country institutions for evidence-based policy making, and generate knowledge in strategic development areas. i2i achieves its objective by (i) developing and running experiments in collaboration with government partners to inform their policy decisions; (ii) building agencies’ capacity to do this systematically; and (iii) drawing lessons and sharing them face-to-face with global audiences.

A Paradigm Shift in Development PolicyThe effectiveness of development policy largely depends on the quality of projects through which policy is delivered. Increasingly, evidence shows that project-level determinants are even more important than country factors in determining success. i2i aims to engage partners to generate scientific evidence on the mechanisms through which effectiveness is secured and systematically adopt the results to jointly achieve development goals and fight poverty. i2i’s main characteristic is that it puts the best development researchers at the service of policy makers to empower countries to find their own solutions to their most critical policy questions.

Results to date are encouraging, but there remain important

challenges to realizing the full potential of IE research.]

impact Evaluation to Development impact

– 1 –


– 2 –

What Makes i2i Different from Other IE Initiatives?First, i2i adopts a bottom-up approach that puts government agencies in the driver’s seat on the road from problem to solution. To do so, i2i

♦ Identifies knowledge gaps with the World Bank’s global practices and Regions to define the most important issues to be addressed in the research studies;

♦ Builds country capacity with trainings, workshops, and midterm learning-by-doing collaborations;

♦ Offers an open platform for getting expert advice and support to teams that conduct IE;

♦ Equalizes opportunities in selection of IE products for technical and financial support;

♦ Provides support to the teams throughout the IE cycle, from design to implementation, analysis, dissemination, and policy action; and

♦ Generates knowledge and builds theory on the policy mechanisms that define policy success.

Second, i2i supports high research quality and policy relevance to ensure that the answers provided are actionable. To do so, i2i works to secure the quality of (i) methods used for causal analysis; (ii) data generation; (iii) study protocols and implementation; (iv) analysis and peer review; and (v) collaboration and policy dialogue to ensure policy actions at multiple stages of the policy cycle.

Third, i2i uses IE as a formative tool to generate evidence throughout the project life cycle. This differs substantially from summative and ex post impact assessments. i2i works with clients to incorporate a dynamic learning agenda that generates lessons during all stages of the IE, from using existing evidence to guide policy design in real time to incorporating evidence from baseline analysis, to adopting tested causal mechanisms, and to taking decisions on the scale of interventions. By testing the underlying causal mechanisms through which policies deliver results, i2i helps to understand and identify regularities in human behavior and policy delivery that might be applicable to different countries and sector contexts.

i2i leverages its position within the World Bank’s DIME program to work in full collaboration with operational teams to create jointly owned research products. Its operating model transfers the knowledge and tools needed to support evidence-informed policy making to country institutions through a learning-by-doing approach. IEs focus on testing specific mechanisms to understand which of these explain policy successes and failures and increase impact. In this way, i2i provides actionable evidence to its implementing partners while aggregating the experience of multiple IEs to generalize and build development theory.



– 3 –

How Impact Evaluation Can HelpMeasuring causal links is of utmost significance for policy decisions because basing the decisions on correlations alone may result in ineffective or even damaging policies. When governments react to an economic recession by toughening immigration policy, for example, the slowdown in migration flows is often attributed to the enforcement efforts, although the slowed economic conditions themselves may be directly reducing the pull on migrants. This type of policy error often goes unnoticed.

Done well, IEs can deliver precise estimates of the cause-effect relationship between policy action and outcomes by comparing predefined treatment and control groups before and after a policy intervention. This is referred to as causal inference or counterfactual analysis. Establishing the direction and magnitude of causal relationships can guide policy makers and donors as to which policy should be used to achieve the desired objective, identify the policy’s cost-effectiveness, and justify a response or a scale-up. When used to compare operational alternatives, IEs can guide and improve the effectiveness of policy interventions. This requires that IEs be prospective; in other words, that they be designed before an intervention takes place in order to measure, during implementation, the causal links between an intervention and changes in short-, medium-, or longer-term outcomes.

The value of IEs is increasingly recognized by governments and donors. The Organization for Economic Co-operation and Development has noted the growing importance of IE, with 58 percent of development agencies reporting that they undertake IEs.1 The World Bank is the largest producer of IEs among development institutions, with the number of IEs rising from less than 20

1 OECD (2010), “Better Aid: Evaluation in Development Agencies.”2 Independent Evaluation Group (2012), “World Bank Group Impact

Evaluations: Relevance and Effectiveness.”3 Center for Global Development (2006), “When Will We Ever Learn?—

Improving Lives through Impact Evaluation” http://www.cgdev.org/content/publications/detail/7973.

4 D. Rider Smith (2013), “Patterns and Influences in the Supply and Demand of Evaluation and Monitoring in Uganda’s Public Sector over the Past Two Decades.” forthcoming Evaluation Matters Bulletin. AFDB.

5 Regional Centre for Learning on Evaluation and Results for Anglophone Africa (CLEAR-AA) (2013), “Study on the Demand and Supply of Evaluation in Rwanda.”

6 CLEAR-AA (2013), “Study on the Demand and Supply of Evaluation in Ghana.”

in 2003 to 193 in 2014. The Bank’s institutional strategies have been significantly influenced by evaluation evidence in education and social protection.2 Some countries, such as Mexico, have made impact evaluation mandatory for a variety of social programs,3 and demand for reliable evaluation evidence is increasing across several developing countries, for example, in Uganda,4 Rwanda,5 and Ghana.6


– 5 –

ii. SnaPSHot of tHE i2i firSt-yEar DElivErablES

The i2i deliverables during the first year of implementation are above target. The i2i results framework in section VI, Portfolio Overview and Results Framework (see table 7), shows the detailed targets for all i2i progress indicators as agreed with the donor, and the rest of this report explains the background and implementation process for each of these deliverables. This section and table 1 provide a snapshot of the main i2i deliverables for IE products, workshops, and people attending and receiving training during capacity-building and dissemination events.

The i2i program has committed to supporting 125 IEs (65 new and 60 ongoing) during the i2i program implementation period of March 2014 to March 2019. For new IEs, the i2i target as per agreement with the donor was to start 5 IEs in the first year, 30 IEs in the second year, and an additional 30 in the third year. As table 1 shows, i2i has met and over-delivered its targets for the first year. As a result, the portfolio of new IEs has

TAbLE 1: i2i Deliverables*

DeliverablesFirst Year

TargetFirst Year

ActualPercentage of Target Completed

Ongoing Impact Evaluations (completed) 0 3 - +

New Impact Evaluations (started) 5 40 800% +

Workshops 4 4 100% ✓

Number of teams attended 36 94 261% +

Number of people trained 216 439 203% +

Number of governments, international organizations, universities, NGOs, etc., attended

12 182 1,517% +

Dissemination Events 15 16 100% ✓

Number of participants 250 543 217% +

Note: + = over-delivered on target; ✓= met target.*For a full results framework, see page 34

the following distribution across the i2i pillars and cross-cutting themes: fragility, conflict, and violence (14 IEs); agriculture (13 IEs); and energy and environment (13 IEs) (figure1). With forthcoming funding windows, i2i will support an additional 25 new IEs by September 2015 in the areas of governance; trade and competitiveness; finance and markets; and transport and ICT.

For ongoing IEs, the i2i target as per agreement with the donor was to complete 0 IEs in the first year, 9 IEs in the second year, and 17 IEs in each of the subsequent three years. As table 1 shows, i2i has completed 3 ongoing IEs in year one, over-delivering on its target. The portfolio of ongoing IEs consists of 44 IE products with the following distribution across the i2i pillars and cross-cutting themes: climate change (7 IEs); shared prosperity (19 IEs); governance (11 IEs); gender (5 IEs); and fragility, conflict, and violence (2 IEs) (figure 2). Across the entire portfolio of 84 IEs (new and ongoing), 16 IEs include a gender-specific


– 6 –

intervention and an additional 36 IEs plan on conducting disaggregated gender analysis.

i2i has internal regional targets for new IEs as per agreement with the Bank’s Regions. During the first year of implementation, i2i has made great progress toward meeting these targets. As explained in more detailed later in this report, i2i adopts a rule-based volume allocation to

regional departments that weighs equally the size of their operational portfolio and their ongoing IE portfolio. Table 2 below shows regional allocations based on this rule and progress toward those targets during the first year of i2i implementation. One out of the six Regions, Latin America and the Caribbean, has already met its regional target. Three Regions (Africa, South Asia, and Middle

FIguRE 1: New IEs by i2i Pillar

Energy and Environment, 13

Fragility, Conflictand Violence, 14

For new IEs, gender is a cross-cutting theme. Four out of 40 new IEs plan to include a

gender-specific intervention.

Agriculture, 13

FIguRE 2: Ongoing IEs by i2i Pillar

Governance11

Gender5

FCV2

Climate Change

7

The 5 gender IEs are pillar-specific for gender.An additional 10 IEs have gender interventions.

Shared Prosperity19

TAblE 2: i2i Regional Targets for New IEs

bank Region

Target for March 2019 Actual as of March 31, 2015

Target (# of IEs)

Proportion of overall portfolio

Actual (# of IEs)

Progress toward regional target

Africa 30 46% 22 73%

East Asia & Pacific 9 13% 1 11%

Latin America & the Caribbean

9 13% 9 100%

South Asia 8 12% 5 63%

Europe & Central Asia 6 11% 1 17%

Middle East & North Africa 3 5% 2 67%

Total 65 100% 40


– 7 –

East and North Africa) have made great progress toward their regional targets. All Regions are expected to meet their targets as new IEs are selected from the forthcoming funding windows between now and September 2015. For ongoing IEs, i2i does not have regional targets or allocations.

In addition to supporting IE products, the i2i program has committed to delivering four IE workshops per year. As table 3 shows, i2i has completed four workshops during the first year of i2i implementation. These workshops are organized by i2i pillars/themes. The rest of the report explains the objectives of the workshops and the process of selecting IE teams for participation as well as the collaborations that take place between i2i managing unit, the World Bank’s Regions and global practices, the donor, and the community of international experts in setting research agendas for each of these thematic programs and workshops.

The four completed i2i workshops received high rates of participation from the World Bank, multilateral development banks, country governments, and universities and nongovernmental organizations (NGOs). As table 1 shows, the target for the number of people and of organizations attending have been overdelivered. The total number of participants across all four workshops in year one was 439, including representation from international development banks, universities, country government staff, and other donor and NGO partners. Table 4 shows the participation of IE teams by Region for each of the four workshops.

As figures 3 and 4 show, of the total 439 participants in the four workshops, 229

TAblE 3: i2i Workshops

Programs and Workshops Location, Date

Fragility, Conflict, and Violence Lisbon, March 2014

Agriculture Kigali, June 2014

Energy and Environment Lisbon, October 2014

Governance Istanbul, January 2015

Trade and Competitiveness, Finance and Markets Istanbul, May 2015

Transport and ICT Rio de Janeiro, June 2015

TAblE 4: i2i IE Team Workshop Participation

bank Region FCV Agriculture Energy &

Environment Governance Total %Sufficient

Participation

Africa 9 16 5 11 41 47% ✓

East Asia & Pacific 1 1 1 0 3 3% -

Latin America & the Caribbean

8 2 6 7 23 26% +

South Asia 0 4 5 5 14 16% ✓

Europe & Central Asia 0 0 0 3 3 3% -

Middle East & North Africa

0 0 1 2 3 3% -

Total 18 23 18 28 87 100%

Note: FCV = fragility, conflict, and violence; + = participation on target; - = participation under target; + = participation over target.


– 8 –

(around 50 percent) were country delegates that represented different government ministries and agencies. Around 48 percent of the participants were international IE and subject matter experts from different universities, research centers, think

tanks, international organizations and NGOs. Appendix D provides a list of all different agencies represented in the workshops, such as donors, multilateral development banks, government agencies, academic institutions, NGOs, and so on.

FIguRE 3: i2i Workshop Participation by Region (total number of people attended)

Africa

Latin America & Caribbean

South Asia


East Asia & Paci�c

Europe & Central Asia

48%

27%

17%

4%

2%

2%

FIguRE 4: i2i Workshop Participation by Category (total number of people attended)

Country Delegations

IOs/NOGs/Think Tanks

Universities

50%

32%

16%


– 9 –

iii. manaGEmEnt anD GovErnancE

i2i has a six-tier governance structure, comprising the Steering Group, council, regional focal points and global practice (GP) working groups, technical committee, technical reviewers, and managing unit. Table 5 defines the membership and responsibilities of each of these governing bodies.

The i2i Council is chaired by the Bank’s Director of Research (or delegated i2i Manager) and composed of representatives from relevant World Bank vice presidential units (VPUs) (Chief Economists or Director of Operations and Strategy of the regional VPs and global practice VP). Each member is appointed by their VPU. The council meets on a yearly basis to oversee i2i activities, set annual priorities, and review overall program progress.

The i2i Steering Committee is composed of i2i donors, partners, and internationally recognized IE experts. The Steering Group meets annually to provide broad upstream guidance on the strategic direction of the i2i work program. Prior to the annual meetings, donors are provided with World Bank financial and program activity reports in accordance with Bank policies and procedures governing management of trust funds.

The i2i Regional and Global Practice Focal Points are representatives from the World Bank

TAblE 5: i2i Governance

body Membership Responsibilities

Steering Group Donors, partners, and internationally recognized experts

Meet annually to provide broad upstream guidance on the strategic direction of the i2i program.

Council Representatives from relevant World Bank VPUs

Meet annually to review and advise on the overall i2i work program and progress.

Regional Focal Points and Global Practice Working Groups

Representative from World Bank Regions/global practices (economists with expert-level knowledge of and experience in IE methods and implementation)

Coordinate regional/GP IE activities on an ongoing basis, including liaising with project teams and supporting the process for ranking policy relevance.

Technical Committee

IE technical experts from the World Bank’s research department and thematic areas

Provide technical expertise, guidance, and advice on technical quality and endorse the selected proposals to ensure quality and technical rigor of i2i IEs.

Technical Reviewers External IE experts Review i2i proposals, Concept Notes, and final papers for technical quality and submit technical ratings.

Managing Unit DECIE staff Manage and deliver the i2i program.

The Steering Group meets annually to provide broad

upstream guidance on the strategic direction of the i2i

work program.]


– 10 –

Regions and global practices. Each Region and global practice appoints a focal point at the level of economist or senior economist with expert-level knowledge of and experience in IE methods and implementation. Focal points coordinate the IE activities on an ongoing basis, including liaising with country and sector teams and supporting the process for ranking proposals for policy relevance. In addition, each global practice has established a working group composed of managers, operational staff and research staff to support each of the i2i thematic programs and IE portfolio.

The i2i Technical Committee is an expert group that ensures the quality and technical rigor of i2i-funded IEs. The committee is appointed by the Bank’s DEC Research Management and consists of IE technical experts. The Technical

Committee provides technical expertise, guidance, and advice on technical quality and endorses the selected proposals to ensure technical rigor.

The i2i Technical Reviewers are externally contracted IE technical experts that review and rate for technical quality all proposals submitted to i2i. They review expressions of interest for preparation grants, full proposals for implementation grants, and final IE outputs (IE completion reports or technical papers).

The i2i Managing Unit is composed of DIME staff. It administers and delivers the i2i work program and monitors and reports on the progress of i2i-supported activities and products. DIME staff provides research support and delivers the IE products. See below figure 5 for the i2i governance structure.

FIguRE 5: i2i Governance

Steering Committee

Council

Technical Committee

Management Unit

TechnicalReviewers

Regional FPs &GP Working

Groups

• Provides broad upstream guidance on the i2i’s strategic direction

• Reviews and advises on the overall i2i program and progress

• Provides technical expertise on technical quality and endorses the i2i selected proposals

• Reviews i2i proposals for technical quality and submits technical ratings

• Coordinates regional/GPIE activities

• Manages and delivers the i2i work program


– 11 –

iv. ProGram comPonEntS anD oPEratinG moDEl

The i2i program supports IEs across the developing world to run experiments that inform decisions, build agencies’ capacity to do so systematically, and share resulting evidence and lessons on a global scale. It does this through (i) supporting new and ongoing IEs for more effective policy; (ii) building local capacity for systematic use of IEs for policy making (matching policy makers with researchers from the outset); (iii) building project specific research teams to implement IEs in collaboration with government agencies; (iv) emphasizing technical quality control and policy relevance; and (v) ensuring a project-specific aspect to global dissemination and policy outreach.

IE Products for More Effective Policyi2i provides financial and technical support for new and ongoing IEs with analytically valid identification strategies to measure the causal link between interventions and outcomes. The role of these IEs is to (i) inform policy design based on IE evidence and baseline analysis; (ii) guide midcourse corrections by testing behavioral and implementation mechanisms; (iii) inform scale-up decisions by assessing the economic significance of policy effectiveness; and (iv) generate knowledge more broadly across communities of practice. IEs are executed in close collaboration with implementing agencies to ensure client-centered learning aimed at building capacity for IEs and

i2i provides financial and technical support

for new and ongoing IEs projects.]


– 12 –

for utilizing the resulting evidence in policy making and project management. This approach is reflected in IE team compositions (including both IE experts and clients) and client agreement on learning priorities.

All i2i-supported IEs are Bank-executed. This means that Bank researchers and IE staff manage the IE products as Task Team Leaders (TTLs) based on Bank’s policies governing these research products. Activities eligible for evaluation include lending and non-lending products as well as Bank- and non-Bank–financed activities.

The i2i support for IEs is delivered through four grant windows:

♦ Preparation grants. i2i preparation grants in the amount of $25,000 per IE are aimed to support preparation of a full IE technical proposal (Concept Note). That includes a fully developed methodological note with clearly defined research questions, a valid identification strategy, a data collection plan, and an IE work plan. The grant is meant to finance research time and travel to work with clients in preparing a technically valid and

policy-relevant Concept Note. An i2i Concept Note and budget template for IE proposals is provided to all i2i-supported IEs (see appendixes A and B for these templates). To be eligible, the IE must be at a pre-Concept Note stage. Normally, identification of potential proposals is done through i2i workshops that are conducted prior to opening the Call for Proposals (see the subsequent two sections on workshops). ♦ Implementation grants. i2i implementation grants in the amount of up to $50,000 per IE per year (renewed annually for up to three years) for ongoing and new IEs are aimed to support implementation activities. The grants may finance research services and

field coordination. To be eligible, proposals must demonstrate co-financing of $10,000 at minimum per IE per year from the World Bank’s global practices or regional departments and must demonstrate financing for data collection costs. For ongoing IEs, a special funding window is opened to receive applications from IEs that are in implementation (passed the Concept Note stage). For new IEs, the process is two-staged, with the IE first applying through a funding window for a preparation grant. If the IE is eligible and successfully passes the Concept Note stage, it automatically qualifies for an implementation grant.

♦ Dissemination grants. i2i dissemination grants in the amount of $15,000–$20,000 per IE are aimed at IEs that are in the post-analysis stage and need support for dissemination activities. The grant may finance data archiving, research paper finalization, and dissemination activities (such as presenting in a conference or traveling to meet with and discuss the results with


– 13 –

the client government counterparts). A key eligibility criterion includes having completed an IE final output review in the World Bank (that is, either a review of a final IE report or of an IE technical paper).

♦ Data collection grants. i2i data collection grants (amounts determined based on need) are aimed for i2i-supported IEs to supplement data collection costs in cases of urgent budget shortfalls. The grant may finance data collection based on the existing IE design as per Concept Note review. A key eligibility criterion includes having completed an IE Concept Note in the World Bank (that is, having developed a full methodological proposal and conduced a peer review process).

Programmatic Approach for IE KnowledgeThe i2i program covers 12 operational sectors of the Bank across four pillars and two cross-cutting themes. The pillars are shared prosperity, governance, climate change, and human development. The cross-cutting themes are gender, and fragility, conflict, and violence. Each pillar and program is led by a DIME economist and supported by a team of field-based IE coordinators, research assistants, and operational and administrative support staff. The unit’s manager provides oversight for all i2i activities. At the programmatic level, i2i sets the overall research agenda, manages research efforts in specific subthemes, and carries out dissemination, policy outreach, and fund-raising activities.

The governance structure for i2i, as explained in the previous section, is at both strategic and operational levels. The broad research agenda is guided by a high-level council composed of chief economists and directors of operations from relevant Bank VPUs. The council meets annually


– 14 –

to review progress. Work under each sector program is guided by working groups consisting of membership from i2i and the relevant global practices and cross-cutting solutions areas within the World Bank Group. Based on knowledge gaps and current operational and policy priorities, the working groups set learning priorities and conduct project case selection. The selected projects then work with i2i researchers to operationalize lessons and introduce experimentation in project design. A technical committee is in charge of IE selection based on double external technical reviews and policy ranking from the regional department of the Bank.

The i2i program delivers on this agenda through high-quality IE research carried out in full collaboration with government and operational counterparts and subject to internal and external technical quality and policy

relevance review agreed to at the concept stage, and synthesizing knowledge across countries and programs to provide the evidence needed to transform development policy. It fulfills both micro and macro knowledge needs. First, governments that collaborate with i2i on an IE benefit from evidence and analysis to improve their specific policies and programming on an ongoing basis throughout the implementation period (from initial concept through final dissemination). Second, i2i aggregates the knowledge produced across countries and sectors to create public knowledge goods.

IE Workshops for Capacity buildingIE workshops gather policy makers, World Bank operational staff, external researchers, and subject experts to advance knowledge on the research agenda and identify new IE products for financial and technical support. These workshops serve three main purposes. First, government counterparts and other partners receive instruction in IE methods and are given tools to allow active participation in the development and implementation of their IE and to become better-informed consumers of knowledge, whether generated through IE or other methods. Second, participants are exposed to the latest available evidence in each focus area, with a view to incorporating relevant evidence in their own program and policy design. Third, each country team is paired with one or more IE researchers to develop a prospective IE that will answer research questions of direct relevance to their specific program. A member of the delegation presents the prospective IE on the final day of the workshop, and it becomes the basis for an ongoing program of IE research and analytical support delivered by i2i to the country team.


– 15 –

Following the completion of a workshop, i2i opens a Call for Proposals for Expressions of Interests (EOIs) to help preselect viable candidate ideas. The Call for Proposals is disseminated to teams that attended the workshop as well as more widely through the Bank’s global practices and Regions. Interested teams are invited to submit a three-to-five-page EOI explaining their objective for an IE and a basic methodological plan to demonstrate technical capacity. All proposals undergo a technical review by i2i external technical experts and a policy review by the Bank’s Regions and global practices. The reviewers also flag the proposals for potential ethical issues. Teams that pass this stage are awarded a preparation grant of $25,000 to help prepare a full IE Concept Note. The preparation grant is meant to finance the research team’s time and travel to work with clients in preparing a technically valid and policy-relevant IE design.

Simultaneously, i2i follows up with the workshop participants to establish a research team and build sufficient capacity to submit a competitive EOI and subsequently support each of the successful IE studies through its completion. Each IE-specific research team includes a principal investigator and co-investigators, a field coordinator based in-country to support the government team on all aspects of the IE, and one or more research assistants. DIME researchers frequently serve as principal investigator or co-investigators, and manage the overall delivery of IE research and analytical services to the client. Field coordinators play a critical role in facilitating the government team’s ongoing participation at all stages of the IE process. This high level of engagement is critical in allowing i2i to (i) implement IEs that correspond

directly to country-specific priorities; (ii) go beyond efficacy trials to examine effectiveness in the context of large-scale government programs; and (iii) facilitate the translation of research findings—arrived at in collaboration with the government—into better informed policy making.

During the first year of its implementation, i2i trained 439 participants, of which 218 were government officials, and targeted around 94 IE teams. The workshops increased participant knowledge as measured by the different between pre- and post-knowledge tests (figure 6). The approach also secured better double-blind external technical ratings for participating teams than for teams that applied for funding independently (figure 7). The link to workshop agendas and presentations is available on the DIME website.

Project-specific research teams are responsible for producing an IE research design with a well-specified identification strategy to test hypotheses conforming to specific policy and programmatic priorities defined in conjunction with government and operational counterparts. Research teams assess data needs and oversee the collection of data to fill any gaps, and conduct interim and

FIguRE 6: Effects of Workshop on Participants’ Knowledge

WorkshopAgriculture Energy & Environment

Scor

e

7472

7273

65

61

706866646262585654

Pre Post

http://web.worldbank.org/WBSITE/EXTERNAL/EXTDEC/EXTDEVIMPEVAINI/0,,contentMDK:23415873~pagePK:64168445~piPK:64168309~theSitePK:3998212,00.html


– 16 –

final analyses. Research teams are the first point of dissemination to country project teams and are responsible for project-specific fund-raising. Throughout the IE implementation process, a series of outputs is produced, including an IE Concept Note/research protocol, data collection instruments and corresponding data sets, and interim and final analyses, data reports, policy briefs, and dissemination activities. Each IE culminates into one or more World Bank working papers, which are then submitted to peer-reviewed journals.

All i2i IEs are carried out in close collaboration with government counterparts, and agency staff are considered an extension of the research team. In some cases, government staff with particular interests or expertise serve as core members of the research team. The model is effective at securing strong policy feedback loops and affecting the way new policies are introduced and scaled up.7

After the teams are awarded i2i preparation grants and complete a Concept Note, they follow World Bank guidelines for Concept Note technical quality review. This review process is normally chaired by the relevant World Bank

country director and includes the participation of the government team. Three technical experts provide peer review comments (including two Bank experts and an i2i external expert). This process further ensures buy-in from the client, the technical quality of the study, and ongoing relevance to World Bank and country policy priorities. Intermediate and final outputs are discussed and disseminated directly to the government and other partners, and feedback is incorporated into final outputs, including in World Bank working

papers for submission to peer-reviewed journals. See figure 8 for the i2i implementation process.

From Project-Specific to global Dissemination and Policy OutreachProject- and country-specific dissemination happens throughout the implementation of i2i-supported IEs. Dissemination activities target four types of audience. First, government agency staff and operational partners directly involved in the implementation of IEs are consulted on an ongoing basis and receive results and analyses as they become available through regular communications, including face-to-face discussions with the research team. Second, broader country and programmatic audiences are reached through country- and program-specific dissemination activities, which include workshops and seminars in-country and at World

FIguRE 7: Effects of Workshop on Technical Rating of IE Proposals

2.252.04

2.182.01

1.76

2.08

FCS Agriculture Energy &Environment

Technical ratings of proposals (0–3 scale)

Attended workshop Did not attend workshop

7 World Bank (2012), 2012 DIME Progress Report, Washington, DC.


– 17 –

Bank headquarters. The broader global development community is reached through online dissemination, social media, workshops and seminars, and policy and technical briefs summarizing evidence from one or more IEs. Finally, the academic community is targeted through academic conferences, World Bank working papers, and peer-reviewed journal articles (for more detail, see section VIII, Knowledge Sharing and Dissemination). Crucially, all data used in i2i IEs are expected to be made available for public use through the World Bank’s Microdata Library (data is anonymized to protect the identities of study participants).

The translation of learnings into policy impact is an important goal of every i2i IE. To help ensure that ongoing and completed analyses are used to generate tomorrow’s policies and to further expand the research frontier, findings are discussed with i2i’s policy-maker networks as part of the periodic workshops carried out under each i2i program. Thus, the new generation of IEs in each programmatic area builds on ongoing and completed work to create virtuous cycles of learning and policy impact. The Zambia example

in figure 9 above shows how an effective IE links to policy and results. First, it identifies an important challenge working with local agencies and partners; second, it tests possible solutions; and third, it provides the analytical basis for scaling up the best solution and ensuring better results.

FIguRE 8: i2i Implementation Process

Global Practices DIME Regions

IE Workshop

Call for Proposals

Technical Review Policy Relevance Review

Technical Committee • Fundingdecisions

• Issued toselected IEs

• Completedby IE teams

• Issued to teamsthat pass CNreview

Preparation Grants

Concept Note Review

Implementation Grants

Annual Review

Final IE Output

All i2i IEs are carried out in close collaboration with government counterparts, and agency staff are considered an extension of the research team. ]

http://microdata.worldbank.org/index.php/home


– 18 –

FIguRE 9: Using IE for Scaling Up Development Impact

IE in Action in Zambia

LocalChangesIdentified

Possible SolutionsDeveloped

Testing withScientific Methods

Results Impact!

Stockouts ok keyhealth

commodities inpublic sector

health facilities

Alternative supplychain innovations

Experimental(random)

assignment ofsystem A and

system B acrossdistricts

System B almosteliminatesstockouts

16,600 deathsof children under 5

averted(projected)

Supply fromdistrict medicalstores to health

facilitiesidentified as

problem

System ACommodity

planner

System BCommodity

planner plus pre-packing at

central level

24 rural andperi-urban districtsrandomly assigned

to 3 groups:

1. System A2. System B3. Control

Both System A andSystem B

performed well.But system B

performed muchbetter.

System B scaled upnationally

2,200 adult deathsaverted.

Child/adultmalaria-related

mortality reducedby 21%/25%

Impact projectedfor 4-year period

with national scaleup of system B


– 19 –

v. rESEarcH PillarS

The i2i program covers operational sectors supported by the World Bank across four pillars and one cross-cutting theme (see table 6 below). The pillars are shared prosperity; governance; climate change; and fragility, conflict, and violence. The cross-cutting theme is gender. Based on current DFID funding, human development, such as education, health, and social protection, is not covered. However, as new donors join, the objective is to fund all sectors.

Some of the pillars/themes are well advanced in the definition of their learning priorities because they have gained from repeated interaction with researchers, operations and policy makers. Others are emerging areas that will be shaped by both internal and external consultations and will evolve as part of the way i2i works with clients to develop locally relevant, dynamic learning agendas.

The areas, subthemes, and broad indicative questions considered under each pillar are discussed hereafter. Appendix G provides a full list of i2i-supported IEs across these pillars.

A. Shared ProsperityThe number of extreme poor living in the world today has halved since 1990 and this trend is expected to continue. However, in developing and developed countries alike, inequality persists and is growing over time. To address the second of its twin goals, the World Bank has laid out

a strategy to boost the incomes of the poorest 40 percent in each country. This requires a coordinated and holistic approach to development. On one hand, poverty concentrations lie in rural areas where agriculture is the main source of subsistence and income. On the other hand, the private sector is the engine of economic growth. The program of work explores the productive sectors of agriculture, financial and private sector development, and infrastructure to understand the drivers of inclusive and sustainable growth.

Agriculture

DIME’s work on agriculture and rural development addresses the need to transition agriculture from subsistence to market-oriented

TAblE 6: i2i Themes and Pillars

Pillars Themes

Shared Prosperity

Finance, private sector, and jobsAgriculture and rural developmentInfrastructure

governance Public sector governanceJusticeLocal development

Climate Change

Energy EnvironmentAgriculture and natural resource management

Fragile, Conflict & Violence

Jobs and productive opportunities for youth at riskPublic sector governanceUrban crime and violenceGender-based violence

gender Jobs and productive opportunities for youth at riskPublic sector governanceUrban crime and violenceGender-based violence



– 20 –

activities—and to understand the strategies that can best address barriers to improve technologies of small- and medium-holder farmers.

One part of the research agenda looks at interventions to help address supply-side and demand-side constraints related to skill formation and information. We investigate nontraditional modalities of extension delivery—for instance, whether peer farming can lead to higher learning and adoption, in a context where traditional extension services are scarce; or whether private extension services can overcome the quality constraints faced in public extension service delivery. Within these modalities, we further investigate the role of incentives in securing peer-to-peer learning channels; the accountability mechanisms that can be used to secure higher quality of various types of extension services; the mechanisms that can successfully mitigate the risks associated with adoption of new technologies; and the types of communicators best suited to disseminate new technology to their community.

An important issue in adoption and sustainability of adoption is financial planning that can facilitate farmers’ access to optimal levels of

productive assets and inputs. Here, we investigate barriers to savings, type of savings products (for example, limited withdrawal property in time helps farmers increase self-control and input savings), types of credit for inputs and changing the timing of the product market, and the complementarities between saving and credit mechanisms in getting farmers to invest in productive assets.

Issues of access to productive inputs and markets consider (i) farmers’ organization in productive groups to help overcome constraints to more intensive farming systems, accessing markets, and reducing price-taking behavior; (ii) the role of information in securing better prices, decreasing a trader’s transaction costs and increasing competition; (iii) the role of technical assistance relative to financial subsidies in helping farmer groups increase their productivity and engage in high-value chains; and (iv) the impact of interventions to reduce land conflicts and increase investment in land, from community-driven land certification process to a top-down land titling process.

Financial and Private Sector Development

DIME’s work on financial and private sector development aims to address two overarching development challenges:

♦ How can we promote micro-, small, and medium enterprises (MSMEs) creation and growth?

♦ How do we grow sustainable and inclusive financial markets?

Private Sector

The majority of businesses in developing economies operate in the informal sector, which limits opportunities for growth. Heavy regulatory burdens have also been shown to hinder the


– 21 –

creation of new firms. However, once firms enter the formal economy, access to capital, markets, and skills remain as barriers to growth, and these barriers are most acute for female and minority business owners that lack networks, capital, and experience. Understanding where interventions can be most effective to support MSMEs can help improve competitiveness, strengthen private sector–led growth, absorb job seekers, and deepen the tax base.

DIME’s private sector program, in collaboration with the International Finance Corporation (IFC), is building evidence on the role of formalization and regulatory barriers; access to capital, training, and capacity building; and overcoming information asymmetries to strengthen networking and access to markets. There is a growing evidence base for microenterprises. Moving forward, the private sector program will have a particular focus on larger small and medium enterprises (SMEs).

Financial Sector

A strong financial sector can facilitate growth. Efficient capital distribution can open opportunities to households and firms. However, asymmetric information and behavioral biases in addition to lack of basic information and behavioral biases limit the extent to which the financial sector can play an enabling role in this development. The financial sector program has generated evidence on the role of financial education on improved financial behavior and aims to build evidence on alternative credit models for household and business financing moving forward.

Infrastructure

Public infrastructure is at the core of a developing country’s long-term growth strategy, requiring

large fiscal and technical resources, but also offering large potential public benefit. Yet the returns to infrastructure investment can be low due to poor construction quality, low utilization, and lack of sustainability. Given that the returns to infrastructure investments are not fixed, the broad question the program seeks to address is, What approaches are most effective in improving service delivery, optimal use, and maintenance of infrastructure programs to maximize social gains from both public and private infrastructure investments? Nearly 50 percent of rural water points in Africa are nonfunctional at any point in time. Despite extensive grid extensions, only 5 percent of Kenyan households living in grid-connected villages have electricity in their homes. DIME’s work on infrastructure will critically


– 22 –

depend on understanding the weak links in the logic of infrastructure investments and finding ways to address them. The DIME infrastructure subpillar covers four main areas: energy and mining, transport, water, and urban development.

In transport, for instance, the interest is

to understand the gains from investment in transportation, especially in contexts such as Ethiopia or Nigeria, where the cost of trading over a unit distance is four to five times larger than within the United States8. Returns to transport infrastructure should be high; however, the evidence is scarce. DIME is conducting two impact evaluations of World Bank projects in Nigeria and Rwanda, and collecting comprehensive surveys to understand the range of outcomes road projects can affect. The expansion of this agenda will include major transport corridors in East Africa, the Metro network in São Paulo, Brazil, and urban transport in China.

The returns (and sustainability) to transport investments depend on their cost, quality, and durability. For instance, in his seminal study in

Indonesia9 finds that top-down audits reduced “missing expenditures” in road construction by 8 percentage points, while bottom-up participatory monitoring did not. More important, thousands of contracts go out every year across the globe to build, rehabilitate, and maintain roads and we know almost nothing about how to improve procurement and contractual arrangements to decrease cost and increase quality. Closing the knowledge gap on what type of contracting, property rights distribution, governance arrangements, and payment systems would ensure the sustainability of our investments is important to the DIME agenda.

In addition to raw payoffs, we are interested in understanding how to increase returns on investment by investigating the role of information and subsidized access to the public transit system in increasing the probability of finding employment; how regulatory systems of rewards and punishment and car insurance market rules can improve safety and reduce public health costs; and how incentives and regulations can help curtail carbon emissions.

b. governanceIn developing countries, a third of gross domestic product (GDP) flows through government organizations. These investments provide essential public services, ranging from health and education to investment in infrastructure. Understanding the determinants of the productivity of these organizations is key to improving the capacity of the public sector and the effective delivery of public goods.

8 Atkin and Donaldson, 2015: https://dl.dropboxusercontent.com/u/2734209/Atkin_Donaldson_WGG_paper.pdf

9 Benjamin A. Olken, 2008. “Direct Democracy and Local Public Goods: Evidence from a Field Experiment in Indonesia,” NBER Working Papers 14123, National Bureau of Economic Research, Inc


– 23 –

Governance reforms are often long term, complex, and difficult to measure. Rigorous evidence on what works in the sector is, therefore, in short supply, with the governance field representing less than 3 percent of registered impact evaluations.10 DIME and the Governance Global Practice launched the ieGovern Initiative in June 2013 to help bridge this gap, leveraging its unique access to government partners to produce rigorous evidence that improves project results and pushes the frontier of available evidence of what works in governance reform. Following an initial portfolio of 10 IEs focused on the Africa Region, ieGovern will hold a global design workshop in Istanbul in January 2015 to regionally expand across the sectors of (i) civil service reform, (ii) justice, (iii) public financial management, and (iv) subnational public sector management/decentralization.

Civil Service Reform

A significant body of IE research exists on shaping performance incentives for frontline staff—such as teachers, nurses, and doctors—that address problems of absenteeism or underperformance.11 But with a few exceptions,12 no equivalent research has been conducted on how to attract, retain, and motivate those civil servants who work in government’s core ministries, such as the ministries of finance and education, and bear the responsibility for designing a county’s policies, collecting its taxes, and so on. In conjunction with the fragility, conflict, and violence (FCV) portfolio,

10 3ie Impact Evaluation Repository (http://www.3ieimpact.org/evidence/impact-evaluations/), accessed 11/11/2014.

11 Among many others, see, for example, Esther Duflo, Pascaline Dupas, and Michael Kremer (2012), “School Governance, Teacher Incentives, and Pupil-Teacher Ratios: Experimental Evidence from Kenyan Primary Schools,” Working Paper, MIT.

12 For example, Adnan Khan et al.’s ongoing work on tax inspectors’ incentives in Pakistan; see http://www.povertyactionlab.org/evaluation/property-tax-experiment-punjab-pakistan-testing-role-wages-incentives-andaudit-tax-inspe.


– 24 –

ieGovern is designing IEs on key policy questions, including civil servant training programs in South Sudan and meritocratic recruitment systems in Guinea.

Justice

In many countries, citizens lack access to equitable justice services because of (i) a lack of efficiency and quality in existing systems, and (ii) an overall lack of access to and demand for justice services, particularly among marginalized groups. ieGovern’s work in the justice sector focuses on both of these key questions, with ongoing evaluations exploring how to improve the speed and quality of case disposal in Kenya and how the demand for legal aid provision can be improved in Jordan.

Public Financial Management

Public financial management (PFM) reforms have been a core area of support in client countries by the World Bank and other donors for a long time. Yet very few impact evaluations exist on the effectiveness of different PFM systems. PFM will be a key focus of the upcoming expansion, with projects planned across the subsectors of (i) tax, (ii) procurement, and (iii) financial management information systems (FMIS). In Colombia, the ieGovern team is already working with the National Procurement Agency to help design an

evaluation of the reform of the procurement for food for the National School Feeding Program.

Subnational Public Sector Management/Decentralization

While the number of decentralization reforms implemented around the world keeps increasing, the evidence on their impact and on the mechanisms through which decentralization can work has stagnated. For example, how to measure performance of local governments? How to link this performance to resource devolution from the central government? ieGovern has ongoing evaluations exploring these questions in Burkina Faso and Tanzania and will have a significant focus on decentralization reforms in Latin America in the next generation of the program.

C. Climate ChangeThe latest climate science says that we are on a path to a 4°C (7.2°F) warmer world by the end of this century. This could have devastating effects on agriculture, water resources, ecosystems, and human health. It is therefore imperative for countries worldwide to take immediate action to slow the growth in greenhouse gas emissions. At the same time, there is an urgency to fight extreme poverty in the developing world, which often implies the provision of carbon-intensive services, such as electricity access, and the exploitation


– 25 –

of natural resources, including forests and water reservoirs. Roughly 1.6 billion people worldwide still have no access to electricity, and many subsistence farmers slash and burn rain forest for firewood or to make room for crops and grazing lands.

At the heart of achieving reductions in greenhouse gas emissions and eradicating poverty, therefore, lies the challenge of people overusing finite or carbon-intensive resources, delivering services inefficiently, and generally acting in ways that are neither privately nor socially optimal in the long run. The climate change program was launched in October 2014 to explore how policies and projects in the areas of energy and natural resources management can be designed to address these challenges and achieve lasting impact. Work in this emerging area under two themes, energy and natural resources management, is briefly highlighted below.

Energy

The knowledge agenda in energy focuses on understanding the importance of electrification and the reliability of electricity supply for households’ welfare and productivity in the industrial sector. It broadly aims to unpack the causal pathways from energy access to development. Moreover, it covers relevant operational questions such as the identification of means to attract private investments in rural electrification, ways to expand access to isolated households in a financially sustainable manner, determinants of willingness

to pay for reliable electricity access, forms of capacity training to increase electrification impacts, and viable tariff as well as payment models. The program will cover a wide range of energy technologies, from grid extensions to off-grid solutions such as green mini-grids and individual solar home systems, and countries from Latin America, Africa, and South Asia.

Natural Resources Management

In natural resources management, the impact evaluations address knowledge gaps on effective ways to fight deforestation and reverse the degradation of land and water resources. The knowledge agenda centers on examining payment for ecosystem services, capacity training in the use of green technologies, and the socioeconomic as well as environmental impacts of the adoption of technologies for sustainable land, water, and forestry management. The impact evaluations also address important questions regarding the optimal design of the interventions. Examples include the empirical determination of the subsidy level that maximizes pro-environmental behavior and forms of capacity training that best ensure proper use of green technologies.

D. Human DevelopmentDIME’s work in human development covers policies and programs in health, education, and social protection. Human development, however, is also a final objective of development policy


– 26 –

a study in Zambia that demonstrated the effectiveness of a supply chain that dramatically reduced stockouts in public health facilities and a study in India that highlighted the importance of local implementation capacity in the context of a community health worker–based malaria control intervention. Ongoing work includes IEs examining HIV/AIDS life skills training for youth in Kenya, changing norms and behaviors related to sexually transmitted diseases and gender-based violence through the MTV series Shuga in Nigeria, and improving midwife retention in underserved areas of Nigeria through monetary and nonmonetary incentives.

Education

DIME IEs in education cover a range of policies and interventions, including relaxing information constraints, increasing demand-side accountability, providing incentives to help households internalize the positive externalities of schooling, improving teacher capacity, and providing school inputs. Examples of completed IEs include a study in Sierra Leone that demonstrates how volatility in the flow of government-provided learning inputs can induce storage of these inputs by school administrators to smooth future consumption (explaining, in this case, why textbook provision to primary schools has no impact on student performance). An ongoing IE in Nigeria tests the impact of providing information to parents on their children’s performance at varying frequencies.

more broadly, and its centrality is reflected in five of the eight Millennium Development Goals (MDGs) referring specifically to targets in this area. Access to effective health care, education, and social protection are also fundamental to reaching the World Bank’s twin goals of ending extreme poverty and boosting shared prosperity. For example, poor health, high out-of-pocket expenditures for health care, and poverty are mutually reinforcing; quality education is crucial to access productive opportunities and raise incomes later in life; and social protection systems keep millions of people out of extreme poverty.

Compared to other sectors, human development has a longer tradition of high-quality impact evaluation; consequently, the body of knowledge on interventions such as cash transfers or behavior change communication strategies is relatively more developed. That said, there is much to be learned about the effective delivery of health, education, and social protection to the world’s poor, and the next generation of impact evaluations in these areas focuses on these unanswered questions and on improving the efficiency with which proven solutions are delivered to those persons that need them most. DIME’s work in each area is briefly described below.

Health

DIME’s health work focuses on malaria, HIV/AIDS, maternal and child health, and health systems strengthening. Completed IEs include

http://econ.worldbank.org/external/default/main?pagePK=64165259&theSitePK=469372&piPK=64165421&menuPK=64166093&entityID=000158349_20140619085232&cid=DEC_PolicyResearchEN_D_INT

http://econ.worldbank.org/external/default/main?pagePK=64165259&piPK=64165421&theSitePK=469382&menuPK=64166093&entityID=000158349_20140904101053


– 27 –

Social Protection

DIME has supported social protection impact evaluations in countries such as Uganda, where an evaluation of the Northern Uganda Social Action Fund found that unsupervised grants to groups of young adults were successful in increasing business assets, work hours, and earnings four years after the program launch; and Tanzania, where an IE of community-based conditional

cash transfers found positive health and education impacts two and half years after the start of the program. Results suggest that households focused on reducing risk and on improving their livelihoods rather than principally on increasing consumption.

E. genderGender equality for development is a core theme of DIME work and crosscuts all thematic areas. The gender knowledge agenda is to assess which interventions are most effective to reduce gender gaps in four key areas: (i) human capital, (ii) economic productivity, (iii) access to finance, and (iv) empowerment. DIME supports rigorous evaluations of policy actions that look to relax supply-side constraints (for instance, improving service delivery for clean water, sanitation, and maternal care) as well as market and institutional constraints (for instance, reducing systematic differences in earnings). The research agenda evolves with DIME’s portfolio of impact evaluation, fueling iterative learning. DIME operationalizes this vision by providing technical assistance to policy makers in identifying relevant gender issues, designing appropriate policy action, and testing their impact to motivate scale-up, scale-down, and new testing.

Human Capital gender gaps through the life CycleGender gaps in human capital are well documented, but there is little evidence on how best to close those gaps. The research agenda focuses on designing and testing interventions that address women’s access to and usage of health services and education, and that reduce women’s vulnerability to shocks that disrupt human capital acquisition. Results are then incorporated into ongoing country programs and strategy. In

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2268552

https://openknowledge.worldbank.org/handle/10986/17220


– 28 –

Malawi, the evaluation of a vocational training program found that family obligations limited young women’s attendance, resulting in poorer skill acquisition relative to young men. Nigeria is currently testing supply- and demand-side interventions to increase women’s access and usage of medical prenatal and birth services, and community-level interventions to increase uptake of malaria prevention technologies and accessibility of antimalarial drugs.

Economic Opportunities

Women’s access to economic opportunities is often undermined by their lower access to production inputs. Studies find that female farmers tend to have less access to information because agricultural extension networks are dominated by men. DIME IEs in Malawi and Mozambique showed that women can make effective extension partners: they are at least as good as men at encouraging adoption of improved technologies. Throughout Sub-Saharan Africa, women are disproportionately limited in their land ownership and transfer rights. A DIME IE in Benin showed that land demarcation increases soil fertility investment in female-

managed landholdings, shifts household decision making, and reduces spousal conflict.

Access to Finance

Whether in the case of women farmers or microentrepreneurs, access to productive assets constrains women’s economic opportunities. A DIME IE in Rwanda tested introduced targeted and pre-commitment savings accounts. Initial findings show that women are more likely to earmark their savings to buy durable goods than men, who invest in agricultural inputs, suggesting that intra-household bargaining over resources plays an important role in

women’s investment decisions. New DIME IEs will test whether insurance features increase women’s take-up and use of pre-commitment savings products.

Promoting Women’s Empowerment and Agency for Economic Development

A growing body of evidence shows that placing women in the center of the development agenda can increase efficiency in the management of institutions and resources, and that female leaders can have beneficial impacts on social norms. A new DIME research agenda will focus on using gender empowerment to combat domestic violence, testing interventions such as cash transfers and active labor market policies to economically empower women to emancipate themselves from abusive situations, and whether perpetrator-centered reintegration programs help lower the incidence of domestic abuse and recidivism. In India, a new DIME IE examines the role of community-based interventions in shifting social norms.


– 29 –

welfare improvement. Therefore, DIME and its partners launched the Impact Evaluation 4 Peace in March 2014 to work toward addressing this gap. This cross-institutional program focuses on four key challenges: (i) jobs for at-risk youth as a tool for resilience, (ii) public sector governance, (iii) sexual and gender-based violence, and (iv) urban crime and violence.

Jobs for At-Risk Youth as a Tool for Resilience

Youth unemployment can be an importance source of social unrest, and it can be particularly destabilizing in fragile contexts. The reverse is also true, however: employment opportunities not only increase incentives to maintain peace but also have the potential to rebuild social fabric in fractured communities through enabling inclusion and providing common economic objectives. Countries participating in DIME-led IEs in this area include the Democratic Republic

F. Fragility, Conflict, and Violence More than 1.5 billion people live in countries affected by violent conflict, and poverty rates are 20 percent higher in countries affected by repeated cycles of violence. By 2015, an estimated 52 percent of the world’s poor will live in areas characterized as fragile or conflict-affected. Developmental challenges in such settings transcend national boundaries through the displacement of populations, spread of disease, reduced trade, and spread of organized crime and terrorism. Furthermore, high levels of fragility and violence exist in countries normally considered relatively stable. Many countries in Latin America and the Caribbean, for example, suffer levels of violence comparable to those in the most conflict-ravaged states.

Despite the development imperative, we know very little about what works, and why, to foster development in settings characterized by fragility, conflict, and violence. This knowledge vacuum impedes our ability to design effective interventions to promote poverty reduction and


– 30 –

of Congo, Kosovo, and Liberia. In the Democratic Republic of Congo, DIME is working with the country’s Social Fund to experimentally evaluate whether short-term employment reduces youth participation in violence and armed activities.

Public Sector governance

An effective public sector is needed to move countries away from fragility, conflict, and violence, yet it is precisely in such countries that these institutions are more likely to be eroded and dysfunctional. To improve our understanding of which interventions and policies can improve public sector governance in such settings, DIME is leading work in Guinea, Liberia, and Sierra Leone, among other countries. In Guinea, DIME is working with the Ministry of Public Service to test the effects of different meritocratic recruitment processes and assignment on the quality of the civil service workforce.

Sexual and gender-based Violence

In many countries affected by ongoing or past conflict, rape and other forms of sexual violence leave permanent scars on affected individuals and communities and impede recovery and positive development in other areas. DIME is supporting an evaluation in India that is testing interventions to change attitudes and norms toward domestic violence in rural Tamil Nadu, the state with the highest incidence of domestic violence. Impact will be measured in terms of changes in attitude and norms toward violence within the home, and in the incidence of violence.

urban Crime and Violence

Countries affected by high levels of urban crime suffer from levels of violence comparable to those in conflict-ravaged states. With seven of the 10 most violent countries, Latin America and the Caribbean is the world’s most violent region. DIME-led work on urban crime and violence prevention is ongoing in countries such as Brazil, Honduras, and Mexico. In Mexico, DIME is working with the Ministry of the Interior to evaluate whether behavioral therapy, intensive mentoring, and jobs can reduce recidivism of youth incarcerated for serious crimes.


– 31 –

vi. Portfolio ovErviEW anD rESultS framEWorK

PortfolioBased on current donor funding, the i2i program has committed to supporting 125 IEs in total (65 new and 60 ongoing) during the period of April 2014 to December 2018. The support across these IEs is phased out by fiscal year (for results framework commitments and deliverables, see next section). As of March 31, 2015, i2i has supported 40 new and 44 ongoing IEs. With the future funding windows in the area of governance, trade and competitiveness, finance and markets, and transport and ICT, i2i will support an additional 25 IEs by September 2015. Currently, the i2i portfolio of IEs spans 42 countries and covers i2i’s four pillars and its cross-cutting theme.13 Based on the portfolio to date, the i2i IEs are concentrated on shared prosperity (35 percent), climate change (27 percent), fragility, conflict, and violence (19 percent), governance (13 percent), and gender (6 percent) (figure 10). The governance estimate exclusively includes ongoing IEs as its funding window for new IEs is currently in the process of selecting proposals. The gender estimate of 6 percent (5 IEs) refers to the IEs that have specifically targeted the gender pillar. In addition, out of the entire portfolio of 84 IEs across all i2i pillars, 16 IEs (19 percent) have reported to include a gender-specific intervention.

Of the 65 new IEs to be supported by i2i, 40 have already been supported in the areas of fragility, conflict, and violence (14 IEs); agriculture (13 IEs) as part of the shared prosperity and climate change pillars; and energy and environment (13 IEs) as part of the climate change pillar. An additional 25 new IEs will be funded. The funding window for governance is currently in the process of selecting proposals. Funding windows for trade and competitiveness and transport and ICT will open in June and July 2015, respectively (see figure 11).

Of the 60 ongoing IEs to be supported by i2i, 44 have been supported. The funding window for ongoing IEs was opened between June and August 2014, covering all i2i-supported pillars and its cross-cutting theme (see figure 12).

The i2i IE portfolio spans all six World Bank Regions, with a concentration in the Africa Region (56 percent). For new IEs, the i2i resource allocation adopts a rule-based model to the Bank’s regional departments that weighs equally the size of their operational portfolio and their ongoing IE portfolio. Based on this model, regional allocations include Africa (46 percent of the portfolio, or 30 IEs), East Asia and Pacific (13 percent, or 9 IEs), Europe and

13 Date is as of March 31, 2015.

The i2i IE portfolio spans all six World Bank Regions, with a concentration in the Africa Region (56 percent). ]



– 32 –

Central Asia (10 percent, or 6 IEs), Latin America and the Caribbean (13 percent, or 9 IEs), Middle East and North Africa (5 percent, or 3 IEs), and South Asia (12 percent, or 8 IEs). Figures 13, 14 and 15 show the regional distribution of new and ongoing IEs in the current portfolio; however, the distribution will change as an additional 25 new IEs are added to the portfolio by September 2015.

Results Framework

Table 7 presents the i2i results framework as planned with the donor, DFID, for the first year of i2i implementation, covering the period April 2014–March 2015. For a complete results framework that covers the period April 2014–December 2018, see appendix C. As shown in

FIguRE 10: Total IEs (Ongoing and New) by i2i Pillar

27%

6%

13%

35%

19%

Climate Change

Gender

Governance

Shared Prosperity

Fragility, Conflict and Violence

FIguRE 11: New IEs by i2i Pillar

40%

35%

0%

25%

Climate Change


Governance (selection of IEscurrently in progress)

Shared Prosperity(funding windows for T&C and

Transport & ICT forthcoming

FIguRE 12: Ongoing IEs by i2i Pillar

Climate Change


Gender

Governance

Shared Prosperity

16%

5%

11%

25%

43%


– 33 –

FIguRE 13: Total IEs (Ongoing and New) by Region

Africa

East Asia & Pacific


Latin America


South Asia

56%

7%

4%

18%

4%

12%

FIguRE 14: New IEs by Region

55%

3%

3%

23%

5%

13%

Africa

East Asia & Pacific


Latin America


South Asia

FIguRE 15: Ongoing IEs by Region

57%

5%

11%

14%

2%

11%

Africa

East Asia & Pacific


Latin America


South Asia


– 34 –

table 7, all the targets for the first year have been met and in some cases overdelivered. Empty cells under the “planned 2014/15” column suggest that no deliverable was expected for that indicator for the first year of i2i.

The results framework includes four policy indicators that aim to capture the influence of IEs receiving i2i funding on policy design and implementation and capacity building (these are indicators 1, 2, 3, and 8).14 For these indicators, the data were collected for all ongoing IEs that received implementation or dissemination grants through i2i as of March 2015. These data were collected through a survey of the 46 grant recipients, with a response rate of 94 percent.

Select Policy Indicators in Ongoing IEs Funded by i2iIt is worth noting that the i2i IEs included in this survey are at different stages of the life cycle, and some of the policy indicators reported are highly dependent on when in the cycle they are located. For instance, only IEs that already have results are

14 The total number of IE responses used for each indicator includes “Do not know” responses. For instance, in some cases, the results have not been discussed but a date has been set up to do so—IEs with such responses were included in the total number of IE responses as well; for example, for the indicator “Number of completed impact evaluations that generate evidence used to support project adoption, scale-up, scale-down, continuation, or cancelation decisions.” These tend to underestimate the indicators.

TAblE 7: i2i Results Framework

Outcomes IndicatorsPlanned 2014/15

Actual 2014/15

Improved awareness, demand for and use of evidence in decision making in key sectors

1. Number of impact evaluations that influence project design or implementationa

- 15

2. Number of completed impact evaluations that generate evidence used to support project adoption, scale-up, scale-down, continuation, or cancelation decisionsb

- 3

3. Number of multi-arm evaluation designs (out of number of impact evaluation started) to test project innovationsc

50% 74%

4. Proportion of project teams participating in the workshop that improve their knowledge of impact evaluation techniques

50% 79%

5. Proportion of project teams participating in an impact evaluation workshop that submit a proposal in the relevant i2i funding window

50% 73%

Improved effectiveness and impact of new and existing development interventions from DFID and wider community

6. Proportion of completed IEs that are cited in the project document or directly used to inform new development interventions

- -

— Continued


– 35 –

asked the questions related to whether the results were discussed with the client or whether the results motived scale-up/scale-down of a policy. We divided the IE life cycle into 3 phases and grouped the IEs accordingly: Phase 1 refers to the beginning of the IE, before implementation starts (20 IEs are in this phase); Phase 2 refers to the time between implementation of the intervention and before the IE results are available (17 IEs); and Phase 3 refers to the time when IE results are available (6 IEs). These phases are indicative and were selected based on important discussions or availability of data or potential for policy decisions

based on evidence available. However, we are currently refining the conceptual framework for these indicators.

(I) Capacity Building

The following set of indicators aims to capture the extent to which the IE assisted the client in building capacity for the policy or program, or for the client’s staff in general. We present four measures of engagement with the client to develop skills or feed evidence into policy, including (i) client participation in initiating IE workshops, which are multisector/multicountry IE workshops

TAblE 7: Continued


Actual 2014/15

A. Increased quantity, quality and relevance of evidence available to improve policy design and implementation in underserved areas

7. Number of previously ongoing impact evaluations completed and reported (e.g., as working papers or policy briefs published online)

- 6

8. Number of new data sets (baseline or follow-ups)d - 29

9. Number of impact evaluations started (new IEs) within the i2i portfolio across the following:

5 40

# of IEs

% of current portfolio

Gender (cross-cutting)e 4 10%

FCS 14 35%

Governancef -

Climate change 16 40%

Shared prosperityg 10 25%

Number of impact evaluations in:

Sub-Saharan Africa 22 55%

Middle East/North Africa 2 5%

Europe and Central Asia 1 3%

Latin America/Caribbean 9 23%

East Asia and Pacific 1 5%

South Asia 5 13%

Low-income and lower-middle-income countries 29 73%

Upper-middle-income countries 11 27%

— Continued


– 36 –

TAblE 7: Continued


Actual 2014/15

b. Increased capacity of policy makers, implementers, researchers, and partners to incorporate rigorous evidence/tools into policy making

10. Number of DIME workshops 4 4

11. Number of teams participating in workshops 36 94

12. Proportion of teams participating in workshops that come from low- or lower-middle-income countries

75% 73%

13. Number of people trained (World Bank, donors, government, other)

216 439

14. Knowledge reviews for policy making, including mechanisms, sector-to-sector and country-to-country adaptation lessons

- 1

C. Increased communities of practice and knowledge platforms for sharing of evidence and its application for policy making

15. Number of organizations (donors, multilateral development banks, government agencies, academic institutions, NGOs, etc.) participating in i2i knowledge sharing events and belonging to i2i networks

12 182

16. Number of dissemination events (conferences, seminars)

15 16

17. Number of participants at dissemination events 250 543

a. Represents the number of IEs with this indicator. The total for this indicator is missing the response for approximately half the IEs due to a bug with the survey. It will be corrected in the next round of surveying.

b. Represents the number of IEs with this indicator. Only IEs that were in Phase 3 were considered for this indicator.c. New IEs from the FCV, agriculture, and energy and environment windows were considered.d. Twenty-one IEs have produced new data sets.e. Include gender-specific interventions.f. Governance funding window is currently open, accepting applications for new IEs.g. New IEs in trade and competitiveness and transport and ICT have not been selected yet, which is why the target has

not been reached yet.

for IE skills development, state-of-the-art evidence in the workshop topics, and learning-by-doing IE design; (ii) training provided for monitoring and data analysis, which includes whether the IE team delivered any skills training of local institutions and/or staff for general monitoring and other data analysis independently; (iii) discussion of baseline results with client; and (iv) discussion of IE results with client to understand their policy relevance and application. Figure 16 presents the results for the surveyed IEs, with indicators that range from 48 percent (training provided for monitoring and data analysis) to 70 percent (baseline results discussed with client).15

(II) Quality of Data

In this set of indicators, we aim to measure if the IE played a role in influencing the quality of the data collected and utilized. We include four indicators related to monitoring and evaluation (M&E) and the generation of data across the IE, which in many occasions results in the first comprehensive data on the topic in the country

15 The total number of IE responses used for each indicator includes “Do not know” responses. For instance, in some cases, the results have not been discussed but a date has been set up to do so—IEs with such responses were included in the total number of IE responses as well; for example, for the indicator “IE results discussed with client.” These tend to underestimate the indicators.


– 37 –

or region where the IE takes place: (i) improved counterpart M&E consisting of whether the IE data requirements led to improvements in M&E data collection and/or reporting activities of the counterpart(s); (ii) high-quality baseline survey, which measures if the IE provided a full high-quality baseline survey (including covariates with sufficient sample size and representative of policy affected population) available to policy makers and researchers creating or building on instruments for policy making before the IE; (iii) high-quality follow-up survey, which measures if the IE included a panel or comparable follow-up data; and (iv) availability of data as a public good, which measures if data has already been made publicly available through the Microdata Catalog within the open-data policy. Figure 17 presents the results, which range between 17 percent (availability of data as public good) to 74 percent (improved counterpart M&E).16

(III) Quality of Policy Decisions

This set of indicators aims at capturing whether the impact evaluation has influenced the policy decisions in one of several ways, including at the beginning through program design or at different stages while the IE produces evidence through

new data or IE results. We present the following four indicators: (i) rationalized policy design, which measures whether the IE improved design based on clear understanding of the underlying theory of change (causal links between the intervention components and the outcomes) and highlighted areas of uncertainty and critical assumptions; (ii) baseline informed policy design/implementation, which measures whether IE baseline data were used by governments and other stakeholders to stimulate policy dialogue and/or help identify problems and solutions;

FIguRE 16: Indicators for Capacity Building

67%

48%

70%

50%

Client participated in initiating IE workshop

Baseline results discussed with client1

IE results discussed with client2

Training provided for monitoringand data analysis1

1. Only IEs in Phase 2 or 3 were considered for this indicator. 2. Only IEs in Phase 3 were considered for this indicator.

FIguRE 17: Quality of Data Indicators

Improved counterpart M&E

High-quality baseline survey

High-quality follow-up survey

Availability of data as a public good1

74%

44%

14%

17%

1. Only IEs in Phase 3 were considered for this indicator.

16 The total number of IE responses used for each indicator includes “Do not know” responses—for example, for the indicator “Improved counterpart M&E.”


– 38 –

(iii) government or other stakeholders adopted causal mechanism(s) based on IE results, which measures whether IE evidence from experimental testing of alternative mechanisms was used by governments or other stakeholders to determine the most effective program alternatives or to inform policy decisions; and (iv) IE results were used to motivate scale-up/scale-down of policy at national level, which measures whether the

FIguRE 18: Quality of Policy Decision Indicators

65%

39%

33%

50%

Rationalized policy design1

IE results were used to motivatescale-up/scale-down of policy

at national level3

Government or other stakeholdersadopted causal mechanism(s)

based on IE results3

Baseline informed policy design/implementation2

1. The total for this indicator is missing the response for approximately half the IEs due to a bug with the survey. It will be corrected in the next round.

2. Only IEs in Phase 2 or 3 were considered for this indicator.3. Only IEs in Phase 3 were considered for this indicator.

IE results reported successful (or insufficient) impact of the intervention in achieving desired outcomes and were used by governments and/or other agencies/stakeholders to motivate scale-up (scale-down) of policy at the national level. Figure 18 presents the results, which vary between 33 percent (adoption of causal mechanisms based on IE results) to 65 percent (rationalized policy design).17

17 The total number of IE responses used for each indicator includes “Do not know” responses. For instance, in some cases, the results have not been discussed but a date has been set up to do so—IEs with such responses were included in the total number of IE responses as well; for example, for the indicator “IE results were used to motivate scale-up/scale-down of policy at national level.” These tend to underestimate the indicators.


– 39 –

vii. monitorinG & Quality aSSurancE

The World Bank has strict corporate-level policies for quality assurance of IE products. All Bank-managed IEs undergo the following four main milestones of conducting an IE: (i) developing a Concept Note (methodological note) about the IE’s objectives: the evaluation questions and identification strategy, target population, the risks involved, and the expected dissemination strategy; (ii) conducting all stages of the IE, together with the client or other development partners; (iii) sharing the findings with the client and conducting broader outreach to the wider development community; and (iv) reporting on how well the objectives were achieved. The quality assurance mechanisms are mainly concentrated at the Concept Note development phase and final output completion stage.

i2i adopts the Bank’s internal quality assurance processes for

IE Concept Note and IE final output reviews and decision

meetings.]

At the Concept Note phase, IE managing teams are required to conduct a technical review and decision meeting. This entails a review from at least two peer reviewers, normally a subject matter expert and an operations expert. In addition, the Concept Note is circulated internally in the Bank for comments in the relevant global practice and Region. A decision meeting, usually chaired by the practice manager or country director, is held to discuss the review comments and necessary adjustments to the Concept Note. After the decision meeting is completed and the chair concurs the Concept Note, the IE implementation activities can commence. This process ensures IE’s technical quality, buy-in from the client, and ongoing relevance to World Bank and country policy priorities.

impact Evaluation to Development impact

– 39 –


– 40 –

After the IE implementation is completed, a similar decision meeting is held to review and discuss the final IE completion report. This meeting is also chaired by the practice manager or country director and involves technical review from at least two peer reviewers and an invitation for comments internally in the Bank to the relevant global practice and Region. After the decision meeting is completed and the chair concurs the final deliverable, the IE teams can engage in dissemination activities.

i2i adopts the Bank’s internal quality assurance processes for IE Concept Note and IE final output reviews and decision meetings. In addition, it complements this process with additional layers of quality assurance for the IEs it supports, both at the Concept Note and final output review stages as well as during implementation. This is discussed hereafter.

Quality Assurance during SelectionAs explained earlier, all Bank IEs are subject to internal quality assurance processes—that is, they require managerial approval, are peer reviewed and discussed prior to implementation, and are linked to the Bank’s and country priorities. The i2i program adds additional layers of quality assurance for the IEs it supports.

First, for new IEs, i2i provides preparation grants to support IE teams to develop a Concept Note. Once the Concept Note is completed,

the Bank IE team follows the Bank’s standard review process for IE products. However, to ensure high quality of selected proposals for preparation grants, i2i opens Calls for Proposals for EOIs, which undergo a two-staged review process:

♦ Technical review by two external IE and subject matter experts. The reviewers are i2i-contracted experts with extensive IE experience. They score each of the EOIs based on the following criteria:

§ Clarity of the IE research questions and their potential to contribute to filling the identified evidence gaps

§ The extent to which prioritized project components/interventions for the IE have logical pathways to intermediary and ultimate outcomes

§ The extent to which a credible identification strategy is proposed for each research question

§ Whether the proposed IE has high potential for learning—for example, through including multiple treatment arms



– 41 –

§ Whether the targeting and recruitment of participants is clearly defined, and the number of participants appears adequate to implement proposed analyses

§ Whether the IE is feasible to implement on the ground (considering sample size, feasibility of intervention, selection of beneficiaries, and country context)

♦ Policy relevance review by an internal committee of IE and subject matter experts from the Bank’s global practices and Regions. They score each of the EOIs based on the following criteria:

§ Evidence of partner engagement and feasibility of implementation; the extent to which the proposed design is realistic and feasible given the project team and current client engagement, and is supported by the client

§ Whether the IE addresses a knowledge gap and proposes a natural extension to the current evidence available

§ Whether it has the potential to influence the design and/or prioritization of current and future development interventions

§ Whether it has the potential to influence policy design and/or scale-up

The i2i Technical Committee makes selection decisions for the EOIs based on technical and policy scores, overall feasibility of implementation, and capacity to target important knowledge gaps. After the IE teams receive i2i preparation grants, they are granted up to six months to develop a full Concept Note in collaboration with the country clients and other partners involved. Once the Concept Note is completed, the team follows Bank process for quality assurance by conducting an internal “Concept Note review.” This review involves three peer reviewers, one of which is identified by the i2i program, and requires managerial approval.

Second, for ongoing IEs, i2i provides implementation grants to support IE implementation activities. To ensure high quality of selected proposals, i2i opens Calls for Proposals for IEs that are in the post–Concept Note stage and have completed internal Bank Concept Note reviews. With their applications, the teams are required to submit Concept Note documents, peer review feedback from the Bank’s review process, complete budget plans, and any other implementation mid-term or progress reviews. These material undergo a technical review by two i2i external IE and subject matter experts, who score each proposal on the following criteria:


– 42 –

§ Hypotheses/Evaluation Questions: the extent to which the hypotheses and research questions are clearly linked to the theory of change and have relevance to important research and/or policy questions

§ Main Outcomes of Interest: the extent to which they are relevant to answering research questions and are feasibly gathered

§ Evaluation Design and Sampling Strategy:

• Whether the identification strategy is well explained and defines a credible counterfactual, is feasible, and will give convincing results

• Whether the design presents any ethical issues and if so, mitigation measures are highlighted

• Sufficient detail on sample size/power calculations provided for each of the primary research questions, given available data

§ Data collection: whether the Concept Note includes basic details on data collection instruments; whether the data collection strategy is thought-out and feasible—includes information on ethical clearance, if applicable

§ IE management: whether the research team and implementing partners have sufficient capacity to carry out the proposed research and the proposed budget is realistic and represents research value-for-money

Monitoring and Quality Assurance during Implementationi2i adopts a World Bank monitoring framework to understand the potential channels through which it can affect capacity and influence policy decisions throughout the cycle of implementation. This framework is based on our understanding of the theory of change underlying these potential channels. It also forms the basis for understanding the results of the independent Client Feedback Instrument (CFI), which collects feedback from clients (see appendix H for results from 2013).

All i2i-supported IEs undergo an annual progress review by the i2i program through the Bank’s standard internal Trust Fund Grant Reporting and Monitoring (GRM) tool. A completed GRM, which provides qualitative reports on the progress and results of grant-financed activities, is required by all i2i-supported IEs on an annual basis. In addition to the GRM, i2i collects a survey from all i2i-supported IE teams that comprises a set of policy influence indicators for IEs that cover the different stages in the policy cycle, including design, capacity building, data collection, policy implementation, and evidence-based decision making (see appendix D for a full list of i2i monitoring indicators). Self-reported frequencies are noted here with the aim of providing suggestive evidence of the channels through which IE improves policy and operations.



– 43 –

viii. KnoWlEDGE GEnEration, caPturE, anD DiSSEmination

Knowledge management within i2i is aimed at encouraging new thinking

and debate, informing and convening stakeholders around topics of discussion, and raising awareness in best practices.

] ♦ Sharing of evidence during DIME workshops:

DIME workshops are targeted at communicating knowledge to policy makers in specific sectors (i2i pillars/themes). These workshops allow government counterparts and other partners to learn methodologies in impact evaluation, discover tools for policy design and implementation, and in general acquire knowledge in IE as relevant to a particular sector. The workshops also provide an occasion for the country implementation teams to develop IEs with exposure to the latest evidence and research teams.

Some sample formats for knowledge dissemination during workshops include ignite talks (these presentations intend to “ignite” the audience on a particular subject) and TED-style talks.

This section focuses on the various activities encompassing knowledge generation, capture and distribution, and all effective tools in expanding and extending the impact of development generated from i2i activities. Knowledge management within i2i is aimed at encouraging new thinking and debate, informing and convening stakeholders around topics of discussion, and raising awareness in best practices. Within i2i, this is done at a variety of levels, depending on the target audience and needs.

♦ Delivery of a single product to a direct client:During the project life cycle, a number of reports can be shared with a particular client, including baseline reports and IE reports. In effect these will lead to policy decisions (see also Quality of Policy Decisions, section VI). Baseline reports can be used by governments and other stakeholders to stimulate policy dialogue and/or help identify problems and solutions. IE reports can lead to adoption of casual mechanisms based on results. These results can also influence the scale-up or scale-down of a policy at the national level.


– 43 –

https://www.youtube.com/watch?v=Q7Ef3ZBXJ_0

https://www.youtube.com/watch?v=Q7Ef3ZBXJ_0

https://www.youtube.com/watch?v=lUa684juhp0

https://www.youtube.com/watch?v=lUa684juhp0


– 44 –

♦ Delivery of single products to a wider audience:This level focuses on knowledge dissemination to the broader global development audience and in-country dissemination as well as industry audiences, including the academic community.

An important facet of the research and knowledge produced within DIME is the wider policy outreach. This link between researchers and policy makers (and other relevant stakeholders), feeding results back into policies, fosters the systematic use of evidence informing policy adoption, midcourse corrections, and scale-up policies. Policy outreach is achieved through (i) policy briefs, (ii) World Bank working papers, and (iii) journal submissions. This proposal considers the processes, budgets, and resources required to implement and standardize the production of policy outreach products for the DIME team.

Examples of output include:

(i) Working papers:Policy research working papers encourage the exchange of ideas on development. It is a way to quickly disseminate the findings of research that may have been completed or is in progress. These working papers aim to demonstrate results with wide applicability across sectors and countries. The series, which began in March 1988, has published over 350 papers each year, with 30 to 35 focusing on impact evaluations.

(ii) Briefs:Policy briefs provide an opportunity for researchers to provide advice or solutions stemming from research. A policy brief can be produced once baseline data is available. A brief presents information on instruments that have not been previously tried or tested and informs learnings available for the first time to governments. Policy briefs are usually short texts (one or two pages) that are in a user-friendly and easily digestible format.

Examples of i2i policy briefs:

WHAT IS IE?

WHAT IS IE?

i2iDIME

TRANSFORM DEVELOPMENT

Impact Evaluation is the study of causal relations between a program, policy or intervention and outcomes of interest. IE is frequently confused

with monitoring and evaluation.

Impact evaluation (IE) and monitoring and evaluation are different but complementary activities. Monitoring is the ongoing process of tracking project implementation, including inputs, activities, and outputs. Periodic evaluations of monitoring data provide assessments of whether a project is being implemented as planned. These can also be used to track changes in target outcomes over time, but they cannot tell us whether these changes are caused by the project.

IE goes one step further to directly attribute changes in specific outcomes to a particular project or intervention. The fundamental problem we face in trying to measure the causal impact of a program is being able to disentangle how much of the outcome change is a direct result of the program, and how much is due to external factors unrelated to the program. To be able to make causal statements, and separate intervention impact from the effect of external factors, IEs are based on counterfactual analysis. The counterfactual represents what would have happened in the absence of some specific action/intervention.

In practice, we cannot observe any specific unit of intervention with and without an intervention at the same time, and so we estimate the counterfactual using impact evaluation methods. The counterfactual, measures what would have happened to participants, had the intervention not taken place, and it is estimated using a comparison group that does not receive the intervention being evaluated. Finding a good

counterfactual, or comparison group, is the key to having a good IE. The ideal comparison will be between two groups that are on average indistinguishable, except that one group receives the intervention (treatment) and the other does not (comparison). An IE design focuses on finding the best possible comparison group to compare their outcomes over time, starting from before the rollout of the intervention. Although there are many ways to set up a comparison group, the randomized controlled trial, where units are randomly allocated to either treatment or control (comparison group), ensures an accurate and balanced comparison and represents the gold standard for evaluation.

HOW CAN IE IMPROVE POLICIES AND RESULTS?

Impact evaluations can be designed to find answers to what works and why:

1. Is a program/project/policy having its intended effect (that is, what works)? These evaluations measure the size of a project/program impact.

2. What are the mechanisms that drive impact (that is, how/why does it work)? These evaluations help understand how to change specific elements of a program or policy to make it work better.

By testing different variants of the same project (for example, different forms of encouraging at-risk youth to participate in a program), IE can determine, with statistical precision, which of these variants is most effective in affecting key outcomes, and by how much.

IE is not an audit of a project. Instead, IE can be designed as part of project implementation to

i2iDIME



WHAT IS IE?

gENDER – MOzAMbIQuE

IDENTIFYING GENDER-SPECIFIC BARRIERS AND INTEGRATING GENDER INTO DEVELOPMENT OPERATIONS THROUGH IMPACT EVALUATION

Context

Innovation in agriculture is crucial to meet the food requirements of Africa’s growing population. Although agricultural intensification has typically been the standard approach to increase yields, there has also been a recent trend to emphasize sustainable land management (SLM) practices. SLM can offer higher yields, more efficient water use, increased resilience to climate variability, and enhanced soil fertility. However, SLM outreach tends to overlook women, despite their major role in the agricultural work force.

In many settings, married women cultivate plots separate from those of other family members. They face different challenges to productivity, such as deficiencies in inputs and weak property rights. They also have low access to extension services. Extension is the traditional vehicle of technological diffusion, yet it often caters to men. Social stigma can decrease information supplied to women and it also decreases womens’ demand for information. In societies with strict norms on gender roles and male-female social interactions, women may be more effective interlocutors to other women.

Intervention

In 2007, the government of Mozambique, with support from the World Bank, invested heavily in the agricultural extension network in five districts in the Zambezi Valley, an area with high agricultural potential. The government increased the number of extension agents per district and provided them with housing and training. In 2010 and 2012, agents received technical training on eight SLM practices (mulching, crop rotation, strip tillage, microcatchments, contour farming, row planting, improved fallowing, and intercropping).

The training lasted three days, split evenly between in-class lectures and field demonstrations. Male messengers, also known as contact farmers, served as points-of-contacts between extension agents and community members. This model assumes

Did you know?Women are major contributors to Africa’s agricultural workforce, yet they lack basic farm inputs, secure property rights, and access to information on innovative technologies.

Improving women’s access to these factors may enhance overall productivity and food security.


RESEARCHERSFlorence Kondylis, DECIE, World Bank

GroupValerie Mueller, Development Strategy and

Governance Division, International Food Policy Research Institute

Glenn Sheriff, National Center for Environmental Economics, U.S. Environmental Protection Agency

Siyao Zhu, DECIE, World Bank Group

i2iDIME


MINDING THE GENDER GAP


– 45 –

(iii) Peer-reviewed journal articles:In addition to policy briefs, researchers produce longer papers. These can either be submitted as a World Bank working paper or to a number of recognized journals. To date, there have been 170 IE working papers published in the World Bank series since 1996. Researchers can also submit information to the American Economic Association’s registry for randomized controlled trials (AEA RCT Registry) (before baseline or just after) and journal submissions can also be registered with Social Science Research Network (SSRN).

TANzANIA – HANDWASHINg

SANITATION AND HYGIENE CAMPAIGNS IN RURAL TANZANIA REVEAL THE CHALLENGES TO CHANGING BEHAVIOR AT SCALE

Context

Poor sanitation is associated with higher rates of fecal-borne illnesses (diarrhea, tropical enteropathy, and worm infections) and a general decline in human health and well-being. Children can be at a higher risk for these illnesses, leading to malnourishment, stunting, and death.

Handwashing has been proven to be an effective way to prevent the spread of illness and infection and promote sanitation. Extensive evidence, including three systematic reviews over the past 10 years covering more than 30 studies has shown that andwashing with Soap campaigns reduce diarrhea by 39–47 percent. However, until now the majority of the evidence has come from small-scale efficacy trials focusing on proof of concept. Little evidence exists on how effective campaigns can be at changing behavior and improving health results as part of a large-scale government program.

With support from the World Bank and the Water and Sanitation Program, the government of Tanzania implemented two campaigns to promote access to proper sanitation and encourage handwashing with soap (HWWS), as part of the Global Scaling Up Rural Sanitation and Handwashing Project. The campaigns embedded an impact evaluation into the program rollout to be able to measure the impact of each campaign individually and in combination to test explicitly for potential synergies between the interventions.

Intervention

The two campaigns targeted 150,000 rural households across 10 districts in Tanzania. The campaigns focused on changing behaviors without the use of subsidies or direct incentives.

Did you know? Globally, diarrhea is the second largest killer of children under the age of five.

In Tanzania, diarrhea afflicts as many as 14 percent of children in any two-week period.

i2iDIME



RESEARCHERSBertha Briceño (Inter American

Development Bank),Aidan Coville (DECIE, World Bank Group),Sebastian Martinez (Inter American

Development Bank),

TAKING PILOT EVALUATIONS TO SCALE

NIgERIA bRIEF

DIME POLICY RESEARCH WORKING PAPER

Impact Evaluation Helps Deliver Development ProjectsArianna Legovini, Vincenzo Di Maro & Caio Piza

Does research add value to aid? Specifically, does impact evaluation research help or hinder the delivery of development projects?

This paper analyzes the question by constructing a new data set of 100 impact evaluations and 1,135 projects approved by the World Bank between 2005 and 2011. The analysis finds that the delivery of projects with impact evaluation is significantly timelier: common delays are avoided and the gap between planned and actual disbursements is reduced by half. Evidence-based mid-course corrections, a clearer implementation road map, strengthened capacity on the ground, and observer effects are possible channels to explain results. This analysis will stimulate discussion over the optimal balance between project financing and the impact evaluation research needed to deliver development outcomes.

Please click here for full version.

COMPLETED IE

Story of Gold: Entertainment as a Nudge to Promote Responsible Saving and Borrowing

This IE assessed the impact of enhancing financial literacy through a feature film, Story of Gold. The movie aimed to teach low-income micro-enterprise owners with limited formal education core concepts around financial planning.

Key Findings:

• The movie, coupled with a micro-finance intervention, was able to influence savings decisions in the short run, but no long-run impact was found

• This suggests that, while short-run behavior can be changed through one-off events, long-run behavior is far less malleable

IE Contacts: Vincenzo Di Maro, Aidan Coville and Felipe Dunsch.

ONGOING IE IN THE HEALTH SECTOR

Malaria Control Booster Project: Enlisting Community Volunteers & Patent Medicine Vendors in the Fight against Malaria

The Bank-assisted MCBP introduced two interventions as a response to poor health coverage, in order to increase the reach of the public sector. In the first—the so-called Community-Directed Intervention (CDI) approach—volunteers were selected from the community and trained on malaria prevention and home treatment to serve their own relatives and neighbors. In the second, private sector Patent Medicine Vendors (PMVs) were trained on diagnostic procedures and the sale of subsidized high-quality antimalarial drugs with the right dosage.

The IE is studying the effectiveness of the CDI and PMV interventions, together and in isolation, in reducing malaria cases and improving diagnosis and treatment, through household behavior change and better access to advice and appropriate treatment. It is also studying the impacts on other important outcomes,

i2iDIME


DIME NIGERIA: IMPACT EVALUATION UPDATES

March 2015

i2iDIME



(iv) DIME seminars:DIME seminars are a biweekly seminar series. The goal is to vividly present recent IE research results and conceptual papers and to link them to practical policy-making, targeting a wider Bank audience. Examples of recent DIME seminars can be found here.

(v) Summary pieces and white papers:First piloted through the FCV program, these documents will establish an analytical framework to prioritize more systematically the main factors purported to drive a particular thematic area/pillar and clarify important underlying mechanisms. These documents will lay out key drivers that address an underlying topic or thematic area (for example in conflict). The white papers will also provide an overview/summary of how well World Bank interventions have addressed (or not addressed) these identified drivers; and define an agenda of emerging research areas and priority research questions that new IEs should help address.

One example of such a document is a recent paper investigating the impact of impact evaluation considering the role of IE on the quality of projects.

The first FCV white paper drafts and summary pieces will be finalized in September 2015. The final versions will be available in Winter 2015.

http://econ.worldbank.org/external/default/main?pagePK=64166018&piPK=64167664&menuPK=477165&theSitePK=469372&docTY=620265&colTitle=impact%2520evaluation%2520series

https://www.socialscienceregistry.org/


http://web.worldbank.org/WBSITE/EXTERNAL/EXTDEC/EXTDEVIMPEVAINI/0,,contentMDK:23415873~pagePK:64168445~piPK:64168309~theSitePK:3998212,00.html

http://t.co/qgf72ChFGz

http://t.co/qgf72ChFGz



– 47 –

APPENDIxES a. i2i iE concept note template 49

b. i2i iE concept note budget template 63

c. i2i results framework 67

D. i2i Workshop Participation 71

E. i2i monitoring indicators 77

f. iE case Studies 85

G. i2i Supported impact Evaluations 93

H. Policy influence of World bank impact Evaluations 105

i. client feedback on iE Products 109


– 49 –

18 Please refer to JEL classification codes, http://papers.ssrn.com/sol3/displayjel.cfm.

19 The concept note is aligned to Ethical clearance (E) and Registry (R) indicative requirements. These indicative requirements are referenced throughout the document.

Impact Evaluation (IE)

Concept Note Template[IE Title]

[Country]

[IE code]

[Date]

Keywords:18 choose one of more keywords/categories that describe your iE.19 (r)



– 51 –

TAblE OF CONTENTSIE Profile Indicators 53

Executive Summary 53

background and Key institutional features 53

literature review (E) 53

Policy relevance 53

theory of change (E) 53

Hypotheses/Evaluation Questions (E,r) 54

main outcomes of interest (E,r) 54

Evaluation Design and Sampling Strategy (E,r) 54

treatment and control Groups 55

Sample Size calculations 55

Data collection (E,r) 55

Quantitative instruments 55

management of Data Quality 55

Ethical issues 55

Qualitative instruments 55

iE implementation monitoring System (r) 55

Data Processing and analysis 55

Data coding, Entry, and Editing (E) 55

model Specification for Quantitative Data analysis 55

Study limitations and risks (E) 56

iE management (E,r) 56

Evaluation team and main counterparts 56

Work Plan and Deliverables 57

budget 57

Plan for using Data and Evidence from the Study 57

Appendix 58

i2i indicators of iE influence on Program/Policy 58

theory of change Example 60


– 52 –

iE ProfilE inDicatorSNo. Indicator Description1 IE Code IE code (hyperlink to IE portal)

2 IE Title Legal title of the IE

3 IE TTL As in IE portal

4 IE Contact Person Name and affiliation/unit

5 Region AFR/EAP/ECA/LCR/MEN/SAR

6 Sector Board/Global Practice Use standard abbreviations

7 WBG PID (if IE is evaluating a WBG operation)

PID (hyperlink to operations portal)

8 WBG Project Name (if IE is evaluating a WBG operation)

Legal project name

9 Project TTL (if IE is evaluating a WBG operation)

TTL in operations portal

10 Intervention <Policy/intervention to be evaluated> E.g., financial literacy in high schools

11 Main Outcomes <Most important outcome(s)> E.g., student financial knowledge, attitudes and behavior

12 IE Unit of Intervention/Randomization

Main level of treatment assignment/randomization if it applies (e.g., school)

13 Number of IE Units of Intervention E.g., 900 schools

14 IE Unit of Analysis Lowest level of analysis on which outcomes are measured (e.g., student)

15 Number of IE Units of Analysis E.g., 20,000 students

16 Number of Treatment Arms Number of treatment arms

17 IE Question 1 (Treatment Arm 1) What is the impact of <intervention> on <outcomes>?

18 Method IE Question 1 Main method to estimate IE Question in (17). E.g., “Random assignment at the school level”

19 Mechanism Tested in IE Question 1 If the treatment arm tests a particular mechanism, classify it accordingly, e.g., information, incentives, behavioral biases, constraints, accountability measures (for general discussion about mechanisms, see http://www.itg.be/internet/ds/tde/doc/Astbury%20%26%20Leeuw%20.pdf). If the treatment arm does not identify a particular mechanism, write “Package.”

20 IE Question 2 (Treatment Arm 2) What is the impact of <intervention variation 1> on <outcomes>? E.g., What is the impact of a parental workshop on financial literacy on student knowledge, attitudes, and behavior?

21 Method IE Question 2 Method IE Question in (20): e.g., “Random assignment at the parent level”

22 Mechanism Tested in IE Question 2 See (19)

23 IE Question 3 (Treatment Arm 3) See (20)

24 Method IE Question 3 See (20)

25 Mechanism Tested in IE Question 3 See (18)

25 Gender-specific Treatment (Yes, No) Yes if it is a gender-specific intervention

27 Gender Analysis (Yes, No) Yes if there is stratification/power for gender-specific analysis

28 IE Team & Affiliations Name 1 (Organization/Unit Affiliation, Role); Name 2 (Organization/Unit Affiliation, Role)

29 Estimated Budget (including research time)

Total in USD

30 CN Review Date Month-Year

31 Estimated Time Frame for IE Month-Year to Month-Year

32 Main Local Counterpart Institution(s) E.g., Ministry of Education

http://www.itg.be/internet/ds/tde/doc/Astbury%20%26%20Leeuw%20.pdf

http://www.itg.be/internet/ds/tde/doc/Astbury%20%26%20Leeuw%20.pdf


– 53 –

iE ProfilE inDicatorS

Executive Summary(1 page)

♦ Describe the proposed IE in non-technical language in one paragraph or less. This could be an abstract of your IE. Include broad motivation/background and policy/research contribution. (E,R)

♦ Present IE questions and main outcome(s) the intervention aims to affect.

♦ Briefly explain how you are proposing to test your main evaluation question(s).

background and Key Institutional Features(1 page)

♦ Present an overview of the local context. ♦ Identify and define the problem: What is the

policy/research problem this IE is proposing to study? Which groups are affected by the problem?

♦ Describe the intervention whether existing or new, implementing organization, institutional setting, and any important considerations.

♦ Describe the intervention geographic/demographic scale and scope: Does it represent the “mode” of delivery in the country? (R, E)

literature Review (E)

(1 page or less)

♦ Describe most relevant literature/scientific background specifically linked to your problem/evaluation question(s).

Policy Relevance(1/2 page or less)

♦ Assess the extent to which the study may influence policy and institutional capacity at the national, regional, and international levels. Explain how you plan to track the policy influence of your study (see appendix on i2i sample indicators of IE influence on program/policy. These indicators, which are currently under revision, will be collected through Grant Monitoring and Reporting on an annual basis from all i2i-supported IEs).

Theory of Change (E)

(1 figure and 2–3 paragraphs)

♦ Describe the main elements of the intervention, and the hypothesized causal chain from inputs, through activities and outputs, to outcomes.

♦ Describe the main assumptions and other factors underlying the causal chain (internal and external).

A theory of change describes how the intervention is expected to affect the outcomes of interest (based on theory), but it does not demonstrate whether the intervention causes the observed outcomes. It usually includes the most important outcomes (intermediate and final) that are critical to the causal chain, even if not all will be measured (see appendix for example).

A theory of change sets the structure for the hypotheses, evaluation questions, and outcomes of interest. It also lists key indicators for developing the implementation protocol and IE monitoring system aimed at understanding what is being evaluated, and whether the critical intervention activities/components were implemented/taken up as planned.


– 54 –

Hypotheses/Evaluation Questions (E,R)

(1/2 page)

♦ List the hypotheses derived from your theory of change.

♦ List the main evaluation question(s) to be addressed by the proposed study. Evaluation questions connect the specific intervention/treatment variation to the outcomes of interest, and end with a question mark. They should be in the following format: What is the impact of <intervention/intervention variation> on <outcomes>? E.g., What is the impact of a parental workshop on financial literacy on student knowledge, attitudes, and behavior?

♦ You may have a broad evaluation question based on the knowledge gap and the strategy proposed. However, the number of specific questions in this section should be perfectly aligned to the number of your treatment arms (i.e., if you have three treatment arms, you should have three specific evaluation questions). Each question can be evaluated on a vector of outcomes (i.e., you may organize them as sub-questions). Methods to answer sub-questions on heterogeneous treatment

effects and spillovers should be described in the methods section.

♦ Describe how the evaluation questions were derived.

Main Outcomes of Interest (E,R)

(1 table)

♦ Briefly list and define main outcomes of interest (primary and secondary/intermediate) as in table 1.

♦ Further details on how the outcomes will be measured/collected will go in the data collection section.

Evaluation Design and Sampling Strategy (E,R)

(2 pages or less)

♦ Present the main features of the proposed evaluation design to address the evaluation question(s).

♦ Describe precisely the identification strategy (e.g., trial design including clustering, factorial, stratification details) for each evaluation question.

♦ Report all inclusion/exclusion criteria to de-fine the target population/population studied, providers, settings, and clusters (as relevant).

TAblE 1: Main Outcomes of Interest

Outcome Type Outcome Name Definition Measurement level

Primary/Secondary


– 55 –

♦ Report any ethical issues that may arise concerning the evaluation design and the sampling strategy (not related to data collection).

Treatment and Control groups

♦ Provide specific description of features of each control and treatment arm (one paragraph per arm).

Sample Size Calculations

♦ Present the sample size estimates. Describe how the sample size was determined, including the sampling frame, and main assumptions, including Minimum Detectable Effect (MDE), variance estimates, intra-cluster correlation, and units per cluster (if applicable).

Data Collection (E,R)

(1 page if basic, 1–2 pages if include all sections for registration and ethical clearance)

♦ Describe main instruments for data collection.

Quantitative Instruments

♦ Describe how primary and secondary outcomes (from section 7) will be measured, their timing and frequency.

Management of Data Quality

♦ Describe methods used to enhance the quality of measurements (e.g., multiple observations, training of surveyors), electronic data collection, protocols for quality assurance.

Ethical Issues

♦ Describe if this IE will require ethical approval, informed consent procedures, and important ethical considerations related to data collection.

Qualitative Instruments

♦ Provide a description of all qualitative instruments (if applicable).

IE Implementation Monitoring System (R)

♦ Describe the IE implementation monitoring system, particularly what specific indicators and system will be used to follow up the studied population, their treatment participation, treatment actually delivered and received based on activities, and outputs (see the theory of change section).

Data Processing and Analysis(1–2 pages)

Data Coding, Entry, and Editing20(E)

♦ Describe planned methods for data entry, and for handling missing data, imputations.

Model Specification for Quantitative Data Analysis

♦ Describe the statistical method(s) that will be used to compare groups for primary and secondary outcomes (the specific equation should be included), any transformations to quantitative data. Specify whether the standard errors will be clustered or corrected.

20 This subsection is optional.


– 56 –

♦ Specify what IE parameter of interest will be estimated (e.g., ITT, TT, MTE, LATE).

♦ Describe how you plan to address multiple hypothesis testing.

♦ Describe methods for additional analyses, including spillovers and subgroup analyses.

♦ Provide a list of any variables to be collected to check balance and correct for potential selection due to attrition, non-response, take-up rate issues (all theoretically important variables to be measured at baseline, including those thought to be related to participation/dropout/non-response and the outcomes of interest).

♦ Lay out a strategy to follow up, test, and correct for (if required) sources of bias (e.g., non-random attrition, non-response, endogenous take-up).

♦ State if you plan to register this IE (see selected links below). • AEA RCT Registry (https://www.

socialscienceregistry.org/) • 3ie Registry (http://www.3ieimpact.org/

evaluation/ridie/)

Study limitations and Risks (E)

(1/2 page)

♦ Provide an assessment of risk and threat to internal validity (related to previous section).

♦ Discuss issues related to external validity, par-ticularly (i) representativeness of the sample; (ii) representativeness of the institution(s) delivering the intervention, and (iii) feasibility that the intervention can be scaled up.

IE Management (E,R)

(All tables)

Evaluation Team and Main Counterparts

♦ Provide list of all IE team members with their position, affiliation, and responsibilities (including lead researcher, other research team members, and all project staff involved in the IE work, and main implementing agency counterparts).

TAblE 2: IE Team and Main Counterparts

Name Role Organization/unit

Principal investigators (specify Lead Researcher)

Other IE team members (specify IE TTL and Field Coordinator)

WBG project staff involved in the IE (if the IE is related to a WBG project, specify project TTL)

Main implementing and policy counterparts



http://www.3ieimpact.org/evaluation/ridie/

http://www.3ieimpact.org/evaluation/ridie/


– 57 –

Work Plan and Deliverables

♦ Table 3.

budget

(1 paragraph)

♦ Present total budget and disaggregated by staff time, data collection, and travel. Include all sources of funding, both Bank-executed and client-executed (BB resources, trust fund and grants, FBS, EFO, project financing for the IE, such as data collection, and other client financing). Estimate and include all research/staff time (not only the time charged).

♦ Attach detailed budget (see excel file template).

Plan for using Data and Evidence from the Study(1 paragraph)

♦ Describe communication, participation, and dissemination strategy (potential users of findings, media channels) at all stages of the IE (design, baseline analysis, mid-corrections, follow-up analysis, and final results).

TAblE 3: Milestones, Deliverables, and Estimated Timeline

Milestones Deliverables Completion Date

Peer-reviewed Concept Note Methodology note May 15, 2013

Data collection plan and pilot TORsQuestionnaires

Data collection (Baseline) Cleaned dataDictionaries

First data analysis Presentation Data fileDo filesBaseline report

Implementation of intervention aligned to evaluation

Rollout planMonitoring reports verifying treatment and control status

Follow-up data collection plan TORsQuestionnaire

Data collection (follow-up) Cleaned dataDictionaries

Final report and policy notes Technical notePolicy noteData fileDo files

Dissemination of findings Presentations May 30, 2017

TAblE 4: Total Budget per Category

Category uSD %

Staff

STC

Data Collection

Travel

Total

https://www.wbginvestmentclimate.org/results/upload/Method_Note_Kenya_HI_06Nov2013_ext.pdf


– 58 –

aPPEnDix

i2i inDicatorS of iE influEncE on ProGram/Policy

Indicator DefinitionExample of Specific Output

(required)

Quality of Policy Design

Rationalized policy design (Yes=1, No=0)

IE improved design based on clear understanding of the underlying theory of change (causal links between the intervention components and the outcomes) and highlighted areas of uncertainty and critical assumptions.

The IE helped the [matching grant project] develop better communication and technical assistance strategies to address [common take-up/participation issues].

Introduced structured learning (multiple treatment arms) (Yes=1, No=0)

IE included multiple treatment arms to determine causal mechanisms, and compared the effectiveness of different interventions in practice.

The IE introduced [alternative drug supply chains] to improve delivery of [medicine to the facilities in Zambia].

Adopted solution from existing IE results (Yes=1, No=0)

IE exposed program providers to existing evidence across sectors and/or countries and affected agreement on what components of intervention might work, and what might need to be removed using existing evidence.

The IE on [Malawi’s gender reservation] was adopted by [Mozambique one year after this IE was designed].

Quality of Implementation

Increased take-up (Yes=1, No=0)

IE improved information and/or incentives that resulted in increased proportion of people that use a program (out of those targeted by the program).

The IE motivated [the court to enforce the digitalization of court cases] and increased the proportion of [judges moving from paper-based to computer-based processes in Senegal].

Improved delivery (Yes=1, No=0)

IE ensured that the treatment reached the treated in an effective and timely manner. IE clarified what the treatment entailed, to whom it will be delivered, when and where it will take place, and provided feedback to the implementation agency to act upon differences between planned and executed treatment.

The incentives provided to increase adherence to treatment also increased [computer use by judges in Senegal].

Improved timeliness of implementation (Yes=1, No=0)

IE helped create a better design that improved efficiency of implementation, translating into better planned disbursement schedule and faster speed of disbursement. IE field coordinators followed/facilitated implementation.

A technical working group, formed as part of the IE [used to define changes in the regulatory framework for health facility inspections in Kenya], helped speed up project implementation.

— Continued


– 59 –



(required)

Quality of Data

High-quality baseline survey (Yes=1, No=0)

IE provided a full high-quality baseline survey (including covariates, with sufficient sample size, and representative of policy-affected population) available to policy makers and researchers creating or building on instruments for policy making even before the IE started.

High-quality follow-up survey(s) (Yes=1, No=0)

IE included comparable follow-up panel data for project.

Improved administrative data (Yes=1, No=0)

IE data requirements for planning of data collection rounds strengthened the M&E function and reporting of key indicators.

The IE data requirements [convinced the Minister of Education to introduce unique school identifiers] that improved their administrative data.

Availability of data as a public good (Yes=1, No=0)

The location of IE project data is known and accessible or it was uploaded in micro-data catalog.

Capacity building

Client participated in IE workshop(s) (Yes=1, No=0)

The client attended IE workshop(s) and was connected to a broader global network of practitioners, policy makers, and experts in a set of relevant areas.

Rio, 06/2011 - DIME-FPD Impact Evaluation of Finance and Private Sector Development

Baseline results discussed with client (Yes=1, No=0)

If baseline results were discussed with client, please specify the client type, how it was discussed with the client, and the location/event of the discussion.

The IE team held a [workshop, meeting, VC] to present baseline findings on [date].

IE results discussed with client (Yes=1, No=0)

IE analysis and results were discussed with the client to understand their policy relevance and application. Please specify the client, how it was discussed with the client, and the location/event of the discussion.

The IE team held a [workshop, meeting, VC] to discuss IE results on [date].

Training provided for data analysis (Yes=1, No=0)

IE improved skills of local institutions and/or staff to develop and implement IE, general monitoring, and other data analysis independently (through discussions, technical assistance, workshops, and other training channels).

The IE field coordinator led trainings on CSPro, Stata, and SPSS to the Minister of Agriculture, allowing the team to use the data for their programming and improve their data-entry function.

— Continued


– 60 –



(required)

Quality of Policy Decisions

Baseline informed policy design/implementation (Yes=1, No=0)

IE baseline data was used by governments and other stakeholders to stimulate policy dialogue and/or help identify problems and solutions.

The IE identified [incentive problems in the size of women groups in Ethiopia] and helped project [introduce specialization in group composition].

Adopted the results of testing causal mechanisms or packages based on the IE (Yes=1, No=0)

IE evidence from experimental testing of alternative mechanisms or packages was used by governments or other stakeholders to inform policy decisions.

The IE identified more efficient [supply chain system] and government agreed to scale up at [national level in Zambia].

IE results were used to motivate scale-up/scale-down of policy at national level (Yes=1, No=0)

IE results reported success (or insufficient) impact of the intervention in achieving desired outcomes and were used by governments and/or other agencies/stakeholders to motivate scale-up (scale-down) of policy at national level.

The IE tested the [financial literacy program] and the [government] decided to scale up/scale down at the [national level].

Quality of Dissemination

Number of presentations to policy makers of IE results

The number of presentations on the IE results given to policy makers.

Number of presentations to academics of IE results

The number of presentations on the IE results given to academics.

theory of change Example

Inspections Regimes in Health Care and Their Impact on Patient Safety Standards and Quality of Care in Kenya

The evaluation targets all health facilities (HFs) in three counties of Kenya (around 1,000, or 10 percent of the total number of health facilities in the country). It will have three arms, each of which is expected to affect quality of care and patient safety for all the population in their catchment areas. The details on the treatment arms are discussed in section IV. The control group will be the “Business-as-usual” Low-intensity Health

Inspections Regime, the first treatment arm will be a High-intensity Public Health Inspections Regime with Public Disclosure of the health facility’s patient safety score, and the second arm will be a Private System of Supportive Supervision.

There are three common components across these arms: (1) a regulatory framework accompanied by clear guidelines on the minimum patient safety standards that facilities are expected to comply with; (2) a monitoring system to track compliance with minimum patient safety standards over time; and (3) a scoring and information report card system to publicize health facilities’ compliance with minimum patient safety


– 61 –

standards. Each treatment arm will include two or three of these components. Intervention activities are defined by these three components as follows:

♦ A regulatory framework accompanied by clear guidelines on the minimum patient safety standards: Activities in this component include the streamlining of the Checklist, the development of a detailed implementation manual, a score system to grade health facilities and a system of warnings and sanctions for non-compliant HFs. All these activities have been taking place during the last several months, from a process that started some years ago (see Checklist Process Note in appendix 2 for further details). The new regulatory framework is schedule to be completed by the end of June 2014 (See Draft of Checklist in appendix 3).

♦ A monitoring system to track compliance with minimum patient safety standards: This component includes activities related to the definition of the parameters for the inspections and the supportive supervision, including who the inspectors/supervisors will be, how often the inspections/supportive supervision will take place, and the system to enforce the warnings and sanctions developed in (1). Several of these elements have been defined, and a planning meeting with the participant counties and the MOH is scheduled for August/September 2014.

♦ A scoring and information report card system to publicize health facilities’ compliance with minimum patient safety standards: The scoring system is part of the regulatory framework from (1), and the report card system will be developed consistent with that scoring in a way that

effectively communicates to patients the status of the health facilities.

Figure 1 presents a simplified theory of change behind the broad intervention to be evaluated, including its main components/inputs, activities, outputs, and the hypothesized causal chain to select outcomes of interest.

The main assumption behind this theory of change is that the combination of two or more of these components leads to inspection/supportive supervision systems that provide incentives for health facilities to comply with/increase patient safety standards (that is, clear rules of the game, good information systems, a working system of warnings and sanctions, an effective supportive supervision, as well as a consumer liability system through the scorecard grading that creates costs/benefits that incentivize health facilities to improve patient safety).

For instance, activities and outputs from component (1) are expected to affect the knowledge of the health facilities in terms of quality and patient safety, which is a necessary (although not sufficient) condition to affect compliance with the standards. Activities and outputs in component (2) are expected to directly affect compliance by creating incentives (and costs) for noncompliance. Finally, the third component is expected to affect consumer demand, which in turn may lead to changes in provider behavior and a reallocation of demand to facilities with higher patient safety scores. In the long-term, better quality of care contributes to improvements in the health outcomes of the population served by health facilities.

We examine the impact of the interventions on a triad of measures that takes into account our setting with public and private providers: Quality and Patient Safety in the market, Quantity in the market, and Prices in the market. The first set


– 62 –

of outcomes are intermediate outcomes in the continuum of patient safety and quality of health care services measured through (a) compliance with a quality of care and patient safety checklist; (b) compliance with key infection control measures in doctor-patient interactions; (c) accuracy of diagnosis and treatment as evaluated

through the use of standardized patients; (d) prevalence of unnecessary or harmful medication; and (e) prevalence of substandard drugs. Consumer and provider behavior are further measured through (a) patient loads in different health facilities and (b) prices charged to patients for consultations and laboratory procedures.

FIguRE 1: Theory of Change of a High-Stakes and High-Consumer-Liability Health Inspection Regime

Inputs Activities Outputs

Regulatory framework and rules on standards of

PS/QC

Develop standards and guidelines on PS/QC

Develop a scoring system for HFs, and warnings and

sanctions for different levels of non-compliance with

standards

Development dissemination plan on standards for HFs

Standards and guidelines on PS/QC available to HFs

Scoring and warnings and sanctions system for

different levels of non-compliance with PS/QC

available to HFs

Number of presentations and information products on framework delivered to HFs

System to check for

compliance/progress

with PS/QC

standards

Develop inspections parameters (e.g., inspectors, tool for inspectors, training, frequency of inspections)

Development enforcement plan for warnings and

sanctions

Develop supportive supervision parameters and plan-if any (e.g., supervisors,

frequency)

Develop dissemination plan about inspections/supportive supervision

systems for HFs

Inspections conducted according to defined

parameters (record of inspections, frequency,

duration)

Warnings and sanctions enforced (number of warnings

and sanctions followed up/enforced)

Supportive supervision sessions conducted according

to the defined parameters

Presentations/dissemination activities conducted

Scorecard grading and information system for

HF

Develop signaling strategy and information system for

HF (e.g., scorecards)

Develop dissemination plan of signaling for patients

and HFs

Signaling system developed

Scorecards/signals posted after inspection/supportive

supervision

Dissemination activities for patients and HFs; patients’ awareness/recognition of

signaling

Short- and intermediate-term outcomes

Long-term outcomes

Improved knowledge of HFs about PS/QC

standards

Improved knowledge and attitudes

(choices) of patients regarding HFs and

their PS/QC

Improved quality of Care Delivered and PS practices

HF: Health FacilityPS: Patient SafetyQC: Quality of Care

Improved Health Indicators

Improved Compliance with PS/QC standards

1

2

3


– 63 –

bu

DG

Et f

or

an

imPa

ct

Eva

luat

ion

- t

EmPl

atE

Incl

ude

over

head

s an

d re

cord

all

the

tim

e fr

om r

esea

rche

rs, T

TLs

, and

Wb

sta

ff e

ven

if it

is n

ot c

harg

ed

Mile

ston

eC

ateg

ory

Fisc

al

Year

Des

crip

tion

Uni

t

Cos

t

(US

D)

Uni

t (N

o)U

nit

N

ame

Tota

l Cos

t (U

SD

)

Sou

rce

of F

unds

(in

clud

e al

l diff

eren

t so

urce

s of

fu

ndin

g: b

b, T

F, o

ther

)M

ilest

one

S

um

Con

cept

Not

eS

taff

FY13

Prim

ary

Inve

stig

ator

/R

esea

rche

r (G

leve

l)w

eeks

-

Con

cept

Not

eET

CFY

13C

o-PI

/Res

earc

her

(F

leve

l)w

eeks

-

Con

cept

Not

e S

taff

FY13

TT

L/R

egio

n S

taff

(F

leve

l)w

eeks

-

Con

cept

Not

e Tr

avel

FY14

Sco

ping

Mis

sion

pers

on x

w

eek

-

-

Bas

elin

e D

ata

Col

lect

ion

Sta

ffFY

14Pr

imar

y In

vest

igat

or/

Res

earc

her

(G le

vel)

wee

ks -

Bas

elin

e D

ata

Col

lect

ion

Sta

ffFY

14C

o-PI

/Res

earc

her

(F

leve

l)w

eeks

-

Bas

elin

e D

ata

Col

lect

ion

STC

FY14

Res

earc

h A

ssis

tant

wee

ks -

Bas

elin

e D

ata

Col

lect

ion

Sta

ffFY

14Pr

ojec

t M

anag

erw

eeks

-

Bas

elin

e D

ata

Col

lect

ion

STC

FY14

Fiel

d C

oord

inat

orw

eeks

-

Bas

elin

e D

ata

Col

lect

ion

Trav

elFY

14S

uper

visi

on M

issi

on

(tra

inin

g/su

rvey

la

unch

)

pers

on x

2

wee

ks -

Bas

elin

e D

ata

Col

lect

ion

Trav

elFY

14Lo

cal T

rave

l (F

ield

Coo

rdin

ator

)pe

rson

x

wee

k -

Bas

elin

e D

ata

Col

lect

ion

Dat

a co

llect

ion

FY14

Hea

lth

Faci

lity

Sur

veys

heal

th

faci

litie

s -

-

Bas

elin

e A

naly

sis

Sta

ffFY

15Pr

imar

y In

vest

igat

or/

Res

earc

her

(G le

vel)

wee

ks -

b. i

2i iE

co

nc

EPt

no

tE

bu

DG

Et t

EmPl

atE

— C

ontin

ued


– 64 –

Bas

elin

e A

naly

sis

Sta

ffFY

15C

o-PI

/Res

earc

her

(F

leve

l)w

eeks

-

Bas

elin

e A

naly

sis

Sta

ffFY

15Pr

ojec

t M

anag

erw

eeks

-

Bas

elin

e A

naly

sis

STC

FY15

Res

earc

h A

ssis

tant

wee

ks -

Bas

elin

e A

naly

sis

Trav

elFY

15D

isse

min

atio

n M

issi

on/W

orks

hop

Bas

elin

e

pers

on x

tr

ip -

-

Inte

rven

tion

Mon

itor

ing

Sta

ffFY

15Pr

imar

y In

vest

igat

or/

Res

earc

her

(G le

vel)

wee

ks -

Inte

rven

tion

Mon

itor

ing

Sta

ffFY

15C

o-PI

/Res

earc

her

(F

leve

l)w

eeks

-

Inte

rven

tion

Mon

itor

ing

Sta

ffFY

15T

TL/

Reg

ion

Sta

ff

(F le

vel)

wee

ks -

Inte

rven

tion

Mon

itor

ing

Sta

ffFY

15Pr

ojec

t M

anag

erw

eeks

-

Inte

rven

tion

Mon

itor

ing

STC

FY15

Fiel

d C

oord

inat

orw

eeks

-

Inte

rven

tion

Mon

itor

ing

Trav

elFY

15Lo

cal T

rave

l (F

ield

Coo

rdin

ator

)pe

rson

x

wee

k -

-

follo

w-u

p D

ata

Col

lect

ion

Sta

ffFY

15Pr

imar

y In

vest

igat

or/

Res

earc

her

(G le

vel)

wee

ks -

follo

w-u

p D

ata

Col

lect

ion

Sta

ffFY

15C

o-PI

/Res

earc

her

(F

leve

l)w

eeks

-

follo

w-u

p D

ata

Col

lect

ion

STC

FY15

Res

earc

h A

ssis

tant

wee

ks -

follo

w-u

p D

ata

Col

lect

ion

Sta

ffFY

15T

TL/

Reg

ion

Sta

ff

(F le

vel)

wee

ks -

follo

w-u

p D

ata

Col

lect

ion

Sta

ffFY

15Pr

ojec

t M

anag

erw

eeks

-

follo

w-u

p D

ata

Col

lect

ion

STC

FY15

Fiel

d C

oord

inat

orw

eeks

-

follo

w-u

p D

ata

Col

lect

ion

Trav

elFY

15S

uper

visi

on M

issi

on

(tra

inin

g/su

rvey

la

unch

)

wee

ks x

1

pers

on -

follo

w-u

p D

ata

Col

lect

ion

Trav

elFY

15Lo

cal T

rave

l (F

ield

Coo

rdin

ator

) w

eeks

x 1

pe

rson

-

Mile

ston

eC

ateg

ory

Fisc

al

Year

Des

crip

tion

Uni

t

Cos

t

(US

D)

Uni

t (N

o)U

nit

N

ame

Tota

l Cos

t (U

SD

)

Sou

rce

of F

unds

(in

clud

e al

l diff

eren

t so

urce

s of

fu

ndin

g: b

b, T

F, o

ther

)M

ilest

one

S

um

— C

ontin

ued


– 65 –

Mile

ston

eC

ateg

ory

Fisc

al

Year

Des

crip

tion

Uni

t

Cos

t

(US

D)

Uni

t (N

o)U

nit

N

ame

Tota

l Cos

t (U

SD

)

Sou

rce

of F

unds

(in

clud

e al

l diff

eren

t so

urce

s of

fu

ndin

g: b

b, T

F, o

ther

)M

ilest

one

S

um

follo

w-u

p D

ata

Col

lect

ion

Dat

a co

llect

ion

FY15

Hea

lth

Faci

lity

Sur

veys

heal

th

faci

litie

s -

Fina

l Ana

lysi

s S

taff

FY16

Prim

ary

Inve

stig

ator

/R

esea

rche

r (G

leve

l)w

eeks

-

Fina

l Ana

lysi

s S

taff

FY16

Co-

PI/R

esea

rche

r

(F le

vel)

wee

ks -

Fina

l Ana

lysi

s S

taff

FY16

Proj

ect

Man

ager

wee

ks -

Fina

l Ana

lysi

s S

TCFY

16R

esea

rch

Ass

ista

ntw

eeks

-

-

Dis

sem

inat

ion

of F

indi

ngs

Sta

ffFY

16Pr

imar

y In

vest

igat

or/

Res

earc

her

(G le

vel)

wee

ks -

Dis

sem

inat

ion

of F

indi

ngs

Sta

ffFY

16C

o-PI

/Res

earc

her

(F

leve

l)w

eeks

-

Dis

sem

inat

ion

of F

indi

ngs

Sta

ffFY

16T

TL/

Reg

ion

Sta

ff

(F le

vel)

wee

ks -

Dis

sem

inat

ion

of F

indi

ngs

Sta

ffFY

16Pr

ojec

t M

anag

erw

eeks

-

Dis

sem

inat

ion

of F

indi

ngs

Trav

elFY

16D

isse

min

atio

n M

issi

on/W

orks

hop

Fina

l

pers

on x

2

wee

ks -

-

Tota

l -

TOTA

L b

UD

GET

PER

CAT

EGO

RY

Cat

egor

yU

SD

%

Sta

ff

STC

Dat

a C

olle

ctio

n

Trav

el

Tota

l



– 67 –

c. i

2i r

ESu

ltS

fr

am

EWo

rK

Out

com

esIn

dica

tors

2014

/15

Plan

ned

2014

/15

Act

ual

2015

/ 20

1620

16/

2017

2017

/ 20

1820

18/

2019

Tota

lS

ourc

e

1. Im

prov

ed a

war

enes

s,

dem

and

for,

and

use

of e

vide

nce

in d

ecis

ion

mak

ing

in k

ey s

ecto

rs

Num

ber

of im

pact

eva

luat

ions

th

at in

flue

nce

proj

ect

desi

gn o

r im

plem

enta

tion

-15

814

83

33D

IME

annu

al r

epor

t (q

uant

itat

ive

&

qual

itat

ive

data

)

Num

ber

of c

ompl

eted

impa

ct

eval

uati

ons

that

gen

erat

e ev

iden

ce u

sed

to s

uppo

rt p

roje

ct a

dopt

ion,

sca

le-u

p,

scal

e-do

wn,

con

tinu

atio

n, o

r ca

ncel

atio

n de

cisi

ons

-3

-7

910

27D

IME

annu

al r

epor

t (q

uant

itat

ive

&

qual

itat

ive

data

)

Num

ber

of m

ulti

-arm

eva

luat

ion

desi

gns

(out

of

num

ber

of im

pact

eva

luat

ion

star

ted)

to

test

pro

ject

inno

vati

ons

50

%74

%5

0%

50

%-

-5

0%

DIM

E an

nual

rep

ort

(qua

ntit

ativ

e &

qu

alit

ativ

e da

ta)

Prop

orti

on o

f pr

ojec

t te

ams

part

icip

atin

g in

the

wor

ksho

p th

at

impr

ove

thei

r kn

owle

dge

of im

pact

ev

alua

tion

tec

hniq

ues

50

%79

%5

0%

50

%5

0%

50

%5

0%

Sta

ndar

dize

d be

fore

an

d af

ter

test

co

nduc

ted

at t

he

wor

ksho

p

Prop

orti

on o

f pr

ojec

t te

ams

part

icip

atin

g in

an

impa

ct e

valu

atio

n w

orks

hop

that

sub

mit

a p

ropo

sal i

n th

e re

leva

nt i2

i fun

ding

win

dow

50

%73

%5

0%

50

%5

0%

50

%5

0%

Onl

ine

subm

issi

on

reco

rds

(to

be

repo

rted

in D

IME

annu

al r

epor

t)

2. Im

prov

ed

effec

tive

ness

and

im

pact

of

new

and

ex

isti

ng d

evel

opm

ent

inte

rven

tion

s fr

om

DFI

D a

nd w

ider

co

mm

unit

y

Prop

orti

on o

f co

mpl

eted

IEs

that

are

ci

ted

in t

he p

roje

ct d

ocum

ent

or d

irect

ly

used

to

info

rm n

ew d

evel

opm

ent

inte

rven

tion

s

--

-25

%25

%25

%25

%D

IME

annu

al r

epor

t (q

uant

& q

ual d

ata)

— C

ontin

ued


– 68 –

Out

puts

Indi

cato

rs20

14/1

5 Pl

anne

d20

14/1

5

Act

ual

2015

/ 20

1620

16/

2017

2017

/ 20

1820

18/

2019

Tota

lS

ourc

e

1. In

crea

sed

quan

tity

, qu

alit

y, a

nd r

elev

ance

of

evi

denc

e av

aila

ble

to

impr

ove

polic

y de

sign

an

d im

plem

enta

tion

in

unde

rser

ved

area

s

Num

ber

of p

revi

ousl

y on

goin

g im

pact

ev

alua

tion

s co

mpl

eted

and

rep

orte

d (e

.g.,

as w

orki

ng p

aper

s or

pol

icy

brie

fs

publ

ishe

d on

line)

-6

917

1717

60D

IME

annu

al r

epor

t

Num

ber

of n

ew d

ata

sets

(ba

selin

e or

fo

llow

-ups

)-

29-

2527

308

2D

IME

annu

al r

epor

t

Num

ber

of im

pact

eva

luat

ions

sta

rted

w

ithi

n th

e i2

i por

tfol

io a

cros

s th

e fo

llow

ing

(new

IEs)

:

540

3030

--

65D

IME

annu

al r

epor

t

# o

f ev

alua

tion

s%

of

curr

ent

port

folio

Gen

der

(cro

ss-c

utti

ng)

410

%25

%D

IME

annu

al r

epor

t

FCS

14

35%

15%

DIM

E an

nual

rep

ort

Gov

erna

nce

-23

%D

IME

annu

al r

epor

t

Clim

ate

Cha

nge

1640

%21

%D

IME

annu

al r

epor

t

Sha

red

Pros

perit

y10

25%

41%

DIM

E an

nual

rep

ort

Num

ber

of im

pact

eva

luat

ions

in:

Sub

-Sah

aran

Afr

ica

225

5%

57%

Mid

dle

East

and

Nor

th A

fric

a2

5%

3%

Euro

pe a

nd C

entr

al A

sia

13%

7%

Lati

n A

mer

ica

and

the

Car

ibbe

an

923

%14

%

East

Asi

a an

d th

e Pa

cifi

c 1

5%

8%

Sou

th A

sia

513

%11

%

Low

-inco

me

and

low

er-m

iddl

e-in

com

e co

untr

ies

2973

%75

%

Upp

er-m

iddl

e-in

com

e co

untr

ies

1127

%25

%

— C

ontin

ued


– 69 –

Out

puts

Indi

cato

rs20

14/1

5 Pl

anne

d20

14/1

5 A

ctua

l20

15/

2016

2016

/ 20

1720

17/

2018

2018

/ 20

19To

tal

Sou

rce

2. In

crea

sed

capa

city

of

pol

icy

mak

ers,

im

plem

ente

rs,

rese

arch

ers,

and

pa

rtne

rs t

o in

corp

orat

e ri

goro

us e

vide

nce/

tool

s in

to p

olic

y m

akin

g

Num

ber

of D

IME

wor

ksho

ps4

44

42

216

DIM

E an

nual

rep

ort

Num

ber

of t

eam

s pa

rtic

ipat

ing

in

wor

ksho

ps36

94

3636

1818

144

DIM

E an

nual

rep

ort

Prop

orti

on o

f te

ams

part

icip

atin

g in

w

orks

hops

tha

t co

me

from

low

- or

lo

wer

-mdd

le-in

com

e co

untr

ies

75%

73%

75%

75%

75%

75%

75%

Num

ber

of p

eopl

e tr

aine

d (W

B, d

onor

s,

gove

rnm

ent,

oth

er)

216

439

216

216

108

108

864

DIM

E an

nual

rep

ort

Kno

wle

dge

revi

ews

for

polic

y m

akin

g,

incl

udin

g m

echa

nism

s, s

ecto

r-to

-sec

tor

and

coun

try-

to-c

ount

ry a

dapt

atio

n le

sson

s

-1

-1

11

3D

IME

annu

al r

epor

t

3. In

crea

sed

com

mun

itie

s of

pra

ctic

e an

d kn

owle

dge

plat

form

s fo

r sh

arin

g of

evi

denc

e an

d it

s ap

plic

atio

n fo

r po

licy

mak

ing

Num

ber

of o

rgan

isat

ions

(don

ors,

m

ulti

late

ral d

evel

opm

ent

bank

s,

gove

rnm

ent

agen

cies

, aca

dem

ic

inst

itut

ions

, NG

Os,

etc

.) pa

rtic

ipat

ing

in i2

i kno

wle

dge

shar

ing

even

ts a

nd

belo

ngin

g to

i2i n

etw

orks

1218

215

1515

1774

DIM

E an

nual

rep

ort

Num

ber

of d

isse

min

atio

n ev

ents

(c

onfe

renc

es, s

emin

ars)

1516

1515

1515

75D

IME

web

site

Num

ber

of p

arti

cipa

nts

of d

isse

min

atio

n ev

ents

250

543

250

250

250

250

125

0D

IME

annu

al r

epor

t



– 71 –

D. i2i WorKSHoP ParticiPation

This appendix provides a list of the different agencies represented in the i2i workshops, such as government agencies, international organizations, universities, research institutes, and nongovernmental organizations (NGOs).

Government Agencies

Argentina Secretariat of Energy

Argentina Secretariat of Environment and Sustainable Development

bangladesh Ministry of Agriculture

bangladesh Infrastructure Development Company Limited (IDCOL)

bangladesh Ministry of Planning

bangladesh Bangladesh Rural Electrification Board

brazil Ministry of Environment

brazil Government of the Regional Action of the State of Bahia

brazil National Rural Education Service (SENAR)

brazil Ministry of Agriculture, Livestock and Food Supply

brazil Secretariat of Sports and Leisure

brazil Municipality of Recife

brazil Municipality of Rio de Janeiro

brazil Government of the State of Pernambuco

brazil City Department of Finance, Information Technology and Internal Control

burkina Faso Ministry of the Environment and Sustainable Development

burundi Ministry of Agriculture

Cameroon Ministry of Agriculture

China Ministry of Agriculture

Colombia Municipality of Cali

Colombia National Planning Department

Congo, Dem. Rep. Ministry of Public Works

Congo, Dem. Rep. Funds Social Agency, Office of the President

El Salvador Ministry of Justice and Public Safety

Ethiopia Bureau of Pastoral and Agricultural Development

Ethiopia Ministry of Finance

Ghana Environmental Protection Agency of Ghana

Ghana Forestry Commission

Ghana Ministry of Environment, Science, Technology and Innovation

Ghana Ministry of Food and Agriculture

— Continued


– 72 –

Guatemala Ministry of the Interior

Guatemala Municipality of Villa Nueva

Guatemala Superintendence of Tax Administration

Guinea Ministry of Civil Service

Haiti Ministry of Agriculture and Rural Development

India Ministry of Rural Development

Jamaica Ministry of National Security

Jamaica Ministry of Finance and Planning

Japan Japan International Cooperation Agency (JICA)

Jordan National Electric Power Company

Jordan Justice Center for Legal Aid (JCLA)

Kazakhstan Supreme Court

Kazakhstan Ministry of Justice

Kenya Ministry of Environment, Water and Natural Resources

Kenya National Irrigation Board

Kenya Ministry of Justice

Lebanon Ministry of Justice

Lebanon Ministry of Finance

Lesotho Ministry of Development Planning

Liberia Ministry of Agriculture

Liberia Civil Service Agency

Liberia Governance Commission

Liberia Land Commission

Liberia Center for National Documents and Records Agency

Madagascar Ministry of Finance and Budget

Mexico National Forestry Commission

Mexico Ministry of the Interior

Mexico Undersecretary for Prevention and Citizen Participation

Mozambique Ministry of Agriculture

Nepal Ministry of Agriculture

Nepal Ministry of Health and Population

Nepal Ministry of Foreign Affairs

Niger Ministry of Planning

Norway Agency for Development Cooperation (NORAD)

Pakistan Ministry of Finance and Economic Development

Pakistan Chief Ministers Secretariat

Pakistan Peshawar High Court

Government Agencies

— Continued


– 73 –

Government Agencies

Pakistan Planning and Development Department, Government of Sindh

Pakistan Government of Khyber Pakhtunkhwa

Pakistan Government of Sindh

Palestine Ministry of Education

Peru Ministry of Planning

Rwanda Ministry of Agriculture

Senegal Ministry of Agriculture

Swaziland Swaziland Public Procurement Regulatory Agency (SPPRA)

Tanzania Ministry of Agriculture, Food Security and Cooperatives

Tanzania Prime Minister’s Office, Regional Administration and Local Government

Togo National Employment Agency

Uganda Ministry of Agriculture

United Kingdom Department for International Development (DFID)

United States Agency for International Development (USAID)

Zambia Ministry of Agriculture

Zambia Ministry of Lands, Natural Resources and Environmental Protection

International Organizations

African Development Bank (AFDB)

Food and Agriculture Organization (FAO)

Global Agriculture and Food Security Program (GAFSP)

Inter-American Development Bank (IADB)

International Finance Corporation (IFC)

International Fund for Agricultural Development (IFAD)

International Rescue Committee (IRC)

Public Expenditure and Financial Accountability (PEFA)

United Nations Development Programme (UNDP)

World Bank

Universities

Palestine Birzeit University

United States Brigham Young University

England Cambridge University

belgium Catholic University of Leuven

United States Claremont McKenna College

United States Columbia University

United States Cornell University

United States Georgetown University

— Continued


– 74 –

Universities

United States Georgia State University

United States Harvard University

Pakistan Khyber Pakhtunkhwa Judicial Academy

England London School of Economics

United States Massachusetts Institute of Technology (MIT)

Australia Monash University

United States New York University (NYU)

Portugal Nova School of Business and Economics (Nova SBE)

United States Rochester Institute of Technology (RIT)

United States Stanford University

Sweden Stockholm School of Economics

United States Texas A&M University

Netherlands Tilburg University

United States Trinity College

United States Tufts University

Colombia Universidad del Valle

belgium Université Libre de Bruxelles

United Kingdom University College London

Canada University of Calgary

United States University of California, Berkeley

United States University of California, San Diego

United States University of Chicago

United States University of Maryland

United States University of Minnesota

United States University of North Carolina

United States University of Pennsylvania

Pakistan University of Peshawar

Canada University of Toronto

England University of Warwick

Netherlands Wageningen University

United States Williams College

Research Institutes

Center for Effective Global Action (CEGA)

Innovations for Poverty Action (IPA)

Institute of Policy and Strategy in Agriculture and Rural Development (IPSARD)

International and Ibero-American Foundation for Administration and Public Policies (FIIAPP)

— Continued


– 75 –

Research Institutes

Institute for Financial Studies

Stanford Institute for Economic Policy Research

Stockholm University Institute for International Economic Studies

University of Chicago Crime Lab

NGOs

Senegal Association for the Promotion of the Development Base (ASPRODEB)

El Salvador Catholic Relief Services

India Christian Aid

Colombia Cultural and Vocational Center

Colombia Foundation Alvaraice

Colombia Fund for the Consolidation of State and Peace

Honduras Honduran Social Investment Fund (FHIS)

N/A International Initiative for Impact Evaluation (3ie)

Jamaica Jamaica Social Investment Fund

Liberia Liberia Institute of Public Administration

Senegal Network of Peasant and Pastoral Organizations (RESOPP)

Rwanda One Acre Fund

Mexico Runway Citizen

Angola Social Aid Fund (FAS)

Sierra Leone Timap for Justice

Colombia Unit for Attention and Reparation for Victims

— Continued

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=9&cad=rja&uact=8&ved=0CDkQFjAI&url=http%3A%2F%2Fwww.povertyactionlab.org%2Fpartners%2Finternational-initiative-impact-evaluation-3ie&ei=-qE2VcG8CMrisASxh4G4AQ&usg=AFQjCNEQ0aQ97qs7XUNNd46D59jcpN7S2g&sig2=Qh6P8OVvk0Mevk52gKPFgg&bvm=bv.91071109,d.b2w



– 77 –

E. i2i monitorinG inDicatorS on iE cHaractEriSticS anD iE Policy influEncE

— Continued

No. Indicator Definition and Examples (where applicable)

Section 1: IE Profile

1 IE Title Legal title of the IE

2 IE Code IE code for all IEs that have an IE code in SAP or IE portal. This variable can be missing for different reasons, including that the IE started before we started using IE codes or because it has not been generated, or because the team has agreed to use other codes. The use of IE code is not necessarily enforced.

3 IE Contact Person Name/ Survey Respondent

WB member(s) from whom we get information about the IE (updates such as this survey, contacts, etc.)

4 IE Contact Email Email of WB contact member

5 IE TTL Name IE Team Task Leader

6 Country IE country

7 Region IE region

8 Global Practice Global Practice that IE is categorized in: 1. Agriculture 2. Education 3. Energy and Extractives 4. Environment and Natural Resources 5. Finance and Markets 6. Governance 7. Health, Nutrition, and Population 8. Macroeconomics and Fiscal Management 9. Poverty 10. Social Protection and Labor 11. Social, Urban, Rural, and Resilience 12. Trade and Competitiveness 13. Transport and Information and Communication Technologies 14. Water

9 Cross-cutting Solution Area Any Cross-cutting Solution Area that IE falls under: 1. Climate Change 2. Fragility, Conflict, and Violence 3. Gender 4. Jobs 5. Public-Private Partnerships 6. Not Applicable

10 Type of i2i Grant Type of i2i grant IE received: 1. Concept note preparation 2. Implementation 3. Dissemination 4. Data collection


– 78 –

— Continued


11 Primary Topic Primary topic related to IE: 1. Agriculture 2. Civil service reform 3. Climate change 4. Education 5. Energy 6. Environment 7. Finance and private sector 8. Gender 9. Gender-based violence 10. Health 11. Infrastructure 12. Jobs 13. Jobs for at-risk youth as a tool for resilience 14. Justice 15. Natural resource management 16. Public financial management (tax, systems and procurement) 17. Public sector governance 18. Social protection 19. Subnational PSM/decentralization 20. Transport 21. Urban crime and violence 22. Other 23. Not applicable (option available only if secondary or tertiary topics)

12 Status Preparation for CN: Anything before Concept Note (CN)—design details are preliminary or not available Preparation for baseline/Intervention: Between CN stage and baseline baseline: In the field or completed baseline Intervention: Ongoing treatment implementation Follow-up 1: In the field or completed follow-up 1 Follow-up 2: In the field or completed follow-up 2 Follow-up 3: In the field or completed follow-up 3 Analysis: Conducting data analysis Completed: There is a reviewed report on final results Working Paper: There is a finished working paper On hold: On hold IEs (if on hold for more than 6 months/unlike to proceed, it will be at the end of the file, separated from the other IEs) Dropped: The IE has been discontinued

13 IDA If “Country” is an IDA country *Please note that list of IDA countries may change over time

14 Fragile State If “Country” is an Fragile State country, 0 otherwise *Please note that list of FCS countries may change over time

15 IE Start Date Proxy: Earliest date among the following: 1. IE Workshop, 2. Concept note approval (if available), 3. Date in which the IE design was finalized

16 IE End Date Proxy: If the IE is completed, include date of the presentation of final results to the client

Section 1: IE Profile


– 79 –

— Continued


17 IE Estimated Budget Total budget for IE, including staff time, data collection, and travel from all sources of funding, both Bank-executed and client-executed, including all research time even if not charged. Not including implementation budget

18 IE Sources of Funds IE fund sources

19 IE Team and Affiliations Name, organization (e.g., WB, IFC, other), unit affiliation and role (e.g., TTL, PI) of each member of IE team

20 IE evaluates WBG project If IE evaluates a WBG project

21 WBG PID WBG project ID (if IE evaluates a WBG project)

22 WBG Project Name WBG project name (if IE evaluates a WBG project)

23 WBG Project TTL WBG project TTL (if IE evaluates a WBG project)

24 WBG Lending Amount WBG lending amount (if IE evaluates a WBG project)

Section 2: Evaluation Design

25 Intervention Policy/intervention to be evaluated (e.g., financial literacy in high schools)

26 Main Outcomes Most important outcome(s) (e.g., student financial knowledge, attitudes and behavior)

27 Evaluation Methods Type of method: experimental design, non-experimental, or both

28 IE Unit of Intervention/ Randomization

Main level of treatment assignment/randomization if it applies (e.g., school)

29 Number of IE Units of Intervention

Linked to the IE unit of intervention/randomization—e.g., 900 (ballpark numbers—not exact)

30 IE Unit of Analysis Lowest level of analysis on which outcomes are measured (e.g., student)

31 Number of IE Units of Analysis Linked to the previous—e.g., 20,000 (ballpark numbers—not exact)

32 Number of Treatment Arms Number of treatment arms IE contains (not including controls)

33 IE Question (for each treatment arm)

What is the impact of <intervention variation 1> on <outcomes>? E.g., What is the impact of a parental workshop on financial literacy on student knowledge, attitudes, and behavior?

34 Method IE Question (for each IE question)

Main method to estimate IE question: 1. Randomization 2. Phased-in randomization 3. IV 4. Matching 5. RDD 6. Other


– 80 –

— Continued

Section 2: Evaluation Design


35 Area/Package/Mechanism If this treatment arm tests a particular mechanism, categorize the treatment arm into one of the suggested categories below. If the treatment arm tests multiple mechanisms at once, categorize the treatment arm as “Package.” This categorization is indicative and work in progress. (For a general discussion about mechanisms, see, for instance, http://www.itg.be/internet/ds/tde/doc/Astbury%20%26%20Leeuw%20.pdf.)

1. Packages include treatments arms that test more than one mechanism at a time in a way that it is only possible to identify the causal effect of the package (of mechanisms) rather than individuals ones (e.g., conditional cash incentives and information). 2. Incentive mechanisms include conditional monetary and nonmonetary incentives applied either to the supply or the demand. 3. Accountability mechanisms include top-down accountability (audits, inspections, supervision, performance assessment and feedback, laws and regulations) and demand-side accountability (information, report cards, user participation, and monitoring).4. behavioral-bias related mechanisms include limited attention, self-control, cognitive capacity and understanding, and asymmetric valuation of gains and losses. These biases may exacerbate other problems—for example, market failures/inefficiencies—but are also present in perfectly functioning markets. Mechanisms tested include choice simplification, default options, information, framing, demonstration, saliency, precommitment devices, and reminders. 5. Constraint-relaxing mechanisms target the range of possible constraints to capital formation and productivity, including credit, financial, cash flow, financial literacy and life skills, information, inputs, institutional, legal, tax burden, corruption, managerial, property, public goods, risk-management, skills, technology, and transaction costs.6. Delivery mechanisms include targeting rules, centralized/decentralized modality of public service delivery, collective vs. individual or peer-to-peer, private sector vs. public sector delivery, paid vs. voluntary, and the use of media and technology.

36 Area/Mechanism in Packages If package, main mechanisms tested in the arm at once: 1. Behavioral-bias related mechanism 2. Constraint-relaxing mechanism 3. Delivery mechanism 4. Incentive mechanisms 5. Top-down accountability 6. Demand-side accountability 7. Other


– 81 –

— Continued


37 Subcategory of Mechanism Categories

Further categorize each mechanism into subcategories as follows: 1. behavioral bias: (a) automation; (b) choice simplification, default options; (c) feedback, (d) information framing, demonstration, saliency; (e) precommitment devices; (f) reminders; (g) other 2. Constraints: (a) credit, financial, cash flow; (b) other capital inputs; (c) managerial skills, (d) sector-specific skills; (e) other human capital skills inputs; (f) legal, tax burden; (g) technology; (h) other institutional inputs; (i) other 3. Delivery: (a) central/decentralized (how to deliver), (b) collective (how to deliver), (c) mainstream media (edutainment) (how to deliver), (d) technology (how to deliver), (e) community worker (who delivers), (f) male/female (who delivers), (g) peer-to-peer (who delivers), (h) private sector, (i) public sector, (j) volunteer, (k) other 4. Demand-side accountability: (a) community monitoring; (b) information; (c) report cards, feedback; (d) user empowerment; (e) user participations; (f) other 5. Incentives: (a) conditional/monetary/demand, (b) conditional/monetary/supply, (c) conditional/monetary/supply and demand, (d) conditional/monetary and nonmonetary/supply, (e) conditional/nonmonetary/demand, (f) conditional/nonmonetary/supply, (g) other 6. Top-down accountability: (a) audits; (b) guidelines, recommendations; (c) inspections, (d) performance assessment and feedback; (e) supervision; (f) other

38 Gender-specific treatment IE has gender-specific treatment (measures the effect of gender)

39 Gender analysis IE has stratification/power for gender-specific analysis (impact on gender)

Section 3: Data Collection

40 Baseline End Date If baseline round has been planned or conducted

41 Follow-up 1 End Date If follow-up round 1 has been planned or conducted



Section 4: IE Monitoring and Quality Indicators

44 IE Has Ethical Clearance/ IRB Review

IE has obtained ethical clearance through an IRB

45 IE Has a Study Registry Whether IE has a study registry

46 Whether IE Plans to Register If IE is not registered yet

47 Registry Date IE registry date

48 Registry Site This includes AEA RCT Registry, RIDIE 3ie Registry, WHO ICTRP

Section 5: DIME Involvement

49 Fund-raising DIME has prepared funding proposals, obtained funding from Country teams or TF/EFO other, or funded part or whole of the IE from general funds

50 TTL-ing* The IE TTL is a member of DIME


– 82 –

— Continued

Section 5: DIME Involvement


51 Securing Client Engagement DIME responded to demand, had at least a discussion with the team, and agreed on way forward

52 Identifying Evaluation Questions*

DIME prepared draft concept (evaluation/research questions agreed)

53 Defining Identification and Data Generation Strategy*

DIME participated in workshop or clinic and/or prepared draft concept (identification and data agreed)

54 Assembling Research Team* DIME coordinated selection of research team

55 Supporting Implementation** DIME has at least one of the following making sure that the IE moves as planned: a TTL, a member of the research team (RT), a field coordinator (FC)

56 Collecting Data** DIME is doing at least one of the following: selecting contracting firm, advising on the selection, preparing instruments (questionnaire/sampling), training enumerators, supervising data collection, providing technology

57 Analyzing Results*** DIME has at least one staff in the research team

58 Disseminating Results— Client Level***

DIME has done one of the following: discussed results with client, organized country presentation or workshop, requested country clearance

59 Disseminating Results— Global Level***

DIME has done at least one of the following: presented results in a DIME workshop, DIME seminar, or any international venue, and/or published the report or paper online

60 IE Participated in DIME/SIEF IE Workshop or Other Initiating IE Workshop

This project was part of an initial multisector/multicountry IE workshop

61 Details of DIME/SIEF IE Workshop or Other Initiating IE Workshop

Details of IE workshop (e.g., title, month and year of workshop)

62 Client Participated in Additional Non-Initiating IE Workshop(s) Organized by Either the IE Team Or Other Stakeholders*

Any IE workshop(s) to connect to a broader global network of practitioners, policy makers, and/or experts in a set of relevant areas to the IE (not including the initial DIME workshop if the IE was part of one)

63 Details of Additional Non-Initiating IE Workshop(s)*

Details of additional IE workshop (e.g., title, month and year of workshop)

Section 6: Counterpart Details and IE Influence on Program/Policy*

64 Main Counterpart Implementing This IE

This includes the: 1. Central government 2. Local government 3. Other government organization (e.g., utility company, financial institution) 4. Nongovernmental organization 5. Private sector 6. Donor 7. Multilateral organization 8. Other


– 83 –

— Continued


65 Direct Counterpart Details Details (e.g., position, institution) of direct counterpart (person IE team interacts with on a day-to-day basis) that can best answer policy questions related to the IE

66 Program Manager* Details (e.g., position, institution) of the person in charge of the intervention that can best answer policy questions related to the IE

67 Mid-level Policy Maker* Details of the mid-level policy maker (the highest-level functionary who is in charge of policy decisions and who is not part of the intervention implementing team) that can best answer policy questions related to the IE, if the counterpart is the government

68 High-quality Baseline Survey** IE provided a full high-quality baseline survey (including covariates, with sufficient sample size, and representative of policy affected population) available to policy makers and researchers creating or building on instruments for policymaking before the IE.

69 Intervention Take-up Rate at Baseline**

Amount from the originally selected intervention group that actually used/took up treatment when baseline data was collected, if available and applicable

70 High-quality Follow-up Survey(s)* IE included a panel or comparable follow-up data

71 Intervention Take-up Rate at Last Follow-up*

Amount from the originally selected intervention group that actually used/took up treatment when follow-up data was collected, if available and applicable

72 Attrition Rate at the Last Follow-up*

Proportion from the original group that could not be interviewed for the follow-up because they were not found

73 IE Actions Led to Improved Counterpart M&E*

IE data requirements led to improvements in monitoring and evaluation (M&E) data collection and/or reporting activities of the counterpart(s)

74 Details on Specific M&E Data Collection and/or Reporting Activities of the Counterpart(s)*

This includes digitizing information, introducing new indicators, designing a new electronic system to capture school quality data

75 IE Team Provided Training for Data Analysis**

IE team delivered skills training of local institutions and/or staff for general monitoring and other data analysis independently (through discussions, technical assistance, workshops, and other training channels)

76 Details on Training** This includes training topic, month and year

77 Rationalized Policy Design IE improved design based on clear understanding of the underlying theory of change (causal links between the intervention components and the outcomes) and highlighted areas of uncertainty and critical assumptions

78 Details on How IE Rationalized Policy Design

For example, the IE helped the [matching grant project] develop better communication and technical assistance strategies to address [common take-up/participation issues]

79 IE Baseline Results Discussed with Clients**

If baseline results were discussed with client

80 Details on How Baseline Results were Discussed**

For example, the IE team held a [workshop, meeting, VC] to discuss baseline results on [date]


– 84 –

Section 6: Counterpart Details and IE Influence on Program/Policy*


81 Baseline Informed Policy Design/Implementation**

IE baseline data was used by governments and other stakeholders to stimulate policy dialogue and/or help identify problems and solutions

82 Details on How Baseline Informed Policy Design/Implementation**

For example, the IE identified [incentive problems in the size of women groups in Ethiopia] and helped project [introduce specialization in group composition].

83 IE Final Results Were Discussed with Clients***

IE analysis and results were discussed with the client to understand their policy relevance and application. Please specify the client, how it was discussed with the client, and the location/event of the discussion.

84 Details on How IE Final Results Were Discussed***

For example, the IE team held a [workshop, meeting, VC] to discuss IE results on [date].

85 Adopted Causal Mechanism(s) Based on IE Results***

IE evidence from experimental testing of alternative mechanisms was used by governments or other stakeholders to determine most effective program alternatives or to inform policy decisions.

86 Details on Contribution from an IE Mechanism***

For example, the IE identified more efficient [supply chain system] and government agreed to scale up at [national level in Zambia].

87 IE Results Used to Motivate Scale-up/Scale-down of Policy at National Level***

IE results reported successful (or insufficient) impact of the intervention in achieving desired outcomes and were used by governments and/or other agencies/stakeholders to motivate scale-up (scale-down) of policy at national level.

88 Details on Contribution from IE Mechanism to Scale-up/ Scale-down***

For example, the IE tested the [financial literacy program] and the [government] decided to scale-up/scale-down at the [national level].

89 Number of Presentations to Policy Makers of IE Results***

The number of presentations given to policy makers on the IE results

90 Number of Presentations to Academics of IE Results***

The number of presentations given to academics on the IE results

Section 7: Documentation*

91 Concept Note* Link to concept note

92 Baseline Report** Link to baseline report, if applicable

93 Policy Brief** Link to policy brief, if applicable

94 Paper/Final Report* Link to final report, if applicable

95 Working Paper** Link to working paper, if applicable

96 Publication** Links to publications, if applicable

97 Data in the Microdata Catalog* Link to Microdata Catalog, if applicable

* Indicator applicable only to IEs at stages after CN review.** Indicator applicable only to IEs at stages after baseline results are available for discussion

(or after implementation starts if the IE has no baseline).*** Indicator applicable only to IEs at the stage after IE results are available for discussion.


– 85 –

f. iE caSE StuDiES

Case Studies: Client Engagement and Policy InfluenceThe following subset of case studies of DIME IEs highlight the collaborative nature of the work and note that policy influence through client engagement lies at the heart of DIME’s work.

Case Study 1: brazil Financial literacy

This was a collaborative effort between the World Bank, the government of Brazil, and private sector stakeholders to conduct a large-scale rigorous evaluation of a school-based financial education program to motivate a change in national education policy.

In 2009, the government of Brazil (GoB) approached the World Bank with a request to provide a credible and independent evaluation of their proposed financial education strategy, the Estratégia Nacional de Educação Financeira (ENEF). The goals of this national strategy were to develop personal finance skills, stimulate savings, and broaden the use of financial products among Brazilians. In a country with a savings rate of 17 percent of GDP and where 82 percent of households that borrow money are not aware of the interest they are being charged, there was an acute need for a financial literacy program to improve consumer protection and strengthen the micro-foundations of economic growth through improved savings behavior. However, existing evidence on financial literacy programs around the world pointed to very little (if any) impact on program recipients. Thus, before the government could move from a national policy (ENEF) to

a tangible national program there was a need to evaluate the flagship school-based financial literacy curriculum and provide an evidence base for any national scale-up plans.

Client Engagement. The World Bank’s involvement through Development Impact Evaluation (DIME) catalyzed a collaborative effort to develop a more ambitious agenda. Together with the Ministry of Education, six state secretaries, the financial regulatory board (Comissão de Valores Mobiliários), the central bank, the Brazilian stock market (BM&FBOVESPA) and other financial regulators, the pilot was expanded to become the first and only randomized evaluation of a financial education program in schools, encompassing nearly 26,000 students in 900 schools across the Brazilian states of São Paulo, Rio de Janeiro, Ceará, Tocantins, and Minas Gerais, as well as the Distrito Federal.

Program Design. The evaluation assessed the impact of a pilot that introduced new books, teacher training, and school incentives to deliver a comprehensive high-quality financial education and behavior formation intervention for nearly 26,000 students attending public high schools across six states. The intervention was comprehensive and intense. The program was integrated in classroom curricula of mathematics, science, history, and Portuguese during 17

The following subset of case studies of DIME IEs highlight the collaborative

nature of the work and note that policy influence through client engagement lies

at the heart of DIME’s work.]

http://intresources.worldbank.org/INTDEVIMPEVAINI/Resources/3998199-1372170088287/brazil_policy-note.pdf

http://intresources.worldbank.org/INTDEVIMPEVAINI/Resources/3998199-1372170088287/brazil_policy-note.pdf


– 86 –

months. The curricula were complemented by teacher training, Web learning tools, and instructor handbooks. The baseline data were collected in August 2010, the midline in December 2010, and the end line in December 2011.

Results. The main results showed that the program was highly effective at improving financial knowledge (the whole distribution shifted to the right with a lower bound average effect of 5 percent), attitudes (increase in intention to save of 4 percent), and behavior (save for purchases, compare prices, negotiate prices, and make monthly budget). Because many take-home problem sets involved interaction with parents, the impact of the program had a trickle-up effect on parents’ financial knowledge, savings, and spending behavior. To better understand the mechanisms underlying the main results, the evaluation looked at students’ financial behavior, and estimates show positive impacts on outcomes related to students’ time preference (measured by a financial autonomy index) and self-control (measure by an index of intention to save).

This impact evaluation showed that, despite students’ socioeconomic background and the quality of schools, an intensive and curricula-integrated financial literacy program can impact students’ financial outcomes with potential long-run implications on individuals’ decision making.

Policy Influence. The results were presented at various rounds of national dissemination events (June 2011 and June 2012), covered by press conferences and interviews in all major national newspapers (and on radio and TV), and reported in financial blogs, which received 2.8 million Web hits. Based on these findings, The Ministry of Education decided to scale up the program nationally, seeking to improve students’ long-run outcomes through changes in their financial behavior and knowledge. The scale-up is currently under implementation.

Case Study 2: Taking Pilot Evaluations to Scale in Tanzania

This was a collaborative effort between the World Bank and the government of Tanzania to test the effectiveness of sanitation interventions when implemented at scale.

Poor sanitation is associated with higher rates of fecal-borne illnesses (diarrhea, tropical enteropathy, and worm infections) and a general decline in human health and well-being. Children can be at a higher risk for these illnesses, leading to malnourishment, stunting, and death.

Handwashing has been proven to be an effective way to prevent the spread of illness and infection and promote sanitation. Extensive evidence, including three systematic reviews over the past 10 years covering over 30 studies, has shown that handwashing with soap campaigns reduce diarrhea by 39–47 percent. However, the majority of the evidence, until now, has come from small-scale efficacy trials focusing on proof of concept. Little evidence exists on how effective campaigns can be at changing behavior and improving health results as part of a large-scale government program.

The government of Tanzania requested support from the World Bank and the Water and Sanitation Program in implementing two campaigns to promote access to proper sanitation and encourage handwashing with soap (HWWS) as part of the Global Scaling Up Rural Sanitation and Handwashing Project. The campaigns were imbedded into the program rollout to be able to measure the impact of each campaign individually and in combination to test explicitly for potential synergies between the interventions.

Intervention. The two campaigns targeted 150,000 rural households across 10 districts in

http://siteresources.worldbank.org/INTDEVIMPEVAINI/Resources/TZHandwashing.pdf

http://siteresources.worldbank.org/INTDEVIMPEVAINI/Resources/TZHandwashing.pdf


– 87 –

Tanzania. The campaigns focused on changing behaviors without the use of subsidies or direct incentives.

Handwashing with Soap (HWWS) Program. Community members were enlisted as frontline activators to educate heads of households and caregivers about proper handwashing behavior. Outreach also came through road shows, radio programs, and other mass media campaigns.

Total Sanitation and Sanitation Marketing (TSSM) Program. Community-led total sanitation triggering events together with roadshows, print materials and a radio soap opera were used to spur household demand to construct improved latrines and stop open defecation. Local masons, working with local supply chains that provided materials, were also trained to build and market latrine sanitation platforms to contain feces.

Evaluation Methodology. The impact evaluation was designed as a randomized controlled trial, randomly assigning 181 rural wards in 10 districts across the country to receive the HWWS program, the TSSM program, or both programs simultaneously (HWWS+TSSM).

A fourth group was composed of “control” communities to compare what would have happened in the absence of the program. Approximately 3,600 households were included in the sample, and the objective was to track outcomes across the entire causal chain of the programs from improvements in hygiene knowledge, to increases in latrine construction and handwashing at critical times, to reductions in open defecation, to improved cleanliness, and ultimately healthier children under the age of five.

Key Findings. Priority health outcomes, including diarrhea, stunting, wasting, and anemia, were unchanged in the TSSM and HWWS wards. While diarrhea decreased in the combined HWWS+TSSM wards, we find increases

in wasting and anemia, suggesting no clear improvements in health outcomes across all of the groups. Why might this be the case?

The intervention improved intermediate outcomes, but not by enough to have a meaningful effect on health outcomes:

♦ Just 11 percent of the households in TSSM communities reported “usually” defecating in the open, compared to 23 percent in the control communities—a 52 percent relative reduction. However, almost half of the population continued to practice some open defecation, which did not improve with the program.

♦ Improved latrine coverage increased from 50 to 65 percent. This was driven mostly by households choosing to build their own private latrines, rather than sharing with others.

♦ The HWWS program generated modest knowledge improvements but did not change handwashing behavior in a meaningful way. After contact with feces, fewer than 13 percent of the households were observed washing their hands, with no difference between the treatment and control communities.

Policy Lessons. When moving to scale, results from small-scale evaluations can help inform expectations and priorities, but operational challenges can drive a wedge between intended and actual impact.

Large-scale sanitation campaigns can be effective in promoting latrine construction and reducing open defecation, whereas promoting behavior change in handwashing seems to be more challenging.

In both the sanitation and handwashing campaigns, increases in intermediate outcomes, while significant, are not enough to generate the lasting health impacts that the program aims to achieve. Both depth (level of intensity of the


– 88 –

intervention) and breadth (complementary interventions to improve environmental conditions and reduce fecal-oral transmission) should be considered in the design of future programs if hygiene and sanitation campaigns expect to affect child health outcomes.

Case Study 3: Minding the gender gap in Mozambique: Identifying gender-Specific barriers and Integrating gender into Development Operations through Impact Evaluation

This was a collaborative effort between the World Bank and the government of Mozambique to examine the role of gender in a program that trained “contact farmers” to provide information to their peers on sustainable land management practices.

Context. Innovation in agriculture is crucial to meet the food requirements of Africa’s growing population. Although agricultural intensification has typically been the standard approach to increase yields, there has also been a recent trend to emphasize sustainable land management (SLM) practices. SLM can offer higher yields, more efficient water use, increased resilience to climate variability, and enhanced soil fertility. However, SLM outreach tends to overlook women, despite their major role in the agricultural workforce.

In many settings, married women cultivate plots separate from those of other family members. They face different challenges to productivity, such as deficiencies in inputs and weak property rights. They also have low access to extension services. Extension is the traditional vehicle of technological diffusion, yet it often caters to men. Social stigma can decrease information supplied to women while also decreasing their demand for information. In

societies with strict norms on gender roles and male-female social interactions, women may be more effective interlocutors to other women.

Intervention. In 2007, the government of Mozambique requested support from the World Bank and invested heavily in the agricultural extension network in five districts in the Zambezi Valley, an area with high agricultural potential. The government increased the number of extension agents per district and provided them with housing and training. In 2010 and 2012, agents received technical training on eight SLM practices (mulching, crop rotation, strip tillage, micro-catchments, contour farming, row planting, improved fallowing, and intercropping).

The training lasted three days, split evenly between in-class lectures and field demonstrations. Male messengers, also known as contact farmers, served as points of contacts between extension agents and community members. This model assumes that information flows efficiently from (i) extension agents to contact farmers, and (ii) contact farmers to other farmers. The intervention was designed to test these hypotheses: in treatment communities, contact farmers were directly trained on the SLM practices, following the same training as the extension agents. The gender aspect of communication was explored, too, because women contact farmers also received the training in a subset of the treatment group.

Evaluation Methodology. A random subset of existing male contact farmers and new female contact farmers was selected to receive a centralized training on SLM practices, identical to that received by extension agents. The intervention was rolled out in five Mozambican districts within the Zambezi Valley, and the training was delivered ahead of planting in both 2010 and 2012.

The treatment group consisted of 150 randomly assigned communities out of 200 eligible; the remaining 50 served as experimental control

http://siteresources.worldbank.org/INTDEVIMPEVAINI/Resources/MZ.Note.pdf







– 89 –

group. The training intervention equipped male messengers to teach new techniques. Even if it improves information dissemination, the information may not reach females. To better understand whether this was indeed the case, a second experimental treatment randomly selected 75 communities within the first treatment group to have a second, female messenger trained with a specific mandate to teach women SLM.

Key Findings. Results were tallied around four indicators: information dissemination, awareness of the existence of micro-catchments as an SLM technique, knowledge of how micro-catchments work and adoption of micro-catchments.

♦ Finding 1: The presence of a trained male messenger does not significantly increase information dissemination to female farmers.

♦ Finding 2: The awareness and adoption indicators for male farmers, which show 10.6 and 6.4 percent point increases, respectively, suggest that female farmers suffer from a gender bias in information dissemination by trained male messengers.

♦ Finding 3: Placing women in extension positions may help other women overcome barriers to adoption posed by inequitable access to agricultural services or exposure to inapt information. Evidence suggests that female leaders may promote policies prioritized by women and cause women to become more engaged citizens. Female messengers may not only improve communication to women but also better meet their informational needs.

Policy Lessons. Training and placing women in extension delivery positions in agriculture benefits the broader population of women in terms of technology awareness and adoption. Diffusing agricultural information through traditional, male-dominated extension networks can perpetuate gender inequities in agriculture.

Case Study 4: The Evolution of Programs Designed to Increase utilization of Skilled birth Attendance in Nigeria: Executive Summary

This executive summary is from an externally written case study, available here, which describes the evolution of maternal and child health policy making in Nigeria, leading to an ongoing partnership between the World Bank, the government of Nigeria, and partners, including the Bill & Melinda Gates Foundation, to use impact evaluation across a series of programs designed to improve maternal and child health.

Some 33,000 mothers and 240,000 newborns die in Nigeria every year. The vast majority of these deaths are preventable, and the interventions that could prevent these deaths are known. Among the most promising of these interventions is utilization of skilled birth attendants (SBAs), a service known to dramatically reduce both maternal and neonatal mortality.

This case study explores recent changes made to programs intended to save the lives of mothers and babies in Nigeria. It focuses on SBA utilization programs and complementary maternal and child health (MCH) interventions. It argues that these changes could represent a transformational shift in the way MCH programs are delivered through Nigeria’s public health delivery system.

The objective of this case study is to learn how and why efforts to increase SBA utilization and complementary MCH services in Nigeria have changed over the past several years. It explores the extent to which evidence, experience, and learning have informed recent improvements in the design of MCH programs. The case study covers ongoing work that will continue to evolve over the next several years. The World Bank’s Knowledge and Learning Group selected the case of Nigeria because

http://intresources.worldbank.org/INTDEVIMPEVAINI/Resources/3998199-1372170088287/nigeria_policy-note.pdf





https://www.dropbox.com/s/byr8qlsl3uitvwm/SoD%20Case%20Study-Nigeria.pdf?dl=0


– 90 –

its evidence-based interventions focused on SBA utilization, and other MCH interventions are both promising and occurring at an unprecedented scale.

The story behind this case is still unfolding: impact evaluations of the two ongoing programs that include significant SBA utilization components are currently under way. IEs rely on statistically valid approaches, and they are based on counterfactual analyses that can attribute outcomes to specific interventions. At the request of the government of Nigeria, IEs of several health care programs are under way in full collaboration with the Nigerian officials overseeing these programs. The World Bank and other international organizations provide technical and financial support of these IEs. Results from the IEs evaluating SBA utilization and complementary MCH programs will be available over the next several years, and IE findings at each stage of program implementation will inform the next stages of implementation. As such, this case study is both retrospective, telling the story up to now, and prospective, pointing to a next phase of implementation, learning, and adaptation.

This case explores two research questions:

1. What changes have been made to programs intended to save the lives of mothers and babies in Nigeria, from 1999 to the present?

2. How have the newer programs intended to save the lives of mothers and babies in Nigeria drawn from learning and adaptation from research evidence and earlier experiences?

1. What changes have been made to programs intended to save the lives of mothers and babies in Nigeria, from 1999 to the present?From Nigeria’s establishment of democracy in 1999 until 2008, the national health care paradigm focused on building a primary health care system from the bottom up. This paradigm relied on a “one size fits all” approach,

regardless of differences in need or economic or cultural context. This overly uniform approach was one of several challenges that hindered effective health care delivery during this period, called “the old paradigm” in this case study. Primary health care programs implemented during this time did not produce major improvements in MCH indicators, including SBA utilization. Nigeria’s progress in these areas fell short of regional and global averages, and by 2008 Nigeria was not on track to meet the Millennium Development Goals aiming to Reduce Child Mortality (MDG 4) or to Improve Maternal Health (MDG 5).

In 2009, a new health care paradigm began to emerge under new leadership at the Federal Ministry of Health (FMoH). This included introduction of the Midwives Service Scheme (MSS), an evidenced-based program focused on SBA utilization, which has a known capacity to significantly improve key MCH indicators. SBA utilization increased by 150 percent and maternal mortality fell by 47 percent at MSS facilities in the first three years of the program’s implementation.21 Nevertheless, the MSS revealed many opportunities to learn about and improve the design and delivery of effective SBA utilization programs.

Although programs designed to increase SBA coverage have evolved considerably, this story of improved program delivery will continue to unfold over the next several years, while new evidence about what works emerges and programs are improved accordingly.

2. How have the newer programs intended to save the lives of mothers and babies in Nigeria drawn from learning and adaptation from

21 FMoH 2013 “Expanding Access” presentation by Dr. Muhammad Ali Pate, May 2013.


– 91 –

research evidence and earlier experiences? By 2008, the government of Nigeria and its supporters from the international development, philanthropic, and civil society communities recognized that their collective efforts to improve national health care were largely ineffective. Many challenges needed to be addressed, including the “one size fits all” approach to primary health care delivery; the lack of coordination between federal, state, and local agencies; problems with funding and resources reaching their intended recipients; and missed opportunities to produce evidence of program effectiveness. Experts interviewed for this case study, including FMoH officials working on program design and implementation during these years, agree that many lessons could have been learned from these challenges to more quickly and effectively adapt and improve the health sector.

Three programs have launched since the old paradigm’s MSS program: the Nigeria State Health Investment Project (NSHIP), the Subsidy Reinvestment and Empowerment Programme MCH effort, and Saving One Million Lives. These new programs use both experience and additional evidence to support program design.

Midwives Service Scheme (MSS)MSS was designed by technical experts, using lessons learned in other countries. The Afghanistan community midwife program, which was associated with a rapid increase in the number of deliveries by SBAs in that country, was an important source of evidence and experience.

Subsidy Reinvestment and Empowerment Programme MCH (SURE-P MCH)The SURE-P MCH program design was based on MSS experience and evidence from other studies, and then reinforced by a pilot implementation

phase conducted to inform aspects of the program prior to scale-up. The program also includes a series of built-in IEs designed and implemented with support from the World Bank’s DIME group. The DIME IE approach is to actively become part of the policy/program process starting at the design stage. This allows its impact evaluations to enhance program strength, promote critical thinking, and provide interim feedback throughout program implementation, and not solely following the program’s completion. This formative role epitomizes an evidence-based approach to program and policy design.

Saving One Million LivesFinally, Saving One Million Lives is a sector-wide approach to improve health outcomes. It utilizes “proven, evidence-based, cost-effective interventions that address the leading causes of morbidity and mortality.”

Nigerian public health officials have demonstrated a strong commitment to learning from experience and to utilizing evidence to inform public health program design and delivery. This is evident overall, and demonstrable in each of the ongoing programs described herein.

The current programs intended to increase SBA utilization and complementary MCH services may have some gaps, such as the level of attention paid to private health care facilities and traditional birth attendants. However, over the past several years, there have been significant changes in Nigeria’s approach to delivering critical services for pregnant women and babies. This shift to an evidence-based, iterative learning approach to improving delivery is real, and it has taken hold across Nigeria in an impressive way. It is too soon to tell what impact these changes will have on MCH outcomes, but the advances thus far are compelling and bode well for producing significant improvements in MCH outcomes in the next few years.



– 93 –

G. i2i SuPPortED imPact EvaluationS

IE Subtheme Country Region IE Title IE Status IE Objective

Public sector governance

Colombia LAC Colombia Justice Services Mobile Units

Concept Note The impact of delivering legal services through the mobile units in Colombia on demand and quality of legal services.


Congo, Dem. Rep.

AFR Attracting and Deploying Talent to Reform Civil Service in the Democratic Republic of Congo

Concept Note How best to improve the ability of public administration to recruit qualified public servants? What are the effects of intrinsic and extrinsic incentives on the types of candidates as well as on their subsequent performance?

Jobs for at-risk youth as a tool for resilience

Congo, Dem. Rep.

AFR Productive Opportunities for Stabilization and Recovery

Concept Note What are the differential effects of Labor Intensive Public Works (LIPWs) on socioeconomic and psychological welfare of unemployed workers? What are the differential effects of different selection/targeting mechanisms? Does participation in LIPWs increase the likelihood of long-term employability? What is the marginal effect of vocational skills training and training in life skills?


Guinea AFR Fighting Irregularities in HR: Public Sector Reform in Guinea

Concept Note What is the effect of strengthening internal and/or external establishment controls on reducing irregularities, including in recruitment, transfer, promotion and retirement?


Honduras LAC Moving Youth Out of the Market for Crime: Interventions in the Honduras Safer Municipalities Project

Concept Note How does the effectiveness of a job opportunity and life-skills training/mentoring program vary by the age of the participant?

Gender-based violence

India SAR Testing a Norm-changing Approach to Domestic Violence

Concept Note Does the norm that tolerates high levels of violence against women in intra-household conflicts in India, can be changed to reduce the tolerance for violence?


Kosovo ECA Kosovo Youth Development Program Impact

Concept Note What are the most effective strategies to increase participation of disadvantaged young men and women from minority groups in youth development programs?


Liberia AFR Measuring the Effect of Extrinsic and Intrinsic Factors in Improving Performance in the Public Sector

Concept Note What is the role of different sources of motivation for public service and how can they be enhanced by extrinsic and intrinsic incentives?

A. NEW IEs IN FRAGILITY, CONFLICT, AND VIOLENCE

— Continued


– 94 –



Liberia AFR Youth, Employment, Skills Project Additional Financing / Community Livelihoods Evaluation

Concept Note What are the (net) effects of short-term, LIPWs on income/savings and investment in self-employment/enterprise development? Does participation in LIPWs increase the likelihood of long-term employability?


Sierra Leone AFR Mining for Development: Improving Concessions Agreements in Sierra Leone

Concept Note Can increasing the bargaining power of local communities through information, collective action, and third-party mediation (i) foster concessions contracts that provide better terms for communities and (ii) increase the extent to which companies honor such contracts and existing national regulations?

Urban crime and violence

Colombia LAC Effects of Park Lights on the Incidence of Crime in Cali, Colombia

Concept Note Does installing nighttime illumination in Cali’s city parks lead to (i) reduced daytime and nighttime crime rates and victimization in parks and surrounding neighborhoods, (ii) increased pedestrian and pro-social organization use of parks, and (iii) improved resident perceptions of safety in the park and surrounding neighborhood?


Tunisia MENA Tunisia Community Works and Local Participation (JSDF) Impact evaluation

Concept Note What are the effects of LIPWs on socioeconomic welfare and psychological well-being of long-term unemployed workers? What are the spillover effects of LIPWs on recipient communities? How best to attract vulnerable groups (e.g., youth at risk; women) in community public works projects?

b. NEW IEs IN AGRICULTURE ADAPTATIONS AND NATURAL RESOURCE MANAGEMENT


Rural infrastructure

Rwanda AFR Impacts and Sustainability of Irrigation

Concept Note Provide estimates of the impacts of irrigation on yields, profits, labor markets, rental and land markets, migration, and education. Test whether those impacts are greater when farmers have expanded access to credit. Explore mechanisms designed to address maintenance and overuse.

Commercializa-tion

Brazil LAC Identifying Barriers to Investment in rural Brazil: Testing Information on Opportunities and Matching Grants

Concept Note Identify the relative impacts of matching grants and identification of profitable business opportunities on smallholders’ competitiveness, access to market, and livelihood.


Rwanda AFR Impact Evaluation: Rwanda Feeder Roads Development Project

Concept Note Track the impacts of improvements in rural feeder roads on market prices, volume and land values.

A. NEW IEs IN FRAGILITY, CONFLICT, AND VIOLENCE

— Continued


– 95 –



Mozambique AFR IE of Water Governance in Irrigation

Concept Note Measure the overall impact of availability of irrigation on overall yield and seasonality of production and test innovations in monitoring to induce collective action for water and infrastructure management.

Financial constraints

Georgia ECA Impact of Land Registration on Land Values, Investment, and Factor Market Performance

Concept Note Document the effect of formal registration of property rights for land on perceived tenure security, land values, land use, and land market participation.


Kenya AFR Increasing the Returns to Irrigation Investments for Smallholder Farmers in Kenya

Concept Note Capture the impact of irrigation on production and test combinations of microcredit and training in improving the returns to irrigation.


Nepal SAR Complementarities of Irrigation and Extension Services

Concept Note Identify complementarities between extension services and access to irrigation in boosting adoption of new technologies.

Natural resource management

Burundi AFR Sustainable Coffee Landscape Project Impact Evaluation

Concept Note Measure the impact of information and subsidies for shade-grown coffee production on adoption of the new technology and quantity and quality of coffee production.

Commercializa-tion

Liberia AFR Smallholder Agricultural Productivity Enhancement and Commercialization

Concept Note Measure the impact of contract farming in reducing coordination failures between production and markets and influencing adoption of technology.

Commercializa-tion

Tanzania AFR Impact Evaluation of the Second Agricultural Sector Development Project (ASDP II)

Concept Note Document the impact of private management of irrigation schemes in building rice and maize value chains.


Mozambique AFR If You Pay, You Gain: Testing a Reward Incentive Scheme to Overcome the Problem of Free Rider in Local Management

Concept Note Test the impact of a performance-based incentive scheme in achieving higher levels of collective action in irrigation management, and isolate the impact of access to irrigation on smallholder agriculture.

Financial constraints

Uganda AFR The Impact of eVouchers and Extension on Input Uptake

Concept Note Measure the impact of innovative modalities of extension delivery and of a declining subsidy on chemical fertilizer on short-term and long-term adoption.

C. NEW IEs IN ENERGY AND ENVIRONMENT


Energy access Senegal AFR Senegal Behavior Change Campaign and Solar Lights Evaluation

Concept Note Examine impact of mass media behavior change campaign on behavior change measured among others by purchase of pico-PV products and learn about benefits of pico-PV use.

— Continued


– 96 –


Energy supply Nepal SAR Benefits of Drip Irrigation

Concept Note Examine patterns of demand for dedicated distribution feeders and the impact of eliminating electricity cutoffs (load shedding) on productivity, profitability, and labor demand in the industrial sector.

Energy access Kenya AFR The Demand for, Costs, and Socioeconomic Impacts of Rural Electrification in Kenya

Concept Note Generate new data on the demand for, costs, and social and economic impacts of national grid electrification in Kenya.

Energy access/ energy efficiency

Argentina LAC Argentina Solar Home Systems and Tariff Impact Evaluation

Concept Note Fill evidence gap on welfare impacts of solar home systems, ascertain the possibility to magnify these impacts through additional salience interventions, and examine solar home system demand patterns.

Energy supply/ energy efficiency

Bangladesh SAR The Socioeconomic Impact of Improved Electricity

Concept Note Measure the impacts of improvements to the electricity distribution network on quality and reliability of electricity that reaches consumers, on investments by households and firms, and the adoption of climate-smart practices by reducing electricity use during peak hours and efficiency of energy use.

Energy Access Bangladesh SAR Expanding Access to Sustainable Energy in Remote Areas of Bangladesh: Green Mini-grids under the Rural Electrification and Renewable Energy (RERED) Project

Concept Note Explore the impact of efforts to meet a substantial fraction of power demand using power generated from renewable energy sources through the development of green mini-grids in rural areas and remote islands that are unlikely to be connected to the grid in the foreseeable future.

Supply/access/ energy efficiency

Kenya, Tanzania

AFR Green Mini-Grids Africa

Concept Note Explore cost-effective ways to attract the initial investments required to build green mini-grids via subsidies and information about relevant site characteristics. Examines measures that will sustain the provision of affordable energy services such as group discounts for peripheral households and collection and payment methods.

Common-pool resources

India SAR Benefits of Drip Irrigation

Concept Note Provide rigorous evidence on the benefits of water-saving irrigation systems over traditional (i.e., flooding) methods.

C. NEW IEs IN ENERGY AND ENVIRONMENT

— Continued


– 97 –


Externalities Brazil LAC Role of Information and Demonstration in Adoption of Sustainable Land-management Practices in the Cerrado

Concept Note Measure impacts of subsidized credit line for adoption of conservation agriculture practices in combination with a randomly assigned menu of training, intensive technical assistance, and performance-based social inducements. Explore social network effects of information interventions.


China EAP Incentivizing Sustainable Livestock Practices in China’s Grasslands Regions

Concept Note Generate evidence on how to overcome overgrazing caused by livestock farmers and effectively shift herder behavior toward the social optimal and examine potential economic and environmental consequences which may be a result of this change.


Burkina Faso AFR Protecting Livelihoods and the Environment: Sustainable Forest Management in Burkina Faso

Concept Note Learning whether payments for ecosystem services are effective in the context of forest conservation and understanding why and under what circumstances specific payment schemes work.

Externalities Ghana AFR Evaluating Efficient Ways to Promote Sustainable Land and Water Management and Payments for Ecosystem Services in Ghana

Concept Note Generate knowledge on the most efficient and cost-effective means (including payments for ecosystem services) to induce adoption of sustainable land and water technologies; learn about the opportunity costs farmers face in providing ecosystem services.

D. ONGOING IEs ACROSS i2i PILLARS/THEME

Theme Country Region IE Title IE Status IE Objective

Shared prosperity

Afghanistan SAR Targeting the Ultra Poor Program Impact Evaluation

Baseline The impact of a comprehensive package of productive assets; skill formation; consumption support, and other complementary services on the long-term livelihoods of ultra-poor households in fragile situations.

Governance Angola AFR DIME IE of FAS Local Development Project

Baseline The impact of community-school scorecards on learning outcomes, school enrollment and attendance, teacher performance, and social capital dimensions.

Climate change

Bangladesh SAR Evaluation of Integrated Agricultural Productivity Project

Second follow-up

The impact of promotion of improved agricultural technologies through Farmer Field School (FFS) methodology on technology adoption, higher yield, and income for farmers.

— Continued


– 98 –


Shared prosperity

Benin AFR Impact Evaluation of the “Entreprenant Status” in Benin

First follow-up The impact of the introduction of the Entreprenant status on formalization rate, business performance (i.e., increased turnover, increased profits), business skills, accounting systems, level of trust, access to new markets, level of advertising, access to banking, number of tax payments, amount of taxes paid, investment, employment, and standards of living.

Climate change

Brazil LAC Ceara Rural Development and Access to Markets Project

Baseline The impact of technical assistance and matching grants for the adoption of new technologies and identification of new markets on organizational capacity, access to credit, sales value, and agricultural production.

Governance Burkina Faso AFR IE of Burkina Faso Local Government Support Project

Baseline The impact of disseminating municipal performance scorecards to municipal administrations on the performance ratings of municipal administrations on service delivery and municipal administrative functions.

Governance Burkina Faso AFR IE of Burkina Faso Local Government Support Project Individual Citizen Intervention

Baseline The impact of personal invitations for individual citizens to attend municipal council meetings on their willingness to participate in and to contribute to local governance; their knowledge about citizen rights to participate in local-level governance; and their trust in the municipal administration.

Shared Prosperity

Colombia LAC Impact Evaluation of Technology Extension Pilot

Intervention The impact of two competing interventions on improvements in management and in productivity in the Colombian auto-parts manufacturing sector: (i) individual consulting services support, and (ii) group consulting support on productivity, production levels, machine downtime, inventory levels, defect rates, worker hours, sales, and worker absenteeism rates.

FCS Côte d’Ivoire AFR Côte d’Ivoire Youth Employment and Productivity Impact Evaluation

Follow-up 2 The impact of labor-intensive public works and related graduation strategies and the provision of formal apprenticeships on employment, earnings, and risky behavior among youth.


— Continued


– 99 –


Shared prosperity

Dominican Republic

LAC Dominican Republic Female Biz and Worker Training

Intervention The impact of three financial literacy modules (Job Skills module; Budgeting, Savings, and Credit modules; and Entrepreneurship module) on management of household finances; savings; usage of credit; increased usage of formal sector savings and credit products; ability to search for, obtain, and retain formal employment; and management of small businesses.

Climate change

Haiti LAC Evaluation of Smart Subsidies for Farmers

Follow-up 1 The impact of providing subsidy vouchers for agricultural inputs on the following outcomes: higher yield and income for farmers.

Climate change

Haiti LAC Evaluation of Matching Grants for Agricultural Postharvest Organizations

Baseline The impact of matching grants provided to cooperatives and agricultural service providers for input provision and post-harvest processing, and business training for women on: increased production and income of farmers, especially women.

Climate change

India SAR Punjab: IE on Rural Water Supply and Sanitation

Intervention The impact of improved access to sanitation services on reducing the incidence of water-borne, water-washed, water-related and excreta-related diseases in rural Punjab.

Climate change

Indonesia EAP Aceh Community Ranger Program IE

Completed The impact of training at-risk youth, patrols, livelihood sub-grant projects and community outreach and awareness on environmental and social outcomes.

Gender Indonesia EAP Evaluating Early Childhood Education Services

Analysis The impact of a package for early childhood services on enrollment and duration of enrollment in various early childhood education services, early childhood developmental outcomes, and test scores in primary school.

Governance Jordan MENA Impact of Legal Aid on Poor Women

Baseline The impact of increasing public awareness of the law on improving individuals’ ability to access justice and resolve disputes; the impact of access to legal counseling on improving individuals’ justice outcomes and socioeconomic welfare.

Climate change

Kenya AFR DIME-WaSSIP Impact Evaluation Nairobi

Baseline The impact of financial subsidies (provided at different levels) on sanitation demand and the impact of an information campaign to promote hygiene on increasing sanitation investment with subsidies.

— Continued


– 100 –


Governance Kenya AFR Kenya Patient Safety Impact Evaluation

Baseline The impact of top-down and bottom-up accountability systems through inspections of private and public health facilities on patient safety and quality of care, prices and demand of health services.

Gender Kenya AFR IE of Video Screenings for Out-of-School Youth in Nyanza Province

Intervention The impact of sensitization cam-paigns on knowledge, attitudes, and behavior of HIV and gender-based violence for out-of-school youth.

Gender Kenya AFR IE of Life Skills for In-School Youth in Nyanza Province

Intervention

Shared prosperity

Kenya AFR Micro-franchising Nairobi IE

Follow-up 2 The impact of micro-franchising on income, skills, and self-efficacy outcomes for young women.

Governance Kenya AFR Impact Evaluation of the Kenya Judicial Performance Improvement Project

Baseline The impact of different kinds of court station feedback mechanisms as tools to improve efficiency in courts and encourage deputy regis-trars and heads-of-station to better address the case backlog and im-prove case processing times on case backlog and time to case disposition.

Shared prosperity

Macedonia, Albania, Serbia

ECA SEE Trade Facilitation Impact Evaluation

Gender Malawi AFR Protecting Early Childhood Development (PECD) in Malawi

Follow-up 2 The impact of intensive training and mentoring of teachers/caregivers at Community Based Child Care centers in rural Sub-Saharan Africa on young children’s physical, emotional, and cognitive development. How can cash incentives be used to retain teachers/caregivers and make them more effective?

Shared prosperity

Malawi AFR Fingerprinting to Reduce Risky Borrowing

Intervention The impact of improved identification in financial institutions on access to credit and repayment.

Shared prosperity

Mexico LAC Impact Evaluation of a Large-scale Female Entrepreneurship Pilot in Mexico

Intervention The impact of business training and personal initiative on key business outcomes, such as sales, profits, size, and probability of survival. What is the additional impact of access to finance on key business outcomes (sales, profits, size, and probability of survival)?

Climate change

Nepal SAR Evaluation of Agriculture and Food Security Project

Intervention The impact of agricultural technology adoption and dissemination of information on best-practice nutrition for children and pregnant women on improved agricultural productivity and improved nutritional practices for mothers and pregnant women.


— Continued


– 101 –


Shared prosperity

Nigeria AFR Impact Assessment of Financial Literacy

Analysis The impact of a financial education movie and the opportunity to open a microsavings account on perceptions, trust, intentions to save/borrow from formal institutions, and saving and borrowing behavior.

Climate change

Nigeria AFR Nigeria: Impact Evaluation – Rural Access and Mobility

Intervention The impact of community-based road upgrading and rehabilitation on transport conditions, access to services, agricultural commercialization, and household well-being.

Gender Nigeria AFR Shuga IE: Changing Norms and Behaviors

Intervention The impact of screenings of the TV drama Shuga and its effects on risky sexual behavior and gender-based violence of viewers and friends. What’s the impact of the screenings when combined with information of beliefs and values of peers?

Climate change

Nigeria AFR IE of the Nigeria Erosion and Watershed Management Project (NEWMAP)

Baseline The impact of infrastructure investments and community involvement to manage land erosion on land erosion levels, operation and maintenance of infrastructure investments, community engagement in construction and maintenance; and behavior change toward more environmentally responsible practices.

Governance Pakistan SAR Evaluation of PPAF Partner Organizations: NRSP and MRDO [Using Inclusion and Governance Mandates to Increase Accountability in Participatory Development Programs]

Follow-up 1 The impact of inclusion mandate and ratification in village-level grant management on type and location of public goods built; and attitudes and behavior of women as leaders.

Shared prosperity

Rwanda AFR DIME Rwanda Rural Finance

Follow-up 2 The impact of innovative financial products designed to help farmers save for agricultural inputs (targeted savings and commitment savings) on use of agricultural inputs, savings patterns, use of input credit, and household expenditures.

— Continued


– 102 –


Climate change

Rwanda AFR Impact of Users’ Feedback on the Demand for Extension Services

Follow-up 1 The impact of randomizing various feedback tools to client farmers of a fee-based extension service provider on: farmers’ attendance and general interactions with the extension services, farmers’ propensity to experience problems, farmers’ response to problems, perceptions of the service, knowledge score, and administrative take-up data on existing and new members.

Climate change

Rwanda AFR Impact of the Land Husbandry, Water Harvesting, and Hillside Irrigation Project

Follow-up 3 The impact of a large invest-ment in land husbandry, water harvesting, and hillside irrigation on hillsides in selected water-sheds in Rwanda on agricultural productivity, agricultural income, cropping pattern, agricultural commercialization, and use of improved technologies.

Climate change

Senegal AFR IE of Stormwater Management and Climate Change Adaptation

Baseline The impact of community-led micro-projects with a value of $10,000–$70,000 on urban cleanliness and flood prevention activities; health, income, consumption, school attendance, and damage to/loss of household property.

Governance Senegal AFR IE for Senegal Economic Governance Project

Follow-up 2 The impact of a reform aiming to shorten the length of civil and commercial pre-trial procedures in Senegal and a reminder pop-up window displaying a number of randomly chosen ongoing cases and their treatment duration to judges in the Dakar regional court on trial duration, split along steps in the legal chain; hearing-level decisions and actions; firm perception of the justice system; and firm outcomes.

Shared prosperity

South Sudan

AFR South Sudan Start-up Business Grant IE

Baseline The impact of unconditional cash transfer and business/life-skill training on human capacity and physical capital. What is the impact of a cash transfer paired with business/life-skill training on the life of a young person in South Sudan?


— Continued


– 103 –


Shared prosperity

Tajikistan ECA Impact Evaluation of E-filing and In-depth Study of Risk-based Audits in Tajikistan

Intervention The impact of adoption and impact of electronic tax filing on adoption, tax compliance costs, and perceptions of corruption in tax administration.

Governance Tanzania AFR IE Urban Local Government Strengthening

Governance Uganda AFR Uganda: Social Protection, Governance, and Fragility

Intervention The impact of strengthened social accountability and local community monitoring in a large-scale decentralized community-driven development (CDD) project on quality of sub-projects delivered by CDD project, procurement quality, fraud and leakage in sub-project delivered by CDD.

Climate change

Vietnam EAP Evaluation of Fisheries Co-management

Intervention The impact of formation of community-based coastal fish-eries co-management groups on longevity and adherence to co-management scheme and increase in uptake of sustain-able fisheries practices.

Climate change

Vietnam EAP Vietnam Results-based Rural Water Supply and Sanitation under the National Target Program: Impact Evaluation

Baseline The impact of newly deployed piped water system at varying connection fees on take-up and usage of piped water supply.

Shared prosperity

Vietnam EAP Vietnam Central Highlands Poverty Reduction Project: Impact Evaluation

Baseline The impact of a package of local infrastructure and livelihoods interventions designed to reduce poverty in the Central Highlands region of Vietnam, particularly among ethnic minorities, on identified development priorities, access to services, quality of local infrastructure, objective and subjective economic well-being, and participation of marginalized groups in decision making.

— Continued



– 105 –

H. Policy influEncE of WorlD banK imPact EvaluationS

In this appendix we look at the policy influence of World Bank impact evaluations by summarizing (i) the results of the econometric analysis of the effect of impact evaluation on World Bank project implementation; (ii) the policy influence monitoring framework for IE products; (iii) the results of the independent World Bank 2013 Client Feedback Instrument; and (iv) case studies of impact evaluation products, including engagement with local partners and how they have affected policy decisions.

Impact Evaluation Helps Deliver Development Projects Does research add value to aid? Specifically, does impact evaluation research help or hinder the deliv-ery of development projects?

background: Development Aid and World bank Impact Evaluation

With $16 trillion in aid delivered since the establishment of the Bretton-Woods institutions, development effectiveness has been at the forefront of the economic debate. Yet the knowledge gap on what really works in development remains large. The literature investigating the relationship between country growth and aid flows has for the most part failed to provide satisfactory answers to the question of whether aid is effective and if so under what conditions. This is in part because aid

measured in dollars and cents implicitly assumes that aid is a transfer of resources. Far from it, aid is delivered through development projects that bundle technical assistance with financial flows, the former affecting the effectiveness of the latter. Recent evidence22 finds that most of the variation in aid performance is explained by project characteristics. What project characteristics make aid effective is therefore a relevant question.

In 2005, the Bank initiated an institutional effort to conduct impact evaluation (IE) to generate knowledge in the context of national policies and programs in developing countries seen in figure H.1. This is also when the Bank started tracking the link between IE and its projects. Over the next few years, starting from a very low baseline, IE came

22 Denizer, Cevdet, Daniel Kaufmann, and Aart Kraay. 2013 “Good countries or good projects? Macro and micro correlates of World Bank project performance.” Journal of Development Economics 105:288 - 302.

FIguRE H.1: WB Impact Evaluations by Start Date

80

70

60

50

40

30

20

10

0

1976

198

2

198

7

199

0

199

2

199

4

199

6

199

8

200

0

200

2

200

4

200

6

200

8

2010

IE started of WB projects

IE started


– 106 –

to cover about 10 percent of the Bank’s projects and helped to establish, through counterfactual analysis, the causal pathways of which policies financed by these projects work and why. The Bank’s impact evaluations involve research and field staff working with project and government teams to introduce structured and iterative learning in project operations. They aim to inform policy design, identify policy-relevant research questions, run experiments, and collect and analyze data to feed evidence back into policy decisions during implementation. At their best, impact evaluations are a framework of collaboration for the generation and real-time use of policy-relevant evidence.

Investigating the Role of Impact Evaluation on Project Performance

This study investigates the role of these impact evaluations on the quality of projects. For this purpose, we assume that we can summarize the quality of projects by three conditions: (i) the problem/goal is correctly identified, (ii) an appropriate course of action and financing is agreed to, and (iii) project activities are implemented as planned. As it is hard to measure or find good proxies for (i) and (ii), we restrict our analysis to (iii), which is measurable and a necessary condition for projects to achieve their impact.

To do this, we construct a panel using administrative data on the entire portfolio of World Bank impact evaluations and projects approved during 2005–11. Linking information on projects’ disbursements with information on impact evaluation allows us to explore the relationship between IE and project implementation using 100 impact evaluations and 1,135 projects implemented in 114 countries in 11 sectors during 28 quarters.

We use project disbursements as a measure of

implementation because disbursements are made against evidence that the agreed activities are being implemented by the relevant government and supervised by Bank project staff. We use three specific quarterly measures: (i) cumulative disbursements relative to loan size, (ii) the gap between planned and actual disbursements relative to loan size, and (iii) the ratio of project to average disbursements for all projects in the relevant sector.

To control for project selection into IE, we include both institutional and idiosyncratic factors, assuming the confounders are observed or time-invariant. These factors range from time, sector and country effects, project and project leader fixed effects, and project leader changes. If project leaders of potentially high-disbursing projects are more likely to select into IE, then projects with IE may disburse faster partially due to the project leader decision to adopt an IE. If, on the other hand, project leaders select into IE because their projects need additional help and support, then projects with IE will disburse at a slower pace. Our data provide some support for the latter case.

Results: How Impact Evaluation Helps Projects get Down to business

We find that doing impact evaluation has large, positive, and significant effects on project implementation, especially when measured against plans. Moving from project design to implementation is an arduous process. During the period before midterm, or about two-thirds of the way into projects’ implementation, projects are delayed, in the average quarter having cumulatively disbursed at a rate that is only around half of what they had planned (15 instead of 29 percent of loan size). The estimates


– 107 –

for the effect of the IE, in this context, have high economic significance. To midterm, the effect size accounts for a two-fifths (40.8 percent) increase in average cumulative disbursements, significant at the 1 percent level. The planned to actual disbursements gap is reduced by one-half (54 percent, significant at the 1 percent level), an 8 percentage point effect that helps realign projects to their planned implementation path. When compared to the disbursement path of other projects in the same sector, the IE increases project disbursements by 40 percent, also significant at the 1 percent level. In short, doing IE seems to be a powerful tool to move projects from design to implementation.

When we do not restrict the sample at the midterm milestone, results carry through for disbursements relative to plan (45 percent, significant at the 1 percent level) and relative to

sector disbursements (49 percent, significant at the 1 percent level), although not for average cumulative disbursements. Average cumulative disbursements estimates are affected by the substantial amount of heterogeneity in duration that is introduced when the whole sample is used, while the heterogeneity is controlled when the same disbursements are benchmarked against project plans or sector averages. The effects, however, continue to be economically sizable.

Conclusion: Project Financing and IE Research are Complementary

We conclude that projects with IE are more likely to implement their activities as planned and, in so doing, may be more likely to achieve their objectives. Furthermore, our results support the idea that project financing and IE research are

TAblE H.1: Summary Table – Fixed Effects Estimates

Rate of Disbursement– Up to Mid-Term

Gap–Up to Mid-Term

Rate of Disbursement–

Adjusted by Sector

Average–Up to Mid-Term

Rate of Disbursement Gap

Rate of Disbursement–

Adjusted by Sector Average

T (quarters with IE=1)

TTL switch dummies

Polynomial of sector averages

0.060***(3.00)

Y

Y

-0.078***(-3.08)

Y

Y

0.40***(3.39)

Y

Y

0.038(1.54)

Y

Y

-0.067**(-2.47)

Y

Y

0.45***(3.15)

Y

Y

Observations

Outcome mean

Effect in %

7918

0.147

40.8%

7921

0.144

54.2%

7889

1.00

40.0%

17784

0.244

15.6%

17784

0.148

45.3%

17728

0.917

49.1

Note: Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle. Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent


– 108 –

complementary outputs of the World Bank. This has implications for the use of IE as an operational tool to improve the delivery of projects, and for weighing the costs of IE research against the benefits on the Bank’s project portfolio. For development organizations like the World Bank, an optimal balance between project financing and IE research may help them reach their efficient frontier. More generally, we hope our results contribute to shifting the discussion away from just questioning the effectiveness of aid to a

FIguRE H.2: Main Results (in percentage of outcome mean)

Main Results(in percentage of outcome mean)

40.8%*** 40.0%***

15.6%

49.1***

–54.2%***

–45.3%**

Rate ofdisbursement(to midterm)

Gap(to midterm)

Rate ofdisbursement

(overall)

Gap(overall)

Rate ofDisbursement–

Adjusted bySector Average(to midterm)

Rate ofDisbursement–

Adjustedby SectorAverage(overall)

Significant level: * = 10 percent, ** = 5 percent, *** = 1 percent

discussion on constructive ways to make aid more effective. Evidence-based midcourse corrections, a clearer implementation road map, strengthened capacity on the ground, and observer effects are possible channels to explain the results. We hope this analysis will stimulate discussion over the optimal balance between project financing and impact evaluation research needed to deliver development outcomes.Source: Legovini, Arianna, Vincenzo Di Maro, and Caio Piza, “Impact Evaluation Helps Deliver Development Projects” World Bank, Working Paper 2014.

http://elibrary.worldbank.org/doi/book/10.1596/1813-9450-7157




– 109 –

(iii) Achieving High Quality Services and Products; and (iv) Accomplishing Key Outcomes of Knowledge Services in terms of informing policy and decision making, building consensus among stakeholders, influencing public debate, strengthening institutional capacity, improving networks, and generating innovative approaches and solutions.

Use and Policy Change

Ninety percent of clients used the IE results, and 67 percent of them thought the use of results led to actual change (versus an average 42 percent for other knowledge and advisory services as a whole).

i: cliEnt fEEDbacK on iE ProDuctS

background

The World Bank’s 2013 Annual Client Feedback Instrument (CFI) is a mechanism to systematically gather clients’ opinions on the knowledge and advisory activities they have received from the World Bank. These activities aim to influence decision making, strengthen institutions, and empower clients to implement reforms. They include economic and sector work (ESW), technical assistance (TA), external training (TE), and starting in 2013, impact evaluation (IE). The 2013 CFI survey covers 1,525 activities and focuses on government executives (78 percent of respondents), the group best positioned to use knowledge services for policy impact.

IE Survey Results

IE products scored highly in the 2013 CFI survey.23

One hundred percent of clients stated that IE work achieved the agreed-to objectives, two-thirds of respondents noted that IE results were used for policy impact, and 92 percent considered quality of IE work as high. The quality score was driven by client satisfaction with the high technical quality of IE products (85 percent), timeliness (100 percent), relevance to the client’s needs (92 percent), and a higher level of engagement with government counterparts (92 percent).

Compared to other knowledge services, IE products received the highest marks for all categories measured, including (i) Use and Policy Change; (ii) Achieving the Objectives Agreed;

23 http://cfi and select IE as the product line.

Government Executive

Government Judiciary

Government Legislature

Media

Other

Academic/training/researchinstitution

For-profit private sectororganization

Civil society/Non-for-profitnon-governmental organization

Client Feedback InstrumentGraphs with less than 6 responses will not be displayed below

Demographics Organization Type

7%

6%

4%

78%

3%

1%

0%

2%

GENDER

Male 66%Female 34%


– 110 –

FIguRE I.1: Product Use

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

All products

EW

IE

TA

TE

10% 17% 30% 42%

11% 16% 25% 47%

5% 5% 24% 67%

11% 15% 29% 45%

10% 18% 31% 41%

No use Planned use Use, no change Use, led to change

FIguRE I.2: Satisfaction with Clarity of Objectives and Accomplishment

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

All products

EW

IE

TA

TE

8% 31% 61%

8% 28% 64%

100%

11% 30% 59%

2% 46% 52%

Not effective Somewhat effective Effective and above

Activity-level averages on a 10-point rating scale: Not Effective: < 6 Somewhat Effective: ≥ 6, < 8 Effective: ≥ 8

Achieving the Objectives Agreed and Achieving High Quality Services and Products

Clients were satisfied with the clarity of objectives and their accomplishment. For 100 percent of surveyed clients, the IE objectives were achieved (compared to 61 percent for knowledge services as a whole as seen in figure I.2), with an average quality score of 9/10 as seen in figure I.3.


– 111 –

FIguRE I.3: Quality of Knowledge and Advisory Services

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

All products

EW

IE

TA

TE

8% 28% 64%

9% 27% 63%

8% 92%

10% 25% 65%

1% 40% 59%

Rating 1–5 Rating 6–7 Rating 8–10

Accomplishing Key Outcomes of Knowledge Services

Impact evaluations scored well in achieving the final objectives of the Bank’s knowledge services—namely, informing and influence policy making to strengthen and empower clients to implement reforms. IEs scored better than ESW, TA, and TE in all measured outcomes: informing

policy and decision-making (92% vs. 63%), building consensus among stakeholders (77% vs. 49%), influencing public debate (69% vs. 43%), influencing thinking of government decision-makers (92% vs. 67%), strengthening institutional capacity (92% vs. 65%), improving networks with experts or facilitated exchange of ideas (100% vs. 66%) and generated innovative approaches and solutions (85% vs. 58%).


– 112 –

FIguRE I.4: Impact Evaluations at Achieving Key Outcomes of Knowledge Services

Informed policy, strategy, and/or decision making (EW, ..

Built consensus among stakeholders (EW, IE, TA)

Influenced public debate (EW, IE, TA)

Influenced your thinking or that of your organization (E..

Strengthened your capacity or that of your organization (E..)

Improved networks or contacts with other experts; ..

Generated innovative approaches and solutions (E..

100%

85%15%

92%

92%

69%23%

23% 77%

92%8%

8%

8%

8%


Source: cfi.worldbank.org.

FIguRE I.5: Other Knowledge and Advisory Services at Achieving Key Outcomes of Knowledge Services

Informed policy, strategy, and/or decision making (EW, ..

Built consensus among stakeholders (EW, IE, TA)

Influenced public debate (EW, IE, TA)

Influenced your thinking or that of your organization (E..

Strengthened your capacity or that of your organization (E..)

Improved networks or contacts with other experts; ..

Generated innovative approaches and solutions (E..

26%

33%9%

27%

27%

44%

6%

11% 40%

30%

66%

58%

65%

67%

43%

49%

63%7%

14%

8%

9%


Source: cfi.worldbank.org.


