Embed Size (px)
Citation preview
WP 2017-03 January 2017
Working Paper Charles H. Dyson School of Applied Economics and Management Cornell University, Ithaca, New York 14853-7801 USA AN EVALUATION OF THE FEEDBACK LOOPS IN THE POVERTY FOCUS OF WORLD BANK OPERATIONS
Shahrokh Fardoust, Ravi Kanbur, Xubei Luo, and Mark Sundberg
2
It is the Policy of Cornell University actively to support equality of
educational and employment opportunity. No person shall be denied
admission to any educational program or activity or be denied employment
on the basis of any legally prohibited discrimination involving, but not
limited to, such factors as race, color, creed, religion, national or ethnic
origin, sex, age or handicap. The University is committed to the
maintenance of affirmative action programs which will assure the
continuation of such equality of opportunity.
3
An Evaluation of the Feedback Loops in the Poverty Focus
of World Bank Operations*
By
Shahrokh Fardoust, Ravi Kanbur, Xubei Luo and Mark Sundberg**
This Version: 31 December, 2016
Contents
1. Introduction
2. Evaluation Methodology
3. Project Level Monitoring and Evaluation
4. Country Level Results Monitoring and Learning
5. Pilots, Scaling Up and Feedback Loops
6. Stakeholder Consultation and Coordination
7. Conclusion
* The authors were part of the team which worked on the report The Poverty Focus of Country Programs: Lessons from World Bank Experience (IEG 2015a), produced by the Independent Evaluation Group of the World Bank. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.
** Shahrokh Fardoust is a research proferssor at the College of William and Mary. Email: [email protected].
Ravi Kanbur is T.H. Lee Professor of World Affairs, International Professor of Applied Economics and Management, Charles H. Dyson School of Applied Economics and Management; Professor of Economics, Department of Economics. Email: [email protected].
Mark Sundberg is Chief Economist, Millenium Challenge Corporation, Washington DC. Email: [email protected].
Xubei Luo is a senior economist in the Poverty Global Practice, The World Bank, Washington, DC. Email: [email protected].
4
Abstract
The World Bank Group in 2013 made the elimination of extreme poverty by 2030 a central institutional focus and purpose. This evaluation, conducted by the Independent Evaluation Group of the World Bank Group, examines how, and how well, the Bank uses feedback loops to enhance the poverty focus of its operations. Feedback loops are important for every element of the causal chain running from data, to diagnostics, to strategy formulation and finally to strategy implementation. The evaluation uses a range of instruments, including surveys of stakeholders and World Bank staff, focus group meetings, country case studies and systematic reviews of Bank lending and non-lending operations. We find that while the Bank generates useful information on poverty reduction from its projects and programs, the feedback loops—from outcomes to data analysis to diagnostics to strategy formulation and implementation—have generally been weak, with sizable variation across countries.
Keywords: World Bank, Poverty Focus, Feedback Loops, Monitoring and Evaluation, Learning.
5
1. Introduction
The World Bank was a product of the Bretton Woods Conference in 1944, when the soon to be victorious powers agreed to form two multitateral institutions: the International Monetary Fund to manage currencies and capital flows, and the International Bank for Reconstruction and Development (IBRD) to provide official finance to nations for infrastructure and other reconstruction for war torn Europe and Japan. The nature of the Bank, supported as it was by the deposited capital and callable capital of the nations of the world, meant that it could borrow relatively cheaply on capital markets and pass on this benefit to borrower nations. After post-war reconstruction, in the 1960s the institution turned its attention to development in the newly independent, and poorest, countries of the world. A second entity was added, the International Development Association (IDA), which offered soft loans to the poorest nations, financed by donor contributions from rich countries. These two operational arms (IBRD, IDA), together with three other entities, now form the World Bank Group. This paper’s focus will be on IBRD and IDA.
In 2013 it set as one of its twin goals the reduction of extreme poverty to 3 percent of the world population by 2030. Globally, the proportion of people living in extreme poverty, in other words surviving on less than $1.9 a day, fell from 37 percent in 1990 to 13 percent in 2012. However, continuing success in reducing extreme poverty will become more difficult. Poverty is becoming more concentrated geographically: as of 2011, almost 1 billion people still lived in extreme poverty, with 390 million in Sub-Saharan Africa, 362 million in South Asia. Under plausible assumptions, growth alone will be insufficient to reach the targets, so new solutions will be needed that go beyond stimulating growth. Distributional changes will almost certainly be necessary.1
Achieving the goal of eliminating extreme poverty will require concerted action by developing countries, and other members of the development community, including the World Bank. The typical levers that a multilateral development bank has start with lending but they also include analytical work, convening power, technical assistance, capacity building, and policy dialogue. However, the financial influence of the Bank Group and other development partners has steadily decreased, dwarfed by much larger international capital flows, mainly from the private sector. With this diminishing financial influence, the World Bank’s effectiveness in poverty reduction will have to improve if it is to meet its goals.
But how well has the World Bank in fact done in helping its clients to reduce poverty? An answer to this question is not easy, and requires a full examination of the impact of the Bank’s operations on each stage of a causal chain which goes from data, to diagnostics, to strategy formulation, to strategy implementation, to outcomes. Further, at each stage of the chain we would have to assess the feedback loops used by the Bank to update and alter its operations. The full evaluation is presented in IEG (2015a). This paper, however, provides a focused assessment of the last element—the feedback loops which help to enhance the Bank’s focus on poverty. The evaluation is for all countries receiving support from IBRD and IDA, as well as “blend” countries that get assistance from both, and covers the period 2004-2012.2
6
The plan of the paper is as follows. Section 2 gives some background to the Independent Evaluation Group (IEG) of the World Bank and sets out the evaluation methodology. The next four sections cover in sequence different aspects of feedback loops: Project Level Monitoring and Evaluation (M&E); Country Level Results Monitoring and Learning; Strengths and Weaknesses of the Feedback Loops across Countries; and Stakeholder Consultation and Coordination. Section 6 concludes.
7
2. Evaluation Methodology
The Independent Evaluation Group (IEG) of the World Bank is charged with evaluating the activities of IBRD and IDA. The goals of evaluation are to provide an objective assessment of the results of the Bank Group’s work and to identify and disseminate lessons learned from experience. Independence of evaluation is ensured by the organizational structure. IEG is an independent unit within the World Bank Group. It reports directly to the Board of Executive Directors, which oversees IEG’s work through its Committee on Development Effectiveness. The managements of the World Bank cannot alter study findings or prevent their release. This independence allows IEG to address accountability issues and to make recommendations based on lessons of experience. The Director-General of IEG reports not to World Bank management but directly to the World Bank Group's Board of Directors. The Director-General (DG) appointment is the last appointment anyone can have in the World Bank Group—the DG and staff at the director level and above cannot take up a post in Bank management after finishing at the IEG.
Each of the World Bank’s activities, and the deployment of the totality of instruments in a sector, or in a country as a whole, comes under the evaluation mandate of IEG. The World Bank provides project finance through its Investment Lending instrument; it provides budgetary support through Development Policy Loans for policy reform; it provides analysis and policy recommendations through a range of instruments such a Poverty Assessments. The World Bank’s strategy for a country is set out in a document called the Country Partnership Strategy (CPS). These documents too, and the country assistance strategies as a whole, are also subject to evaluation.
At the close of every lending operation, World Bank staff prepare an Implementation Completion and Results Report (ICR). IEG reviews all ICRs of completed operations and provides ratings for a number of categories such as outcomes, but also including quality of monitoring and evaluation. The operation is rated by IEG on a six point scale ranging for highly satisfactory to highly unsatisfactory. Other Bank products are also evaluated by the IEG. At the end of every period for which a Country Partnership Strategy has been prepared, IEG produces the Country Partnership Strategy Completion Report Review (CPSCRR). With these basic evaluations, IEG also carries out meta evaluations for a country program as a whole (Country Partnership Evaluation--CPE), or for a sector (for example education), or a type of operation (e.g. Development Policy Loans, or support for Social Assistance) across countries.
Most recently, IEG carried out a meta evaluation of the poverty focus of the World Bank’s operations (IEG, 2015a), from which this paper is drawn. Meta evaluations start with a results chain and deploy a range of evaluation instruments to generate evidence on the quality of the Bank’s work at each point in the chain. For this evaluation, the results chain examined goes from data, to diagnostics, to country strategy formulation, to strategy implementation (see Figure 1). It is recognized that attribution of the final outcome (poverty reduction) to Bank specific actions is not possible. But each of the elements in the chain can be evaluated using a number of evidentiary tools. Finally, the evaluation can also look at the feedback loops and learning in Bank processes which better serve to
8
provide a poverty focus to its operations. The full evaluation is in IEG (2015a). The focus of this paper, however, is on the feedback loops.
Figure 1. Evaluation Chain for the Poverty Focus of Country Programs
The specific question asked in this paper is: Did the World Bank have adequate feedback loops to collect and draw lessons from poverty-reduction interventions to improve the effectiveness of its country strategies and programs? A strong mechanism for learning from results can help strengthen the design and implementation of the Bank’s projects and programs and improve the effectiveness of its limited resources on poverty reduction. The exploitation of information provided by M&E activities at the project, program, and country levels to feed back into data, diagnostics, and strategy formulation and implementation is therefore essential. We assess these feedback loops and their relationship to the Bank’s M&E mechanisms at the project and country strategy levels. The evaluation considers the strengths and weaknesses of the feedback loops, as well as stakeholder coordination and consultation.
In order to answer this question, the evaluation relies on: (i) an external survey with client government officials and nongovernment stakeholders in 20 countries, (ii) an internal survey with Bank staff, (iii) a series of focus group meetings with Bank staff who work on poverty, (iv) 10 in depth country case studies, (v) a review of Country Partnership Strategy Completion Report Review in 66 countries, and further reviews of Country Partnership Evaluations. While all of these instruments are referred to in this paper, there is particular reliance on the review for partnership strategies in 66
World Bank
Country Strategy Formation
Analytical Work on Poverty
World Bank
Country Strategy Implementation
Poverty Data
Feedback Loops
9
countries, 10 in-depth country case studies, and Bank staff survey and focus groups. We also rely on IEG data bases which document evaluation scores, and past IEG meta-evaluations, which are referenced as needed.
The 10 country case studies drew in-depth lessons from the Bank’s support of government efforts to reduce poverty. The countries were selected from a population of 144 countries comprising all IDA, IBRD, and blend countries. 3 The selection is purposive and does not aim to fully represent the various categories of countries. It tries to cover a range of countries at different income levels to reflect the differing approaches and challenges to poverty reduction in countries at different levels of development. The case study countries were selected also to roughly reflect regional balance. An emphasis was placed on countries with significant Bank engagement (lending and nonlending activities) 4. The 10 countries selected for study are Bangladesh, Egypt, Guatemala, Lao People’s Democratic Republic (PDR), Malawi, Nigeria, Peru, the Philippines, Romania, and Senegal. Focusing on 2004–2012, each case study consisted of desk reviews, structured interviews with Bank staff, and in-country consultations with stakeholders. The findings of the case studies are summarized in detail in Appendix A of IEG (2015a). A shorter summary, focusing on feedback loops, is provided in the Appendix to this paper.
The internal staff survey sought insight into staff perspectives of the constraints on data and the drivers of the difference in quality of poverty data and diagnostic work. The survey focused on three areas: constraints to obtaining poverty data; best practices and challenges to creating poverty diagnostic work; and challenges of translating poverty diagnostic work into country strategies. The survey was sent to all professional staff above a certain grade who were working on poverty issues in the Bank’s operations and research departments.5 A response rate of 21 percent is on a par with similar surveys administered within the Bank.6 Full details are available in Appendix C of IEG (2015a).
Six focus group meetings gathered in-depth information about how the availability of poverty data affects the Bank’s poverty diagnostics, and whether and how they translate into country strategies (Appendix D of IEG, 2015a). The representatives at the meetings included three distinct groups: task team leaders or co-leaders of Poverty Assessments/Poverty Updates, country economists of countries that have not had a poverty assessment since 2009, and task team leaders or co-leaders of country partnership strategies.7
10
3. Project Level M&E
Deficiencies in the M&E design for projects are most frequently identified as shortcomings in Bank support at entry (IEG 2015b). In general, collecting results information from projects has been weak. Between 2007 and 2013, among the 1,841 projects for which IEG assessed the M&E frameworks, 54 percent were rated “modest” and 15 percent were rated “negligible.” Only 31 percent had M&E frameworks that were rated “substantial” or “high.” There is no discernible improvement over time (Figure 2), leaving two main shortcomings. First, outcomes are not measured as often as they should be, nor are the intended outcomes of Bank projects and programs the focus of measurement, particularly for the groups targeted for poverty reduction. Second, when outcomes are measured they are seldom attributed to Bank-supported interventions to reduce poverty.
Figure 2. IEG Project M&E Ratings
Source: IEG Results and Performance Database. Note: The data are based on the number of rated projects that received ratings of negligible, modest, substantial, or high as a percentage of total projects rated (excluding those deemed not evaluable). Fewer than 55 percent of projects were rated in 2000–2006, and so were excluded from the analysis. In 2007–2013, 98 percent of projects had M&E ratings.
The design, implementation, and use of M&E in Bank projects and programs can be strengthened considerably from the perspective of poverty focus. Often what gets measured are the project milestones (mainly outputs or readily measured intermediate outcomes) but not the final outcome of poverty reduction, although practices vary. Good practice includes the Philippines Social Fund for Peace and Development project, which tracked improvements in income, food security, literacy rates, enrollment rates, infant mortality, malnutrition, water supply, and market access. In Guatemala, the latest series of fiscal Development Policy Loans includes project outputs that are directly targeted to poverty-relevant outcomes,8 there are both poverty-related targets (child health and nutrition) and procedural targets. By contrast, most project-level indicators in Senegal examined outputs only implicitly linked to poverty or tracked the non-income Millennium Development Goals.9
11
In Nigeria, the Bank did not produce substantial information on poverty reduction that would lead to M&E. Poverty was not included in the proposed outcomes and results to be monitored as part of assessment of the Bank’s program.
In most of the country case studies, the results frameworks of projects were not linked in a substantive way to the country strategy results framework or results chain. The focus of the monitored indicators on outputs or intermediate outcomes failed to provide sufficient information about the project’s impact on poverty reduction. If the objective is reducing poverty at the regional or national level, project-level M&E needs to pay more attention to explicit linkages to poverty reduction and the potential for scaling up interventions to achieve a wider effect. These findings confirm the results of a recent IEG evaluation, Learning for Results in World Bank Operations: How the Bank Learns (2014a), which indicates that lessons from project experience are not systematically used or developed at either initiation or completion. Project completion reports, a main instrument for learning, also are generally weak in documenting and drawing lessons on poverty reduction.
Lessons in the implementation completion reports (ICRs) were often too general to be useful and had limited external validity across countries. While useful as information, the lessons were not specific about the implications for poverty reduction. The Senegal case study for this evaluation, for example, found that the lessons from project completion reports included the need for government ownership, dealing with institutional issues early in the project design, setting up an adequate M&E system with appropriate baselines and indicators, and moving to multidonor and harmonized budget support. Most staff, the IEG learning study found, viewed ICRs as an accountability tool focused on project ratings rather than learning (IEG 2014a). And though reading the reports before designing projects could help prevent the repetition of mistakes, the lessons from ICRs were often only copied into appraisal documents without adjusting project design.
12
4. Country Level Results Monitoring and Learning
The monitoring of Country Partnership Strategy results covers a wide range of poverty-related areas, with most attention to education and health, but the monitoring of direct indicators for poverty is limited. Figure 3 shows the percentage of countries that monitored indicators in areas directly related to poverty reduction, drawing from the 66 cases reviewed. Most country strategies focused poverty-related monitoring on three to seven areas.10 The indicator most frequently monitored is basic infrastructure (such as water and sanitation), followed by schooling quality. But only about 40 percent of the cases included the monitoring of a direct “poverty” indicator in the first period, which declined to some 20 percent in the second period. Full details on all of these calculations are available in Appendix F of IEG (2015a)
The country case studies conducted for the evaluation show varying experience. In Malawi, for example, the poverty data and feedback from the overall picture fed directly into the monitoring of the strategy and design of the subsequent country strategy. There is constant reference to the dire poverty situation and the disappointment that the poverty context of the country is not changing more quickly. An explicit M&E system is used and referred to in subsequent Progress Reports. However, in many countries, only a few poverty indicators are monitored in the Country Assistance Strategy. For example in Guatemala, the strategy documents reviewed during the period of evaluation present neither poverty indicators (such as headcount poverty ratios) in their results frameworks, nor changes in poverty indicators targeted as part of the M&E framework, though such changes are discussed in the background sections of the strategy documents. The inclusion of more explicit poverty-relevant indicators, and monitoring and tracking them regularly in the strategy updates, would help programs adapt to improve their effectiveness in reducing poverty.
Figure 3. Country Monitoring of Poverty-Related Indicators
Source: World Bank Database. Note: The figure shows the percentage of countries that monitored at least one indicator in areas directly related to poverty reduction from the 66 country strategies reviewed, which have completed two rounds of Country Partnersip Strategy Completion Report Review (CPSCRR) during
0
20
40
60
80
100
PovertyReduction
Education Health SocialProtection
SocialDevelopment
Agricultureand Rural
Development
BasicInfrastructure
Perc
ent o
f Cou
ntrie
s
Area:
CPS1 CPS2
13
the period of 2004-2013. CPS1 and CPS2 refer to the sets of countries in the first and second round of CPSCRR, respectively.
While there was consistency between country strategy objectives and Bank interventions in some countries, it was not always clear how inputs were expected to produce poverty-reduction outcomes. In Nigeria, the 2010–2013 country strategy updated the poverty context and generally observed the importance of non-oil growth for poverty reduction. But it was not clear how the individual strategic components related to the poverty challenges at the national or sector levels. In Guatemala, there was a thematic disconnect between poverty diagnostics and the choice of programs and projects, with an overreliance on development policy lending and limited links to poverty reduction.
Focusing on a small number of monitorable targets, realistic indicators, and good baseline data is the key to success. There are some good examples to be found in the Bank’s work. Guided by lessons from the preceding country strategy review, the Philippines country team committed to strengthening the design and monitoring system for the 2010–2012 country strategy. The team created working groups mapped to each of the country strategy’s objectives. Within these groups, specific teams were assigned to track progress on their objectives. Five full-day workshops held throughout implementation allowed the teams to come together to share their progress. These workshops became part of the monitoring system for the country assistance strategy. As a result, the M&E system improved at both the country program level and the project level, though it continues to focus on the national monitoring system. The country strategy included milestones to monitor progress, and supplemented them with specific and quantifiable indicators for each outcome, an improvement over the previous system. The main factor driving the difference between the two systems was the commitment of Bank staff and management to enhancing the results focus of their strategies.
Similarly, in Bangladesh, a system in response to the lessons learned from the preceding review was developed. However, though the country management effectively used the results monitoring process to modify and streamline the World Bank program, the regularity of this process declined over time, particularly the element which involved government participation. This was in part due to changes in Bank country management and also to shifts in the country program as well as the Bank’s relationship with the government.
14
5. Pilots, Scaling Up and Feedback Loops
A main weakness of the Bank’s feedback loops is that M&E of individual interventions often fail to provide broader insights to feed back into country strategies. Although many Bank country strategies articulate poverty links to individual components of the strategy, there is seldom a direct or explicit link between the activity-level outcomes and the country strategy “higher-level” objectives, including scaling up of interventions.
Piloting and scaling up of interventions are occurring in some countries. In Senegal, several pilot cases were designed for Bank operations encouraging the scaling up of small or pilot projects. For access to land, a pilot activity was envisaged in the 2003–2006 country assistance strategy. The aim was to introduce market-based mechanisms for land allocations and, once proven successful, to scale them up. The 2007–2010 strategy supported pilot projects in education and health to provide performance-based incentives and to motivate key staff to relocate to underserved regions of the country. During the global financial crisis, the Bank scaled up its earlier pilot under the Nutrition Enhancement Project in the context of its Emergency Nutrition/Cash Transfer Project.
In the Philippines, the Bank focused its support primarily on piloting and scaling up the conditional cash transfer program and the community-driven development programs.11 The lessons from these pilots, and from the various other learning methods, helped strengthen the design, implementation, and scaling up of projects, enhancing the poverty focus of the Bank’s projects. The Bangladesh strategies explicitly identified building on good performance during the previous period for the Reaching Out-of-School Children Project. Initially funded as a $50 million “pilot,” it received additional financing of $30 million and was followed by a second project of $130 million.
The evidence is thin, however, on whether successful pilots were used and whether they leveraged non-Bank resources, compared with a general expansion through additional financing. From 2005 to 2014, 14 percent of the Bank’s $233 billion in investment lending went to additional financing.12 IEG (2015c) has initiated evaluation of the Additional Financing instrument which increases funds for approved projects to assess whether financing was effectively used to scale up interventions or to cover cost overruns. Data from the transport sector shows the latter is more often the case and that opportunities to strengthen the results framework and improve learning, reflected in a disconnect between project outcome indicators and their ability to sufficiently or accurately measure achievement of project development objectives.
The focus group meetings suggested that although most strategy documents stated that the lessons from reviews of previous strategies were considered in developing the current strategy, there often was no direct evidence to that effect. The strategy documents often only referred to the lessons in the main text, but did not indicate how and where these lessons were taken on board—or how the lessons may have changed or affected Bank staff views. The Senegal 2003–2006 country strategy refers to two key lessons from the review of the previous strategy: more aid is a poor substitute for better aid, and the financial management and procurement faces serious problems in
15
Senegal. However, it did not link these lessons to the measures it proposed. The focus groups indicated that while formally the Bank takes steps toward learning lessons from the previous strategy, and in some cases even convenes action or lessons meetings with the country management units, usually the new strategy depends heavily on the existing project pipeline, the government’s national agenda, and at times the preferences or priorities of the country director.
There are some good examples, however. Continual learning and strong feedback loops can go a long way in strengthening the poverty focus of the country programs, as in Lao PDR. The Bank’s country team produced, or helped produce, sufficient evidence on poverty reduction and made good use of it in the design, implementation, and evolution of the poverty focus of the country program. Adequate poverty data, and good and extensive poverty diagnostics, provided strong analytical underpinning. The programmatic nature of the Poverty Reduction Support Operations facilitated the evolution of the country program and its poverty focus.
The strength of feedback loops in a country varies along the results chain from data, to diagnostics, and to strategy formulation and implementation—and across sectors. In Nigeria, the lack of champions demanding poverty data or diagnostics is a major reason for the incomplete feedback loop from results back to data and diagnostics. Bank-financed poverty-focused interventions were small in scale relative to the problems and the government’s own resources, although several demonstrated strong technical approaches to evaluation. Explicit attention to poverty-reduction objectives was either at a high level of generality (as in debt relief and non-oil sector growth programs) or detailed in a subset of the program (as in community-driven development programs). The Bank program did monitor the Millennium Development Goals, which include an indicator on income poverty. However, in general, the Bank did not produce substantial information on poverty reduction that would lead to effective M&E. In neither of the country strategies reviewed was poverty explicitly included in the proposed outcomes and results matrix to be monitored as part of the assessment of the Bank’s program.
In Bangladesh, the strength of feedback loops varied across sectors. In some cases they are strong, as in the social protection program in which a history of good analytic work contributed to a well-prioritized set of operational recommendations (focusing on efficiency, efficacy, and targeting), which then formed the basis for dialogue with the key implementing agencies and project interventions when circumstances were ripe. Feedback is also relatively strong in education beginning with a focus on improved targeting of education stipends to increase girls’ school attendance leading to the realization that attendance rates for boys were falling and the adjustment of stipend programs to tackle this problem. Poverty-focused feedback loops have been weaker in infrastructure lending although there is an increasing interest in these sectors (notably rural roads and rural electrification) in drawing on factors as the priorities of Bank management and country teams and such external factors as the political commitment and administrative and technical capacity of counterparts. M&E in project implementation also varies considerably with the incentives and skills of staff. The quality of M&E design, implementation, and use has been an area of persistent weakness in Bank projects.13
16
Evidence also suggests much more needs to be done to integrate knowledge from evaluations into staff practices and the project cycle (IEG, 2014b).
6. Stakeholder Consultation and Coordination
Most Bank country strategies include some participatory consultations with both government and nongovernment stakeholders. The experience is mixed. In Guatemala, all three country strategies involved extensive consultations and resulted in shifting the emphasis in the proposed programs. Changes to the 2005 strategy document increased emphasis on environment across Bank-supported activities. They also broadened a proposed Local and Rural Development Project to include basic infrastructure services, such as water, sanitation, electrification, and information and communication technologies. And they sharpened the focus on human development, infrastructure needs, and access to finance in indigenous communities.
In Senegal, the preparation of the country strategies included extensive consultations with government, development partners, the private sector, and civil society to ensure that the poverty policies and priorities would be well thought out and broadly supported. The 2003–2006 strategy asserted that the Bank would seek to ensure better country ownership of the poverty-reduction agenda through closer policy dialogue, especially with civil society. Along with extensive public consultation on the development of the strategy, several Bank operations used participatory and community development methods as inputs to define local priorities.14
However, focus group discussions with Bank staff noted that in many cases there was no clear evidence that consultations had a significant effect on either the design or the implementation of Bank strategies. They were often treated as an opportunity for Bank staff to inform the government and other stakeholders of the proposed strategy and interventions. The limited evidence available from the strategies and other documentation indicates that suggestions from stakeholders were not fully taken on board. The consultations usually followed a formal process, often late in the strategy formulation process, and partly due to the tight timetable.
In Peru, the topics discussed in the consultations differed from the major directions of the strategy in areas of relevance to reduce poverty. Multiple consultations were carried out with civil society for the 2007 strategy, in addition to consultations with the incoming and outgoing authorities and with the private sector. In different parts of the country, they focused on good development practices and income generation for the poor. But the topics discussed were not related to public spending at the local level, to access to services, or to a new social contract in health and education, a major area in the country partnership strategy.
Preparing meaningful and well-informed consultations in the short timeframe for preparing a strategy is difficult, according to staff at focus group discussions. Identifying stakeholders who know the Bank well for consultation can be a challenge. In some countries, the Bank relies on the
17
government to invite stakeholders to consult on the country strategies, possibly clouding the transparency of the selection process. Finding the right stakeholders is even more difficult in fragile and conflict-affected states, where inviting stakeholders from across the country may preclude safety and security issues.
7. Conclusion
This evaluation has examined feedback loops in the World Bank’s support for poverty reduction in its country programs over the period of 2004–2012. The overarching question is—”How, and how well, has the World Bank focused its programs on reducing poverty?”. Using a range of instruments, the evaluation finds that the Bank’s mechanisms for learning from project experience, from results to data analysis to diagnostics to strategy formulation and implementation, have generally been weak, though with significant variation across countries. The Bank does generate information and learning about poverty reduction from its programs, and in most cases, the formal processes of M&E are required but inconsistently implemented. At the project level, the processes tend to focus narrowly on outputs or readily measured immediate outcomes and often fail to reflect the broader impact of an intervention in the medium or long term. Deficiencies in the M&E design for projects are most frequently identified as shortcomings in Bank support at entry, and the M&E activity needed to drive feedback loops most often remains weak through project implementation.
At the strategy level, during the evaluation period 2004-12, feedback loops tend to focus on “checking the box” without an assessment of whether a real difference is being made to poverty. In both cases, the processes are not systematically integrated in the strategy or in individual projects. Project and program-level M&E are most effective when they measure outcomes and inform the design and implementation of country strategies, providing a basis for scaling up support to better leverage resources. Improving the use of poverty data in project M&E to improve planning and implementation is crucial. Finally, the formal process for preparing a country strategy often includes some form of participatory consultation. However, there is limited evidence that such consultations have had strong effects on either the design or implementation of Bank strategies.
Specific recommendations which follows from the evaluation are firstly, to develop and adopt explicit evaluation protocols for piloted interventions to capture lessons from experience on poverty reduction, with a view towards opportunities for scaling up successful interventions and learning from failures. Secondly, Bank staff need to ensure attention at project inception to evaluability through (1) developing standards for baseline measurement, (2) explicit linking of the baseline to indicators relevant to project objectives, including any that refer to poverty or inclusion impacts, and (3) robust planning for monitoring data required for ex-post evaluation.15
18
Appendix: Summary of Feedback Loops in 10 Country Case Studies 2002-2012
Fuller Summaries are available in IEG (2015a, Appendix A).
Bangladesh The sequence from data to diagnostics to strategy formulation and implementation varied across sectors. In some cases, feedback loops were strong. In the social protection program, for example, a history of good analytic work (feeding into diagnostics work) contributed to a well-prioritized set of operational recommendations that then formed the basis for policy dialogue. Feedback loops were also relatively strong in education. Poverty-focused feedback loops were weaker in infrastructure lending. The Bank could have devoted more attention to the challenge of poverty monitoring between the household income and expenditure rounds, and a programmatic approach to poverty assessments might have provided more timely inputs into strategy formulation. The results chain could have been more explicit in sectors that have a more indirect influence on poverty reduction. Had the strategies presented and analyzed poverty reduction more rigorously, the poverty focus would likely have been more clearly linked to the growth objective, with stronger results.
Egypt Feedback loops in Egypt were affected by the fact that poverty data were not fully accessible during the period under review and the data that were made available may not have fully captured the new type of poverty that has been growing around metropolitan areas and in major shanty towns. Although the Bank’s poverty diagnostics were generally of good quality, the data used may not have provided the full picture. Moreover, during much of the period under review, the authorities did not allow the Bank to prepare some of the needed analytical pieces, such as Public Expenditure Reviews. Although the strategy formulated by the Bank over the years attempted to include some focus on poverty with a reasonable results chain for reducing poverty, the Egyptian authorities resisted focusing on poverty and borrowing for social sectors project, which may have undermined the Bank’s strategy.
Guatemala Feedback loops in Guatemala were not evenly strong in all areas. The Bank seems to have done an excellent job in supporting the generation of poverty data and in preparing poverty diagnoses over the past decade. As a result, lack of adequate data and diagnostics are no longer major obstacles to addressing poverty in Guatemala. The strategies and the choice of lending and nonlending operations appeared generally well aligned with the poverty diagnostics; a thematic disconnect between poverty diagnostics and the choice of programs and projects does not stand out as a major problem. The one area in the feedback loop where the Bank’s support for antipoverty efforts appears to have been least successful during the past decade is country strategy implementation. The result was an overreliance on development policy loans with limited links to poverty reduction.
Lao PDR Feedback loops in Lao PDR were strong. Given the programmatic nature of most of the core elements of the poverty reduction focus of the country program, there has been a continual learning process and feedback loop. The programmatic nature facilitated the evolution of the country program
19
and its poverty reduction focus. The Bank and other development partners worked intensively to improve the quality and accessibility of statistical capacity in Lao PDR. Strong Bank and development partner support also played an important role in improved monitoring and evaluation (M&E) capability in several Bank-supported projects and activities in Lao PDR.
Malawi Feedback loops in Malawi were strong. Progress was made in maternal and child health and nutrition, moderately poor households moved out of poverty through farm and nonfarm diversification and improved access to irrigation and other agricultural services, and there was no repeat of famine. Yet the poverty challenges have only increased over time, particularly as population growth puts further pressure on limited land, with both rural inequality and the share of the ultra-poor growing.
Nigeria Feedback loops were incomplete in Nigeria. No Nigerian champions demanded data or diagnostics. The Bank-financed poverty-focused interventions were of small scale relative to the scale of the problems and the government’s own resources, although several demonstrated strong technical and interpersonal approaches to evaluation. Evidence on poverty reduction in Nigeria was modest. Explicit attention to poverty-reduction objectives was either at a high level of generality (as in debt relief and nonoil sector growth) or detailed in only a subset of the program (as in community drive development programs). The Bank program for Nigeria did monitor the Millennium Development Goals, which include an indicator on income poverty. In general, however, the Bank did not produce substantial information on poverty reduction that would lead to M&E. In neither of the strategies under review was poverty included in the proposed outcomes and results to be monitored as part of assessment of the Bank’s program, including within a sectoral context.
Peru The diagnostic reports prepared by the Bank covered the most relevant poverty issues in Peru. Overall, these studies were of a very good quality in terms of meaningful use of empirical evidence (complemented with good-quality statistics), relevance and timeliness, depth of analysis, and identification of policy directions. The most recent strategy had a stronger link to poverty in Peru and better reflected the findings and recommendations of the Bank’s diagnostic work. A key issue was the significant divergence between the planned and implemented portfolios. The situation improved in more recent years, and there is now strong complementarity between the Bank’s poverty-related lending and nonlending activities. Nevertheless, in several key areas, such as the policy steps supported by development policy loans, there are no details on specific policy steps to be supported by development policy loans. The 2006 strategy included no efforts to develop a results chain linking the operations to objectives in this area. Where the strategy did envisage operations to reach the rural poor, it lacked a result chain for addressing the issue.
Philippines The Bank’s poverty diagnostic works were of high quality and provided strong analytical underpinnings for policy making and strategy formulation. However, the timeliness and quality of poverty statistics constrained monitoring the progress of poverty reduction. The absence of a full Poverty Assessment or Poverty Update since 2001 limited the depth of the work in certain aspects,
20
including the identification of the drivers of poverty at a disaggregated level and the tailoring of recommendations to overcome obstacles. There were also significant deviations between the envisaged and implemented country programs. Although the M&E system improved at both the program and the project level, it continued to focus mainly on national-level and intermediate outcomes. There is a need to strengthen the design of the feedback loop system to more closely monitor the intended outcomes of Bank efforts rather than the outputs of intermediate outcomes.
Romania The Bank closely monitored overall poverty reduction outcomes in Romania. Evidence from the Poverty Assessments and other diagnostic work, including evidence on poverty drivers as an underpinning for support to disadvantaged groups, informed the design of its programs. The 2006 and the 2009 strategies articulated and included poverty-related outcome targets for its projects and covered extreme poverty. More systematic measurement of poverty impacts might have strengthened the feedback loops that help inform the poverty content of new programs and operations. The feedback loops were strong in response to findings from analytical work, perhaps less so from project M&E. Feedback loops from analytical and advisory contributions worked primarily because the Bank has been a credible counterpart that built its reputation on poverty issues over the years with a strong record of work on data, poverty measurement, and poverty diagnostic issues. The Bank could perhaps learn more about poverty by strengthening the poverty M&E in projects and extracting more lessons from its project experience.
Senegal Implementation of Bank interventions was broadly consistent with the available poverty data and diagnostics as well as the Bank’s strategy. Projects that focused on rural areas, basic education, and the vulnerable parts of the population were particularly consistent with the poverty data. Budget support operations were key instruments for poverty reduction, but there was no planned dialogue or conditionality regarding the reallocation of public spending toward programs reaching the poor. Projects that focused on rural areas, basic education, and the vulnerable parts of the population were particularly consistent with the poverty data. The Bank provided strong support for generating good-quality poverty data and diagnostics. The government appeared to be committed to poverty reduction and made good use of the technical support provided by the Bank (and other donors) for its poverty data, poverty diagnostics, and preparation of its strategies. However, feedback loops from data to diagnostics to strategy formulation and implementation in Bank country strategies were weak, and there is a widespread belief that the Bank’s strategy was not closely monitored by the government or by the Bank.
21
NOTES
1 See more details in World Bank 2014.
2 The population includes 144 countries, listed as Bank borrowers as of October 2013.
3 The IBRD, IDA, and blend country classification is up to date as of October 2013.
4 Significant levels of Bank engagement are indicated by a minimum of 25 approved projects between FY2004 and 2012, since 25 projects is the median number of projects for the population of the 144 countries during this period.
5 The survey was conducted from April 15 to May 13, 2014, and was sent to 4,150 Bank staff, of which 866 responded (a rate of 21 percent). To focus on staff with close experience in operations and country strategies, staff mapped to procurement, human resources, information and technology, business solutions, World Bank Institute, and IEG were excluded. The survey was confidential and anonymous. See Appendix C of IEG (2015a) for details.
6 For example, the WHO Stakeholder Perception Surveys set the response rate for the internal staff survey at 20 percent as a good threshold (see http://www.who.int/about/who_perception_survey_2012.pdf)
7 The 18 countries represented in the discussions were purposively selected to cover all Regions considering the balance between the types of countries by lending group, accessibility of data, and FCS and non-FCS countries. The selected countries were Afghanistan, Botswana, Cambodia, the Democratic Republic of Congo, Jordan, India, Indonesia, Kosovo, the Kyrgyz Republic, Moldova, Pakistan, the Republic of Yemen, Russia, South Sudan, Sri Lanka, and Papua New Guinea, along with West Bank, Gaza, and the member countries of the Organization of Eastern Caribbean States. although all of the participants from relevant focus group categories in each country were invited to participate, the final list depended on the staff availability in June-August 2014. The total number of focus group participants was 22 staff members.
8 In addition to specific targets for the Tax/GDP ratio, which are ultimately meant to place public social spending on a more sustainable basis, the operation also has as outcome targets: (i) the percent of children under 1 year in 83 municipalities receiving the basic health/nutrition package, and (ii) the number of Zero Hunger Plan (Plan Hambre Cero) offices that have been established in the country to coordinate nutrition initiatives.
9 Of the projects reviewed, the subset of lending interventions for which poverty-specific data were collected included the Community and Social Development Project and the Community Poverty Reduction Project.
10 Out of the 66 countries reviewed, which have two strategies completed and reviewed during the period of evaluation, 48 monitored three to seven areas with direct poverty focus in the first period reviewed, while 58 monitored three to seven areas during the second period reviewed.
11 The program was initially piloted with 6,000 households in 2008. By 2009 it had been scaled up to 376,000 households, and it is currently being expanded nationally. To complement the initiative, the Bank also supported the development of the National Household Targeting System for Poverty Reduction which has become the main system of identifying the poor, providing objective information for the conditional cash transfer, community-driven development, and national health) projects.
12 The Board has approved 608 additional financing projects with a total volume of $32.7 billion, representing 14 percent of a total of $233 billion in investment lending during June 2005–June 2014. Data source: World Bank Business Warehouse (database).
13 Annual evaluation reports of Bank project performance have drawn attention to severe shortcoming of M&E quality for decades, most recently in IEG’s Results and Performance Report (IEG, 2015).
14 The operations included the Urban Development and Decentralization Program, the National Rural Infrastructure Program, and the Social Development Fund.
15 These recommendations are reflected in the Management Action Record, which tracks the program of management actions in response to evaluation recommendations, and to which Bank management has committed itself to undertake. The MAR database can be found at https://ieg.worldbankgroup.org/managementactionrecord.
23
References
IEG. 2012. World Bank Group Impact Evalautions: Relevance and Effectiveness. Washington, DC, World Bank.
IEG. 2014a. Learning for Results in World Bank Operations: How the Bank Learns. Washington, DC: World Bank.
IEG. 2014b. Results and Performance of the World Bank Group 2014. Washington, DC: World Bank.
IEG. 2015a. The Poverty Focus of Country Programs: Lessons from World Bank Experience. Washington, DC: World Bank. https://openknowledge.worldbank.org/bitstream/handle/10986/22486/The0poverty0fo0orld0Bank0experience.pdf?sequence=1&isAllowed=y
IEG. 2015b. Results and Performance of the World Bank Group 2014. Washington, DC: World Bank.
IEG 2015c. Learning Note: Additional Financing for Transport and Information and Communication Technology (ICT). Washington, DC: World Bank.
World Bank 2015. A Measured Approach to Ending Poverty and Boosting Shared Prosperity: Concepts, Data, and the Twin Goals. Washington, DC: World Bank.
WP No Title Author(s)
OTHER A.E.M. WORKING PAPERS
Fee(if applicable)
Secondary Towns and Poverty Reduction:Refocusing the Urbanization Agenda
Christiaensen, L. and Kanbur, R.2017-02
Structural Transformation and IncomeDistribution: Kuznets and Beyond
Kanbur, R.2017-01
Multiple Certifications and ConsumerPurchase Decisions: A Case Study ofWillingness to Pay for Coffee in Germany
Basu, A., Grote, U., Hicks, R. andStellmacher, T.
2016-17
Alternative Strategies to Manage WeatherRisk in Perennial Fruit Crop Production
Ho, S., Ifft, J., Rickard, B. and Turvey, C.2016-16
The Economic Impacts of Climate Change onAgriculture: Accounting for Time-invariantUnobservables in the Hedonic Approach
Ortiz-Bobea, A.2016-15
Demystifying RINs: A Partial EquilibriumModel of U.S. Biofuels Markets
Korting, C., Just, D.2016-14
Rural Wealth Creation Impacts of Urban-based Local Food System Initiatives: A DelphiMethod Examination of the Impacts onIntellectual Capital
Jablonski, B., Schmit, T., Minner, J., Kay,D.
2016-13
Parents, Children, and Luck: Equality ofOpportunity and Equality of Outcome
Kanbur, R.2016-12
Intra-Household Inequality and OverallInequality
Kanbur, R.2016-11
Anticipatory Signals of Changes in CornDemand
Verteramo Chiu, L., Tomek, W.2016-10
Capital Flows, Beliefs, and Capital Controls Rarytska, O., Tsyrennikov, V.2016-09
Using Unobtrusive Sensors to Measure andMinimize Hawthorne Effects: Evidence fromCookstoves
Simons, A., Beltramo, T., Blalock, G. andLevine, D.
2016-08
Economics and Economic Policy Kanbur, R.2016-07
W. Arthur Lewis and the Roots of GhanaianEconomic Policy
Kanbur, R.2016-06
Paper copies are being replaced by electronic Portable Document Files (PDFs). To request PDFs of AEM publications, write to (be sure toinclude your e-mail address): Publications, Department of Applied Economics and Management, Warren Hall, Cornell University, Ithaca, NY14853-7801. If a fee is indicated, please include a check or money order made payable to Cornell University for the amount of yourpurchase. Visit our Web site (http://dyson.cornell.edu/research/wp.php) for a more complete list of recent bulletins.
