African Development Bank · 2 Contents List of Acronyms

1

AFRICAN DEVELOPMENT BANK GROUP

Independent Development Evaluation

Evaluation of the Bank’s Country Strategy and Programme for Zambia (2002-2014) Inception Report Final Draft

AGRER September 2015

2

Contents

List of Acronyms ............................................................................................................................................ 4

1.0 Introduction ...................................................................................................................................... 5

2.0 Overview of Zambia's Development Context ................................................................................. 10

2.1 Development Context in Zambia ................................................................................................ 10

2.1.1 Political Context .................................................................................................................. 10

2.1.2 Economic Context ............................................................................................................... 11

2.1.3 Social Development ............................................................................................................ 11

2.2 Key Development Challenges ..................................................................................................... 11

2.3 Government of Zambia Development Strategies ....................................................................... 13

2.4 Zambia's Progress toward the Millennium Development Goals ................................................ 14

3.0 Overview of Bank Support to Zambia ............................................................................................. 17

3.1 The Bank's Country Strategies in Zambia .................................................................................... 17

3.2 Bank Corporate Policies and Strategies ...................................................................................... 19

3.3 The Bank's Portfolio in Zambia (2002-2014) ............................................................................... 22

3.3.1 Lending operations ............................................................................................................. 22

3.3.2 Non-lending activities ......................................................................................................... 24

3.4 A Theory of Change for the Bank's Activities in Zambia (2002-2014) ........................................ 25

4.0 Evaluation Scope and Methodology ............................................................................................... 28

4.1 Evaluation Scope and Objectives ................................................................................................ 28

4.1.2 The Evaluation Issues and Questions .................................................................................. 28

4.2 Proposed Methodological Approach .......................................................................................... 30

4.2.1 Data Collection Activities Completed to Date ..................................................................... 31

4.2.2 Proposed Data Collection Methods and Analysis ............................................................... 32

4.2.2 The Project Sample ............................................................................................................. 35

4.2.4 The Evaluation Framework ................................................................................................. 38

4.3 Special Considerations for Case Studies and Project Results Assessments ...................................... 51

4.3.1 Contribution Analysis - Methodology and Approach .......................................................... 51

4.3.2 Implementation .................................................................................................................. 52

4.4 Challenges, Limitations and Risks ............................................................................................... 53

5.0 Implementation Plan and Next Steps ............................................................................................. 54

Communication & Outreach Strategy ......................................................................................................... 64

Management and Quality Control .............................................................................................................. 65

List of Annexes ............................................................................................................................................ 66

Annex B - The Bank's Portfolio in Zambia (2002-2014) .............................................................................. 68

Annex C - List of Key Stakeholders .............................................................................................................. 70

Annex D - Interview Protocol ...................................................................................................................... 73

Interview Protocol - Task Managers ....................................................................................................... 73

Interview Protocol - Country Team ......................................................................................................... 75

Interview Protocol - Headquarters Staff ................................................................................................. 78

3

Interview Protocol - Government of Zambia .......................................................................................... 80

Interview Protocol - Project Implementation Units ................................................................................ 82

Interview Protocol - Other Development Partners ................................................................................. 85

Interview Protocol - Civil Society ............................................................................................................ 87

Annex E - PRA Assessment Template .......................................................................................................... 90

Annex F - Project Results Assessment Guidance ........................................................................................ 95

4

List of Acronyms

AfDB African Development Bank CEDR Comprehensive Evaluation of Development Results CODE Committee on Operations and Development Effectiveness COMESA Common Market for Eastern and Southern Arica CPIA Country Policy and Institutional Assessment CSP Country Strategy Paper CSPE Country Strategy and Program Evaluation EAC East African Community ESW Economic and Sector Work FNDP Fifth National Development Plan GDP Gross Domestic Product GNI Gross National Income IDEV Independent Development Evaluation JASZ Joint Assistance Strategy for Zambia MDG Millennium Development Goal MSME Micro, Small and Medium-sized Enterprises MTS Medium Term Strategy PRA Project results Assessment PRSP Poverty Reduction Strategy Paper RISP Regional Integration Strategy Paper SADC Southern African Development Community SNDP Sixth National Development Paper TYS Ten Year Strategy UA Unit of Account WSS Water Supply and Sanitation

5

Inception Report – Zambia CSPE

1.0 Introduction

This revised Inception Report is submitted by the AGRER Consulting Team currently tasked by the AfDB to perform the final evaluation of 3 Zambia Country Strategies, covering an extensive period of time (2002-2014. This report builds on and updates an extensive draft prepared by the AfDB IDEV staff. Their first draft captures and describes in detail Zambia’s macroeconomic trends, all the key AfDB goals, as well as the implementation process regarding 3 Zambia Country Strategies.

The AGRER Team carefully reviewed the IDEV draft Inception Report and concluded that it provides an excellent starting point for the actual evaluation exercise about to get started. This amended Inception Report integrates the draft prepared by the IDEV team with updates that include the introduction of the AGRER Team, the proposed AGRER Methodology and the preliminary agenda for meetings in South Africa and Zambia prepared by the IDEV staff.

The AGRER Team consists of 6 experts – all of them with extensive experience in the Region and in Zambia itself, plus significant experience in M & E and an excellent background in all the relevant key sectors along which the Country Strategies were planned and executed. The AGRER Team will work in close consultation with IDEV representatives (who will join the Team in both South Africa and Zambia).

The objective of the mission is to conduct the actual evaluation via extensive document review and targeted interviews planned and organized by the IDEV staff that will take place both in South Africa and in Zambia itself. The AGRER Methodology, included in this paper, summarizes the Action Plan envisaged by the AGRER Team in order to complete this important evaluation assignment.

AGRER Team

The AGRER Team is composed of the following experts:

Paolo LIEBL VON SCHIRACH - KE1 - Team Leader, Private sector and Finance Specialist

He will be in charge of the overall coordination of the team of consultants but will also be responsible as sectoral expert for the Private sector and Finance portfolio. He will also take over green growth crosscutting issues.

Mr. LIEBL VON SCHIRACH is an Italian citizen and American resident. He has an extensive academic background, holding a Doctorate in Political Science (International relations &Law) from the University of Pisa in Italy, as well as a diploma in Law & Political Science from the Scuola Superiore di Studi Universitari S. Anna in Pisa, Italy. Additionally, he obtained a Master Degree in International Relations from the London School of Economics and a Master Degree in Government (Fulbright Scholar) from Georgetown University, Washington, DC – USA.

He has more than 22 years of professional experience working as International Economic & Trade Development Expert, Monitoring & Evaluation Expert, and Regional Economic Development Specialist. He has a thorough experience in policy oriented research/analysis and a wide field practice in Africa: Ethiopia, South Africa, Namibia, Zambia, Zimbabwe, Tanzania, Kenya, Egypt, Mauritius, Botswana,

6

Rwanda, Uganda, Madagascar, Seychelles, Comoros, Malawi, Mozambique, Benin, and Ghana. He has worked for AGRER in 2013 on a previous Evaluation mission in the framework of the Final Evaluation of the Regional Food Security & Risk Management Program for Eastern and Southern Africa” (REFORM), including a field visit in Zambia.

Examples of other key projects relevant to this assignment are:

- Mid-Term Review of the EU-SA Country Strategy (2007-2013) - Evaluation Expert, Economic and Trade Specialist - South Africa - 10/2009;

- Zambia Development Agency (ZDA) - Team Leader & Senior Trade & Investment Expert - Zambia -01/2008-09/2009;

- Final Evaluation of the EU Country Strategy Development Program 2000-2006 for South Africa - Evaluation Expert, Senior Economics & Trade Specialist – South Africa – 06-09/2005.

Vikramdityasing BISSOONAUTHSING - KE2 - Research Officer

Thanks to his long experience working on regional programmes, including in Zambia in the recent years, he will bring his support to the TL through his research and analysis skills on Regional programmes, and will support other sectoral experts in integrating this dimension in their analysis, as well as informing on current programmes of other donors in Zambia. He will also participate in quality-control of Technical reports upon demand of the Team Leader.

Vikramdityasing BISSOONAUTHSING is a Mauritian, holding a master degree in the Development Economics from the Ansted University of Mauritius. He has more than 25 years of professional experience as an international consultant in the Development, Regional integration, Trade Policy and Facilitation sectors. In the course of his career, he undertook a great number of evaluations, drafted macroeconomic reviews, feasibility studies and drafted country strategy papers, in-country reviews and carried out monitoring missions in the both Eastern and South Eastern Africa regions. He has held multiple positions as a team leader, technical advisor, facilitator, country officer and task manager. For instance he worked several years based in Zambia for the Management and execution of the EDF support to the Inter-Regional Coordinating Committee (IRCC) for advisory support to, and effective coordination among, COMESA, EAC, IGAD and IOC on 9th and 10th EDF EU-funded regional programmes (nearly EUR 1 billion), the wider regional integration and trade agenda, and Aid Effectiveness. He has excellent written, management, analytical, communication, reporting, and facilitation skills.

Habtom ASMELASH - KE3 - Agriculture Sector specialist

Mr Asmelash is an agricultural specialist with strong experience as a monitoring and evaluation expert. He served as Technical Adviser & Great Lakes Regional Coordinator, Team leader, Project Manager & Desk officer in long term Project implementation, management & backstopping; as well as Senior monitor for on-going, ex-post national & regional (IGAD & SADC) projects. He is an effective team player who is flexible and adaptable to new and challenging situations. He has worked with this project’s Team Leader on an Evaluation of an EU funded Food Security project in Zambia in 2013. Good knowledge of PCM-Project Cycle Management, good understanding in community based poverty reduction strategies and development (LRRD programmes), capacity of poor communities and participatory based approach. Feasibility, field survey & assessment studies, identification of intervention options that support Farmers’/Growers’ organization and livelihoods to cope with food insecurity. Familiar with the international policies related to nutrition improving approaches through key thematic areas (education, health, gender, social protection, food security & enhanced agriculture).

7

Bernd DRECHSLER – K4 – Governance, Multisector, Education, Social (and Gender) Sector specialist

Mr. Drechsler, an Austrian citizens, holds multiple degrees, including an M.Sc. in Sustainable Development; spec.: Rural Development and Change, London University, School of Oriental and African Studies, Centre for Development, Environment and Policy; 2009-2011. His extensive background includes: (>20 years) at Senior Level in Technical Assistance for long- and short-term projects and programmes in numerous countries mainly the Southern African and Sub-Saharan African Region; extensive Long- and Short Term Experience in Zambia itself, extensive experience (>20 years) in governance projects and programmes as well as institutional capacity building for governmental institutions, ministries, universities, etc.; extensive work experience in programmes focusing on capacity building for higher education, universities and training institutions, extensive experience in gender programmes and projects, including gender components in various development programmes; extensive experience in assessments of institutional set-ups and monitoring & evaluations of institutional strengthening and capacity building programmes. He is very familiar with the concept of Aid Effectiveness and the aims of the Paris Declaration and the Accra Agenda for Action

John MURPHY - KE5 - Transport and Water & Sanitation Sector specialist

John MURPHY is of Irish nationality. He is a civil engineer. He graduated from the Institute of Engineers of Ireland and holds a bachelor degree in civil engineering from the University College in Dublin. He has more than 35 years’ experience of design, supervision, construction and evaluation of multiple infrastructural development projects, including: roads, bridges and water and sanitation sectors. During his career he undertook around 20 Project Evaluations/Mid-term Reviews and held positions as Team Leader for transport and infrastructure-related programmes. He carried out 50 Results Oriented Monitoring missions from 2000 to 2013 in the ACP countries. He has working experience in Zambia, Tanzania, Uganda, Ethiopia, and Botswana. He is very well familiar with the Country Support Strategies, sector documentation and ROM methodologies, as well as with the Project Cycle Management. He is a Director of a consultancy practice in Zambia and has been Project Manager and/or Team Leader for many of its infrastructure design and construction projects.

Charles HAANYIKA - KE6 - Power Sector specialist

Charles HAANYIKA is a Zambian citizen. He is an electrical engineer with 27 years of experience in the power sector in the Zambia region and internationally. He obtained a Master degree in Business Administration Energy and Environment sectors at the University of Twente in the Netherlands. He worked as project engineer, consultant, researcher and director. In the last fourteen years, he has advised clients on electricity supply projects, and the Government of Zambia on various energy policy and regulatory issues. He has undertaken a great number of technical and economic evaluations of power generation and transmission projects. From 1998 to 2001, he worked as Manager – Electricity at the Energy Regulation Board in Zambia. He has been involved in reviewing power purchase agreements between power companies including small hydropower stations and large electricity consumers in Zambia. In addition, he undertook risk assessments of electricity generating plants, study visits to electricity markets in the Southern Africa region.

Backstopping mechanism and support staff for the AGRER Team

The ongoing availability of responsive technical, logistical and administrative support staff from AGRER will be a key element for the successful implementation of the project.

8

For that purpose, AGRER established a permanent management & backstopping unit (MBU) within the premises of AGRER in Brussels and its representation for Indian Ocean based in Antananarivo-Madagascar. The MBU will be led by the Project Director from AGRER, Mr. Marc VANDERLINDEN, who will be assisted by a Junior Project Manager, Mr. Baptiste FORQUY in order to assume the overall management of the contract.

Mr. Marc VANDERLINDEN, of Belgian nationality, has 26 years of professional experience in managing development cooperation projects. He is permanent employee and Project Director at AGRER’s Headquarters since 2004. Prior to this he worked as Technical Assistance on various projects for the EU, the FAO or the BTC in several African and Latin American countries.

He has strong experience in management of World Bank and AfDB projects and especially in agriculture, rural development or Food Security. He is currently Project Director on an AfDB contract regarding the Agriculture Sector Development Study (ASDS) in the Seychelles. In this framework, he led numerous supervision and backstopping missions and is regularly in charge of the quality control of outputs and publishing reports. He has also specific experience in setting-up quality control plans (on reporting activities but also for rural infrastructures and value chains etc.…). He has also been in charge of quality control of hundreds of short-term missions in his position of Deputy Project Director of the Europaid FWC Benef 2009 Lot 1 - Consortium led by AGRER from 2009 to 2013 which includes around 50 short-term evaluation missions. Marc VANDERLINDEN holds a Master Degree in Agronomy and is fluent in French, English and Spanish.

Mr. Baptiste FORQUY, of French nationality, is permanent employee and Project Manager at AGRER since 2011. He holds a Master degree in Dynamics, Management and Tools of Territorial Development (OGDDT) with a major in Project Cycle Management in Developing Countries. Mr. Forquy spent 4 years working in West Africa, first within an EU programme and then for French Embassies.

He has thorough experience in contract and project management, backstopping and quality control of short-term missions and long-term Technical Assistance. From 2011 to 2012 he was responsible for day-to-day activities under the Europaid FWC Benef 2009 Lot 1 - Consortium led by AGRER from 2009 to 2013 and supervised around 30 short-term missions including around 15 evaluation missions. In 2013, he was in charge of the backstopping and quality control of over 30 short-term experts involved in the “Support to Key Accountability Institutions in South Sudan”. In this framework, he led short-term backstopping missions on the field. He is currently managing a World Bank contract in Togo in the framework of the support to the coffee and Cocoa value-chain. Mr. Baptiste FORQUY is fluent in French, English and Spanish.

The tasks of the MBU include but are not limited to:

Managing the TA contract Ensuring overall quality control Mobilising the required expertise in due time (long experience with framework contracts) Providing TA with background information and documentation Assisting TA in technical issues Ensuring financial means to TA team locally Follow-up and control of progress reports Contacts with Contracting Authority Proving assistance to TA for administration and logistics

9

Other tasks not listed or an adaptation of the above tasks may become necessary during the project implementation. For more technical aspects, AGRER can also count on the support from other permanent employee from his staff in HQ Brussels. They are specifically experienced in agricultural and rural development project planning, rural infrastructure and water engineering, etc.

DUGAUQUIER François Director, Manager Development Department

Dipl. Management (Solvay Business School – ULB, 2003)M.Sc. Agricultural Engineering (University of Gembloux, 1987) M.Sc. Sanitation Engineering (University of Gembloux, 1993)

Management, supervision and backstopping of projects & programmes. Project management in developing countries: institutional strengthening, rural development and environmental impact assessments, water management and rural infrastructures.

FEYS Frank Director, Manager Environment Department

M.Sc. in Agriculture - Tropical Agriculture (University of Gent, 1977)

Management, supervision and backstopping of projects & programmes. Project management in developing countries : rural development; natural resources (tropical forestry and wildlife) conservation and management; economic and social forestry

COURTOY Charles Senior Project Manager

M.Sc. Agricultural Engineering (University of Gembloux, 1970)

Project management in developing and emerging countries: rural development, poverty alleviation, land and water resources management; irrigation engineering; agricultural policy advice; regional development

CAMBERLIN Frédéric Junior Project Manager

Ingénieur Agronome du Génie rural (Faculté des Sciences Agronomiques de Gembloux, Belgique 2001) MSc Water Management, Community Water Supply (Silsoe College, Cranfield University, 2001)

Technical studies and project management. Hydro-agricultural development, irrigation, rural infrastructures, watershed management, growth poles development, monitoring & evaluation of projects.

Evaluation Purpose

The purpose of the Zambia Country Strategy and Program Evaluation (CSPE) is two-fold: (i) to inform the development of the subsequent CSP, planned for 2016; and (ii) to contribute to the Comprehensive Evaluation of the Bank's Development Results (CEDR), an initiative which examines the Bank's results and development effectiveness over the past decade. The CEDR involves a synthesis of information from 14 different CSPEs for countries which represent nearly 60% of the Bank's lending operations over the past decade. Zambia has been selected as part of the country sample for the CEDR.

The purpose of this Inception Report is to establish the scope, design, methodology and implementation plan for the Zambia CSPE. The report proceeds by presenting:

(i) the context and background of the evaluation, including the Bank's positioning in Zambia;

10

(ii) an overview of key policies, strategies and operations over the past decade; (iii) the proposed evaluation design and methodology, including the evaluation matrix; (iv) an implementation plan; and (v) a reporting and communications strategy.

2.0 Overview of Zambia's Development Context

2.1 Development Context in Zambia Though landlocked,1 Zambia is endowed with abundant copper and cobalt resources, as well as large tracts of arable land to support agriculture, forestry and wildlife. With a surface area of about 752,000 square kilometres and a population of approximately 13.0 million (2010 Census, estimated 14.5 million in 2013), Zambia has one of the lowest population-to-land ratios in Africa at 17.5 persons per square kilometre. In 2013, it was estimated that 40 percent of the population lives in urban areas in 2013, and that urban areas were growing at an annual rate of 4.4 percent. The main urban concentrations are in the Copper belt mining complex.2

2.1.1 Political Context The 1970s and 1980s were politically turbulent decades for Zambia, exacerbated by falling copper prices (the main export commodity) and rising fuel costs. Austerity measures introduced in mid-1980s compounded food shortages and unemployment, resulting in massive rioting and strikes, which intensified the mood for political change and the abolition of Zambia’s one-party state initiated by President Kenneth Kaunda in 1972. Since then, elections have been held every five years, with the president serving a maximum of two five-year terms. Michael Chilufya Sata of the Patriotic Front replaced the incumbent Rupiah Banda of the Movement for Multi-party Democracy in the September 2011 General Elections. The President died in office in November 2014 resulting in an interim election in early 2015 and a substantial election expected in 2016. Assessment of the governance environment in Zambia is mixed.3 Compared to neighbouring countries, Zambia has one of the lowest female representations in Parliament (gender inequality remains high) and poor open budget policy. Zambia performs well with regard to promoting a competitive business environment and addressing corruption, but has mixed outcomes with respect to press freedom and judiciary independence.

1 Zambia is bordered by Angola in the west, the Democratic Republic of Congo (DRC) to the west and north, Tanzania to the northeast, Malawi to the east, and Zambia, Zimbabwe, Botswana and Namibia to the south. 2 Population growth estimates are derived from World Bank (http://data.worldbank.org/country/zambia), accessed April, 2015. 3 Using the definition by the UNDP (2013), “GOVERNANCE is the exercise of political, economic and administrative authority to manage a nation's affairs. It is the complex mechanisms, processes and institutions through which citizens and groups articulate their interests, exercise their legal rights and obligations, and mediate their differences”.

11

2.1.2 Economic Context

Following substantial economic down-turn during the 1990s,4 the economy has been marked by improved macroeconomic stability, economic liberalisation, and rising investment and production in the mining sector since 2000 (Table 1). Macroeconomic developments since 2012 have weakened on account of rising fiscal imbalance occasioned by declining international copper prices, although growth remains buoyant. Zambia continues to be challenged by the dependence on one major commodity (copper) and pressures on its fiscal management and balance of payments.

Table 1: Zambia – Selected Economic Indicators

Indicator 2002 2004 2006 2008 2010 2012 2013

GDP growth (annual %) 7.3 7.0 7.9 7.8 10.3 6.7 6.7

Agriculture growth -3.5 1.6 -1.4 -2.7 -1.4 6.8 -7.4

Industry growth 11.0 12.7 11.1 5.1 11.2 5.7 7.6

Service growth 6.1 6.7 9.9 11.7 11.5 7.3 8.9

GNI per capita (USD) a/ 370.0 480.0 740.0 1,220.0 1,380.0 1,730.0 1,810.0

Reserves (months of imports) - - 1.9 3.2 3.3 3.4 3.1

Exchange Rate (LCU/USD) b/ 4.4 4.8 3.6 3.7 4.8 5.1 5.4

Inflation (CPI annual %) 22.2 18.0 9.0 12.4 8.5 6.6 7.0

Fiscal balance (% GDP) c/ - -7.3 -5.3 -4.0 -3.9 -5.0 -8.6

Broad Money (% GDP) 18.3 18.8 18.1 19.1 18.4 20.0 21.4

External debt (% GNI) 187.5 149.6 24.9 23.3 29.8 26.6 25.9

Concessional debt (% Ext debt) 59.0 60.4 28.1 28.4 25.5 40.0 43.5

Note: a/ World Bank Atlas Method in current USD. b/ Period average exchange rate. c/ Excluding grants, from IMF (2014), Regional Economic Outlook. Source: World Bank, (http://data.worldbank.org/country/zambia), accessed April, 2015.

2.1.3 Social Development Despite recent improvements in economic performance, Zambia has yet to achieve significant gains in social and human development. The poverty headcount remains high, with about 60.5 percent of the population living below the poverty line and 42 percent considered to be in extreme poverty (2010). The estimated GINI index of income inequality has increased from 42 percent in 2003 to 57 percent in 2010. Life expectancy at birth remains low at 57 years (2012).

2.2 Key Development Challenges

Competitive Business Environment and Access to Finance Recent public resource management reforms have improved Zambia's competitiveness and increased the ease of doing business, making Zambia a favourable place to do business relative to neighbouring countries. In the World Bank's 2010 Doing Business Report, Zambia ranked 6th out of 46 Sub-Saharan African countries.5 However, some key challenges remain, including the high cost of borrowing, low access to credit, high communication and transport costs and burdensome business licensing processes.6

4 High inflation, severe drought, declining export prices, and failed economic policies all took their toll. After steady declines in per capita GDP, Zambia was re-designated a least developed country by the United Nations. 5 World Bank (2010) "Doing Business - Reforming Through Difficult Times" http://www.doingbusiness.org/~/media/GIAWB/Doing%20Business/Documents/Annual-Reports/English/DB10-FullReport.pdf 6 AfDB (2011) "2011-2015 Zambia Country Strategy Paper."

12

Difficulty in accessing credit is particularly prominent with only 37% of Zambians having access to Banks in 2009. Those with access to banking services often have trouble meeting high collateral demands (100-200%), which pose great challenges to individuals who own limited property, particularly women. Microcredit is more readily available, but the terms are often too prohibitive and too expensive compared to commercial loans to be of use.7

Regional Integration

As a land-locked country, promoting regional integration and trade is particularly important to Zambia's development. Although many gains have been made in this regard, including the introduction of a one-stop border post in 2009, the quality of basic infrastructure and low level of trade facilitation capacity limit the realisation of potential benefits.8

Social Inclusion

Despite considerable economic growth, an estimated 42% of Zambians continue to live in extreme poverty.9 This lack of broad-based growth could be attributed to low labour productivity and a concentration of growth in highly capital-intensive, urban-based sectors such as mining, construction and services. In addition, government revenues have not kept pace with increased expenditures, putting pressure on service delivery and pro-poor spending in the health and education sectors.

With respect to gender, Zambia had initially established a "Gender in Development" Division as part of its commitment to mainstream gender in all its development strategies. This division is mandated to coordinate, monitor and evaluate the implementation of the 2000 National Gender Policy and achieve gender responsible development. In 2012, this initiative was expanded with the creation of the Ministry of Gender and Child Development, which seeks to protect and promote women's rights, curb gender-based violence and reduce gender inequalities by making progressive changes to legislation and creating a protective environment. Programmes implemented by the Ministry include: (i) gender and child policy formation; (ii) economic empowerment of women; (iii) gender mainstreaming; and (iv) the prevention of gender-based violence and multi-disciplinary management of survivors.10

Gender gaps continue to exist in Zambia with respect to enrolment in secondary and tertiary education as well as representation of women in parliament (12%) and local government (7%). Finally, particular challenges exist in overcoming the dual application of customary law, which often discriminates against women and is applied alongside civil law.11

Environment and Climate Change

Serious challenges exist with regard to ensuring access to improved water and sanitation sources as well as ensuring environmental sustainability. Little progress has been made in increasing access to improved water and sanitation facilities, rendering achievement of the 7th MDG to be unlikely. Furthermore, rural

7 AfDB (2011). 8 AfDB (2011). 9 UNDP and Government of Zambia (2013) "2013 MDG Progress Report" http://www.zm.one.un.org/mdg_status_zambia 10 Republic of Zambia Ministry of Gender and Child Development, http://www.mgcd.gov.zm/ 11 AfDB (2011).

13

poverty and lack of options for alternative livelihoods other than those which place pressure on natural resources have contributed to environmental degradation.12

Management of environmental risks is increasingly important in Zambia given the growth and potential impact of the mining industry. However, inadequate data and weak monitoring systems, such as forest inventories and data on animal populations, limit informed policy development.13

2.3 Government of Zambia Development Strategies The Government’s strategy during the period under review was guided by the Poverty Reduction Strategy Paper, PRSP (2002), the National Vision 2030, and the Fifth and Sixth National Development Plans. The overall goal of the PRSP was to reduce poverty through achieving broad-based economic growth, diversification in production and exports, improved delivery of social services, and addressing important cross-cutting issues, such as HIV/AIDS and gender equity. Complementary to this initiative, the long term development objective of the National Vision 2030 is for Zambia “to become a prosperous middle income country by the year 2030.” The National Vision has been operationalized through the Government’s five-year development plans, starting with the Fifth National Development Plan (2006-2010) and annual budgets14. The current Sixth National Development Plan (SNDP), covering the period 2011-2015, builds upon the same objectives.

Fifth National Development Plan

This plan (implemented during 2006-2010) represents the Government of Zambia's first building block toward achieving its Vision 2030’s objectives. Its main strategic objectives are to: (a) achieve broad- based wealth and job creation; and (b) support the development of economic infrastructure and human resources.15 The Plan focuses on (i) improving the business and investment climate (Private Sector Development Programme); (ii) providing for the delivery of basic services (Public Service Reform Programme); (iii) strengthening expenditure and financial accountability systems (Public Expenditure Management and Financial Accountability Programme); and (iv) developing the financial system (Financial Sector Development Programme).

Overall, the FNDP underpins the strategic framework for the medium and short term policy actions aimed at poverty reduction. In this respect, the Government’s comprehensive poverty reduction strategy under the FNDP places emphasis on agriculture as the engine for pro-poor GDP growth via: (i) support for small business development via credit schemes; (ii) provision of marketing and technological information; (iii) provision of infrastructures; and (iv) land reform. Priority investment is also expected on social services such as, basic education and health facilities, focused social safety nets, decentralization and the strengthening of the state institutional capacity for implementing poverty reduction programmes. With respect to public financial management, the medium-term objective of the

12 UNDP and Government of Zambia (2013). 13 AfDB (2011). 14 The NDPs are buttressed by the Medium-Term Expenditure Framework (MTEF) which defines the short term macroeconomic framework, aimed at sustaining and building on the gains of the recent past. 15 The plan was accompanied by and the rolling Medium Term Expenditure Framework for 2009 -2011.

14

FNDP is to increase spending under the national budget on poverty reduction programmes by at least 0.5 percentage point of GDP a year.

Sixth National Development Plan

The Government’s current policy is guided by the Sixth National Development Plan (SNDP; 2011-2015), which aims to: (i) accelerate infrastructure development, economic growth and diversification; (ii) promote rural investment; (iii) accelerate poverty reduction and enhance human development. With respect to infrastructure, key investments proposed include the development of feeder roads, water canals, tourist access roads and electricity infrastructure. In addition, enhanced provision of health, education, skills development and social safety nets are planned to enhance skills development and promote poverty reduction for the poorest and most vulnerable. The Government intends to place particular emphasis on stimulating agricultural productivity and promotion of agro-businesses while improving the provision of basic services such as water and sanitation.

The priority growth sectors identified by the SNDP are:

agriculture,

livestock and fisheries,

mining, tourism,

manufacturing,

commerce and trade.

Government strategies respond to the three key development challenges facing the country:

1) poor state of the country’s infrastructure, 2) human resource limitation (compounded by HIV and AIDS), 3) and weaknesses in the land tenure system.

The preparation of both the FNDP and SNDP was participatory in that both were developed through extensive consultations at all administrative levels, including stakeholders from the district, provincial and national planning structures. The process also involved consultation with Sector Advisory Groups, which consisted Government officials, development partners, civil society members, private sector associations. This participatory process enhanced the setting of sector priorities, ensuring improved planning, monitoring and evaluation, and reporting on development outputs and outcomes.

2.4 Zambia's Progress toward the Millennium Development Goals The latest MDG Progress Report for Zambia (UNDP, 2013) indicates that Zambia may need significant reforms and investments in order to achieve the following key MDG targets:

reduce by half the proportion in extreme poverty and halve the Gini-Coefficient (MDG1);

reduce by two-thirds the under-five mortality rate and infant mortality rate (MDG4);

reduce by three-quarters the maternal mortality ratio (MDG5); and

halve the proportion of population without access to an improved water source and an improved sanitation facility (MDG7).

The progress achieved against these goals is described in greater detail below. A full list of Zambia's MDG indicators is provided in Annex A.

15

Broader indicators of development performance confirm that the effect of economic growth on overall poverty reduction has been small, as much of the benefits of growth have accrued to those already above the poverty line. Access to healthcare and improved water and sanitation also remain below those of comparable lower middle income countries.

Zambia’s ranking in the United Nations Development Program (UNDP) Human Development Index (HDI), a composite measure which takes into account life expectancy, access to safe water, access to health services, literacy and nutrition, is below the Sub-Saharan Africa average. In 2012, Zambia's HDI was 0.448, and ranked 163 out of 187 countries with comparable data.16 Overtime, the HDI for Sub-Saharan Africa increased from 0.365 in 1980 to 0.463 in 2011, and Zambia has fallen below the regional average.

Reduction in Poverty (MDG1)

Despite recent improvements in economic performance, Zambia has yet to achieve significant gains in the social and human development aspects of the MDGs. The poverty headcount remains high, with about 60.5 percent of the population living below the poverty line and 42 percent considered to be in extreme poverty (Table 2). Moreover, the absolute number of poor has increased from about 6 million in 1991 to 7.9 million in 2010, primarily due to population growth. The urban picture is far better than the rural: in the Copper-belt and Lusaka provinces, for example, poverty incidence is fairly low (22 percent and 34 percent respectively), whereas in the rest of the country, which is dominated by agriculture, poverty rates are greater than 70 percent.

Almost 90 percent of Zambians who live below the extreme poverty line are concentrated in rural areas, and the poverty gap ratio (a measure of how far average incomes fall below the poverty line) is far higher for the rural population than their urban counterparts (20 percent and 3.7 percent, respectively). Poverty incidence among female-headed households was slightly higher than male –headed households (62.5 percent versus 60 percent). In addition, Zambia’ Gini coefficient of 0.65 puts it among the most unequal countries of the world.

Table 2: Zambia – Dimensions of Poverty

Indicator Zambia Lower MIC Low-Income

GNI per Capita USD17 1160.00 >1025.00 < 1025.00

UNDP HDI (2012) 0.45 (Global) 0.69 0.45

Headcount Ratio18 60.50 Cameroon – 39.90 Ethiopia – 38.90

Ghana – 28.50 Uganda – 24.50

Under 5 year mortality19 119.00 69.00 108.00

Maternal mortality20 591.00 300.00 590.00

Life expectancy 49.00 65.00 59.00

Gender Inequality Index21 0.62 (Global) 0.46 0.57

16 The value of HDI ranges from 0 (worst) to 1 (best); refer to: UNDP HDI, 2013 (http://hdr.undp.org/en/media/HDR_2013_EN_complete.pdf). 17 Classification of middle income country (MIC) and low-income country (LIC) is based on World Bank July 2012 18 Based on national poverty line 19 Per 100,000 live birth 20Per 100,000 live birth 21 A composite measure reflecting inequality in achievements between women and men in three dimensions: reproductive health, empowerment, and the labor market (http://hdr.undp.org/en/media/HDR_2013_EN_complete.pdf).

16

Child and Maternal Mortality (MDG 4&5)

While the most recent Zambia Demographic and Health Survey (2007) shows notable reductions in maternal and under-five mortality rates during 2002-2007, these levels are still above those for comparator MICs. Furthermore, the annual rates of reduction of 2.1 percent for under-5 mortality and 2.5 percent for maternal mortality for 1990-2011 are less than half of the annual rates of reduction required to achieve the MDG targets. Mortality rates of children under 5 years dropped from 190.7 deaths per 1,000. Similarly, the infant (death before first birthday) mortality rate fell from 107.2 deaths per 1000 live births in 1992 to 76.2 deaths per 1000 live births in 2010. Child mortality has declined by almost 30 percent since 1992 but remains unacceptably high.

In terms of maternal mortality, the UNDP estimates that 38 mothers die each month in Zambia as a result of complications of pregnancy or childbirth. The number of mothers dying during pregnancy and childbirth has decreased from 649 per 100,000 live births in 1997 to 483 in 2010.

Access to Safe Drinking Water and Sanitation Facilities (MDG7)

Despite making progress in increasing access to safe drinking water and sanitation facilities since 1990, Zambia’s rate of progress remains too slow for achieving the MDG (Figure 1). The proportion of the population without access to improved water source decreased from 51 percent in 1991 to 36.9 percent in 2010 (far below the target of 25.5 percent by 2015). Similarly, the proportion of the population without access to an improved sanitation facility worsened from 26 percent in 1991 to 67.3 percent in 2010.22 Zambia is unlikely to meet the MDG target of 13 percent by 2015.

Figure 1: Zambia Access to Improved Drinking Water and Sanitation Facility

22 It is important to note that a significant part of this decline may be explained by the fact that the definition of improved sanitation changed in 2010 from including to excluding pit latrines without a slab.

0

10

20

30

40

50

60

70

80

1991 1993 1996 1998 2002 2004 2006 2010

Improved Water (%)

Improved Sanitation (%)

17

3.0 Overview of Bank Support to Zambia

3.1 The Bank's Country Strategies in Zambia The Bank's lending and non-lending operations in Zambia over the period of 2002-2014 have been guided by three Country Strategy Papers (CSPs):

1) the 2002 CSP and its update in 2006; 2) the first Joint Assistance Strategy in Zambia (2007-2010); and 3) the 2011-2015 CSP.

Table 3 outlines the strategic objectives of these CSPs mapped against the contemporary national development strategies. Broadly speaking, the Bank's strategy in Zambia has shifted over the evaluation period to place greater emphasis on the promotion of economic growth through infrastructure development, particularly roads, power generation and power supply. This emphasis complements the gradual shift in Zambia's national strategies toward a focus on economic growth and diversification through infrastructure development and leverages the Bank's comparative advantage in infrastructure development.

CSP 2002-04

The CSP covering the period 2002-04, was approved by the Bank’s Board on July 23, 2003, and later updated to cover the period 2004-06 while the Bank participated in the preparation of the first Joint Assistant Strategy of Zambia (JASZ). The 2002 CSP and its 2006 update responded largely to the Government’s 2002 Poverty Reduction Strategy Paper (PRSP, 2002-04) and the lessons from the implementation of the Bank’s Country Strategy Paper (CSP) for 1999-2001. The 2006 update confirmed the Bank’s medium term strategy in Zambia as defined in the 2002 CSP.

The Bank's strategic objective for its 2002-2004 CSP is to promote poverty reduction through improved social service delivery and economic growth. With respect to economic growth, emphasis was placed on promoting the development of the agricultural sector through: (i) expanding agricultural infrastructure; (ii) making existing infrastructure less susceptible to climate change; and (iii) implementing out-grower schemes encompassing the development of new farm blocks, provision of credit to farmers, guaranteed market access and crop storage. With respect to social service delivery, emphasis was placed on: (i) access to water and sanitation services, particularly in peri-urban and rural areas; and (ii) promoting child welfare through increasing access to health and education services and building capacity to address the needs of vulnerable children. Finally, with respect to governance, the strategy provided for policy based support in order to help manage public debt, reduce interest rates and allow for further decentralisation of governance.

The First Joint Assistance Strategy for Zambia 2007-2010

The first Joint Assistance Strategy for Zambia 2007-10, signed in April 2007 by 16 cooperating partners of Zambia,23 constituted the CSP for the Bank during 2007-10.24 The JASZ derived from the National

23 They are the AfDB, Governments of Canada, DfID, Denmark, Finland, Germany, Ireland, Italy, Japan, Netherlands, Norway, Sweden, USAID, the EU, the UN System, and the World Bank.

18

Vision 2030, the Fifth National Development Plan 2006-10 (FNDP) and the rolling Medium Term Expenditure Framework 2006-08, was informed by an extensive consultation process. The Bank’s support was based on the first two pillars of the JASZ, namely, Support to Infrastructure Development and Support to Good Governance, the latter through provision of budget support. It responded to the development challenges articulated in the FNDP, and the past interventions of the Bank in Zambia.

The 2007-2010 JASZ retains the strategic objective of the FNDP, which is to improve the wellbeing of the Zambian population. The Bank narrows its support under the JASZ to two pillars: (i) support to infrastructure development; and (ii) support to good governance. Under the first pillar, the Bank retained its previous operations in water supply and sanitation and agriculture, but, at the regional level, added new operations in the development of regional transport corridors and the promotion of regional power interconnectivity. The Bank's support to good governance consisted of continued support to the promotion of transparency and accountability in management of public resources through policy-based lending targeting: (i) domestic debt management; (ii) public procurement capacity; and (iii) the strengthening of accountability and transparency mechanisms. In addition, the JASZ added a new emphasis on the implementation of structural reforms to improve the business environment and promote economic diversification.

The 2011 CSP

The 2011 CSP was prepared before the country’s tripartite elections in 2011. It attempts to break away from the previous CSPs because past Bank interventions were considered to lack a holistic perspective, responding solely to sectoral challenges as defined in the NDPs. The renewed approach aims at improving alignment with the Bank’s Medium Term Strategy (MTS), as elaborated in the Southern Africa Regional Integration Strategy (RISP), as well as directly responding to the Zambia’s renewed emphasis on the diversification of the economy. The 2011 CSP is anchored on two strategic pillars: (i) supporting economic diversification through infrastructure development, and (ii) building capacity for transparent economic and financial governance.

Strategic Objective

The strategic objective of the 2011-2015 CSP is to promote economic growth, economic diversification and poverty reduction. In order to achieve this objective, the Bank has channelled its support through two pillars of activity: (i) Infrastructure Development; and (ii) Economic and Financial Governance. With respect to infrastructure development, the Bank places emphasis on promoting regional integration through the building of new roads, rehabilitation of existing roads, development of regional transport corridors and the promotion of national and regional power connectivity. For economic and financial governance, the Bank has chosen to continue its support for building capacity for public financial management, and strengthening accountability mechanisms and public procurement capacity. However, under the 2011-2015 CSP, renewed emphasis is placed on the promotion of gender-responsive budgeting, increasing access to finance and leveraging private sector resources through Public Private Partnerships (PPPs).

Table 3: 2002-15 CSP strategic pillars and objectives

Bank Strategy National Development Plans

2002-2004 Country Strategy Paper Poverty Reduction Strategy Paper (PRSP; 2002)

24 The 16 signatories of the JASZ committed themselves to either use the document as a replacement for their current strategies for development cooperation in Zambia or base their maintained country strategies on it. The Bank’s participation in the JASZ is part of its commitment to the Paris Declaration on harmonization, alignment and management for results.

19

Strategic Objective Strategic Objective

Poverty reduction through economic growth and social service delivery

Poverty Reduction and Broad-based Economic Growth

Areas of Emphasis Areas of Emphasis

Strengthening and climate-proofing agricultural infrastructure.

Increasing access to safe water and sanitation.

Increasing access to health and education services, particularly for vulnerable children.

Management of public debt

Decentralisation of governance

Diversification of production and exports through support to the agricultural sector

Improved delivery of basic social services

Cross-cutting emphasis on HIV/AIDS and gender

2007-2010 Joint Assistance Strategy for Zambia Fifth National Development Plan (2006-2010)


Increased wellbeing of the Zambian population Broad-based wealth and job creation


Strengthening of agricultural and water and sanitation infrastructure.

Development of regional transport corridors.

Regional power interconnectivity.

Promotion of domestic debt management.

Development of public procurement capacity.

Strengthening of accountability and transparency mechanisms.

Development of economic infrastructure in the agricultural sector.

Human resources development.

Improved delivery of basic social services.

Accountable and transparent public expenditure management

2011-2015 Country Strategy Paper Sixth National Development Plan (2011-2015)


Economic Growth, diversification and poverty reduction

Poverty reduction through economic growth and diversification


Development and rehabilitation of national and regional transport infrastructure.

Strengthening power generation capacity and national and regional power connectivity.

Increased capacity for efficiency, transparent and accountable public financial management.

Private sector regulatory reform

Increased access to finance.

Infrastructure development, including feeder roads, water canals, access roads and power infrastructure.

Promotion of agri-business.

Basic Service Delivery.

Sources: The 2002 CSP and its update in 2006, the first Joint Assistance Strategy of Zambia 2007-10, and the 2011-15 CSP.

3.2 Bank Corporate Policies and Strategies Over the full evaluation period (2002-2014), the Bank's activities in Zambia have also been guided by a series of macro, regional, sector and thematic-level policies and strategies with which the Bank's interventions are expected to be consistent. One part of the evaluation process will involve determining the extent to which the Bank's operations in Zambia have aligned with these key documents. At the macro-level, the Bank's interventions have been guided by the Bank's Ten Year Strategy (TYS; 2013-2022), Medium Term Strategy (MTS; 2008-2012) and the Strategic Plan of the AfDB Group for 2003-2007, adopted in 2002. The purpose of these documents is to outline the Bank's strategic

20

objectives as well as the operational priorities which are then applied to the Bank's operations in Regional Member Countries. In addition to these strategies, the Bank has also identified an Organizational-level Results Measurement Framework (RMF) to help assess the Bank's contribution to Africa's development at various levels. The current version of the One Bank Result Measurement Framework (2013-2016), was adopted in 2013. A brief overview of these macro-level documents is provided in Table 4, below.

Table 4. Key Macro Policy Documents of the African Development Bank Group (2004-2014).

Document Key Policy / Strategic Directions

Ten Year Strategy (2013-2022)

Introduction of two strategic objectives for the Bank's operations: (i) inclusive growth; and (ii) green growth.

Identification of five operational pillars for which the Bank has a comparative advantage (infrastructure, regional integration, private sector development, skills development and governance) as well as crosscutting areas of emphasis, including gender, agriculture and fragility.

Emphasis on operational mechanisms for maximizing the Bank's value, including: (i) selectivity; (ii) leveraging; (iii) dialogue; and (iv) results based management

Medium Term Strategy (2008-2012)

Identified poverty reduction through stronger and more equitable economic growth as a core objective for the Bank's operations.

Introduced greater selectivity in sectoral operations targeting infrastructure, governance, private sector operation, higher education, technology and vocational training.

Strategic Plan for the AfDB Group (2003-2007)

Identified four core objectives for the Bank's operations: (i) productivity growth and poverty reduction; (ii) Development results within the MDG framework; (iii) Stronger partnerships with multilateral, bilateral and regional agencies; and (iv) Increased market access and economic diversification.

One Bank Results Measurement Framework (2013-2016)

A management tool for tracking progress against the Bank's strategic objectives at the corporate level.

Assisting the Bank to deliver better value for money while tracking results and development effectiveness throughout the project life cycle.

With regard to regional-level operations, the Bank's activities have been guided by the current Bank Group Regional Integration Policy and Strategy (RIPS; 2014-2023) as well as the Regional Integration Strategy (RIS; 2009-2012) and the Strategic and Operational Framework for Regional Operations (2008), described in further detail in Table 5, below.

Table 5. Key African Development Bank Group Regional Policies and Strategies

Document Key Policy / Strategic Directions

Bank Group Regional Integration Policy and Strategy (2014-2023)

Identifies the creation of larger, more attractive markets and supporting infra-African trade as the strategic objective for the Bank's regional operations.

Delineates pillars of operation, including: (i) supporting regional infrastructure development; (ii) enhancing trade and industrial development; and (iii) strengthening country and regional mechanisms and institutional capacities.

Regional Integration Strategy (2009-2012)

Identifies four strategic outcomes underpinning the Bank's regional operations: (i) increased economic competitiveness; (ii) enhanced intra-African trade and African presence in the global marketplace; (iii) establishment of a regional African "voice;" and (iv) more efficient provision of regional public

21

goods.

Identifies two main pillars of operations: (i) regional infrastructure; and (ii) institutional capacity building.

Strategic and Operational Framework for Regional Operations (2008)

Identifies a clear need to address donor funding gaps for regional development issues and the development of regional institutions.

Proposes a strategic niche for the Bank in terms of addressing regional infrastructure needs as well as capacity development for the provision of regional goods.

At the regional level, the Bank has also identified a Regional Integration Strategy Paper for Southern Africa (2011-2015) which covers 12 Regional Member Countries: Angola, Botswana, Lesotho, Madagascar, Malawi, Mauritius, Mozambique, Namibia, South Africa, Swaziland, Zambia and Zimbabwe. The two core pillars of the strategy seek to promote the objectives of the COMESA-EAC-SADC (CES) Tri-partite Agreement through regional infrastructure development and capacity building.

With respect to sectoral and thematic strategies and policies, a set of nearly 30 documents have been identified which are relevant to the Bank's operations in Zambia. These documents are presented below.

Table 6. AfDB Thematic and Sectoral Strategies

Thematic Area Strategy / Policy Document

Infrastructure AfDB Group Energy Sector Policy (2012)

Policy for Integrated Water Resources Management (2012)

Transport Sector Policy (1993; currently being revised)

Agriculture

Framework and Guidelines on Land Policy in Africa (2010)

2010-2014 Bank Group Agriculture Sector Strategy

Improving Statistics for Food Security, Sustainable Agriculture, and Rural Development - An Action Plan for Africa (2011-2015)

Agriculture and Rural Development Sector AfDB Group Policy (2000)

Social AfDB Human Capital Strategy for Africa 2014-2018

Strategy for Higher Education, Science and Technology (2006)

Education Sector Policy Paper (ESPP) (1999)

Private Sector

Private Sector Development Policy of the AfDB Group (2013)

AfDB Group Private Sector Development Strategy (2012-2017)

Microfinance Policy and Strategy for the Bank Group (2006)

Multi-sector and Governance

Bank Group Strategy on Program-Based Operations (PBOs) (2012)

Governance Strategic Framework and Action plan (GAP II) (2014-2018)

Governance Strategic Directions and Action Plan (GAP) 2008-2012

Strengthening Country External Audit Systems in Africa - A Joint Strategy of the African Development Bank and the World Bank (2010)

Bank Group Policy on Good Governance (1999)

Bank Group Policy on Poverty Reduction (2004)

Gender

2014-2018 Bank Group Gender Strategy.

Gender Equality and Women's Empowerment: An Updated Gender Plan of Action 2009-2011.

Checklist for Gender Mainstreaming in the Water and Sanitation Sector (2010)

Checklist for Gender Mainstreaming in Governance Programmes (2010)

Checklist for Gender Mainstreaming in the Infrastructure Sector (2009)

Checklist for Gender Mainstreaming in the Education Sector with the Special Focus on Higher Education, Science and Technology Sub-Sector (2009)

22

Environment Environmental Safeguards Policy (2004)

Climate Change Action Plan (CCAP) 2011-2015

3.3 The Bank's Portfolio in Zambia (2002-2014)

The Bank’s portfolio in Zambia is characterized by two types of interventions: the lending operations which involve investments in projects and programs; and the non-lending operations, which include studies (referred to as economic and sector work), training and capacity development, partnership, and policy dialogue.

3.3.1 Lending operations Between 2002 and 2014, the Bank' Board approved a total of 39 projects amounting approximately to 660.15 million UA, excluding cancellations. This portfolio includes 32 national operations which amount to 504 million UA and 7 multinational operations covering regional integration activities between Zambia and neighbouring countries (Zimbabwe, Botswana, Mozambique, and Malawi) amounting to 155.4 million UA. The Bank also implemented three emergency support packages responding to climate-related disasters (floods and drought) amounting to 1.3 million UA. Of these projects, 19 are completed or closed, 11 are still ongoing and the remaining 9 have been approved but are not yet effective or implementation has not yet started. The portfolio under review also includes a project approved in 2000, but became effective only in 2002 and closed in 2010.

Most Bank financing has been done through the ADF window, accounting for 58% of commitments, followed by the ADB window with 32%. The remaining financing has been provided through the strategic climate fund (3.7%), the GASF (3%) and others (2.8%). Projects financing is largely in the form of loans or grants. Thus, 8% of total net commitments are grants and the remaining 92% are loans.

Power, Transport, Agriculture and Environment

To date, the majority of the Bank's commitments have been in the Power sector, Transport sector and Agriculture and Environment sector (27.24%, 19% and 14.34% respectively). The majority of projects implemented by the Bank have been in the agriculture, finance and water supply and sanitation sectors (9, 7 and 5 projects respectively). Together, these three sectors account for 11 of 19 projects which have been closed or completed. A breakdown of the Bank's lending activities by sector is provided in Table 7, below as well as figures 2 and 3, below.

Table 7. Breakdown of the Bank's Portfolio in Zambia by sector

Zambia Portfolio by sector (2002-2014)

Sector Total Commitments (UA'000)

Percentage No. of Ops CLSD & COMP

OnGo AVDP

Infrastructure Development

Water & Sanitation 85,199.50 12.91 5 3 1 1

Agric. & Environment 86,910.00 13.17 9 3 5 1

Power 183,942.06 27.86 4 4

23

Education 22,220.00 3.37 1 1

Transport 127,075.01 19.25 5 1 3 1

Governance (Economics & Finance)

PSD 71,707.20 10.86 7 5 1 1

Multi-sector 81,801.00 12.39 4 4 1

Emergency 1,301.87 0.20 3 3

Total 660,156.63 100.00 38 19 11 9

Figure 2: Net commitment by sector 2002-2014

Figure 3: Number of projects by sector

The Bank’s national portfolio in Zambia has grown over the years; under the first CSP (2002-2006) a total number of 8 projects were approved amounting to 111 million UA, against 13 for the 2007-2010 CSP (164 million UA) and 17 for the 2011-2015 CSP (399 million UA). A substantial amount of the Bank’s investment in Zambia occurred during the ongoing 2011-2015 CSP period, whereas very few projects were approved in 2005, 2007 and 2009. This trend is presented in Table 4, below.

Figure 4: Bank net commitment by year in Zambia 2000-2014

183.94

127.08

81.80

88.20

85.20

71.722.2

Net Loan by sector (UAM)

Power

Transport

Multisector

Agric &Environment

Power, 4

Transport, 5

Multisector, 4

Agric & Envir., 12

WSS, 5

PSD, 7

Social, 1

Nbr of projects by sector

60 0

21 20

0

64

0

61

1

102

66

94

163

76

5.8

21.3 20.1

0.3

16.0 15.2

0.5

34.0 33.0 31.3 32.6

20.5

0

20

40

60

80

100

120

140

160

180

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014Total Approval… Average

24

The Bank's Multinational Projects in Zambia

The Bank also financed several multinational operations. Most of these projects were implemented in the Transport sector (5 projects) for a total commitment of 127 million UA (81.7%), followed by the Power sector(1) with a total commitment of 25.1 million UA (16%) and finally the Agricultural sector (1) with 3.2 million UA (2%). Zambia is a land-locked country; therefore regional integration has featured as a priority among the Bank's interventions as a means of promoting economic development and regional trade.

3.3.2 Non-lending activities In addition to its lending operations, the Bank was involved in several non–lending operations in Zambia at both national and multinational levels. Table 8, below, shows the main ESWs undertaken by the Bank in Zambia and a brief description of their objectives.

Table 8: The Bank’s ESWs and operational studies in Zambia – 2002-2014

Title of review Sector and year Description

Zambia multi sector Gender profile

Agriculture (2006)

The Multi-sector Country Gender Profile (MCGP) intends to set a platform for all stakeholders to understand the critical gender development issues in the current economic and development planning process in Zambia.

SADC North- South Transport corridor improvement study

Transport (2006)

The objective of the Study was to prepare the economic feasibility, detail engineering design, and tender documents for the Kazungula Bridge over the Zambezi River and other key infrastructure facilities along the SADC north-south corridor within Botswana and Zambia.

Zambia Post harvest losses needs assessment report

Agriculture (2008)

This report describes the post-harvest situation in Zambia for selected commodities (cassava, maize, rice, sorghum, beans, groundnuts, fruits [mangos and pineapples], vegetables [tomatoes], animal products and fish) including key constraints, risks and opportunities. The objective is to suggest possible priority interventions and operations to be proposed to the AfDB and other DPs.

Nacala Corridor studies

Transport (2009)

The specific objective of this study was to determine the best technical, economic, environmental and socio-economic options for: (a) the upgrading of the Cuamba-Mandimba road in Mozambique; (b) the rehabilitation of the Mangochi-Liwonde Junction road in Malawi; and (c) preliminary design of one-stop border post layouts at the common borders. The study involves the preparation of detailed engineering designs, cost estimates and bidding documents for (a), (b) and (c), above, as well as preparation of a legal framework for extra-territorial operation of the one-stop border posts.

Jobs, prosperity: building Zambia’s competitiveness

Multi-Sector (2010)

The overarching question guiding the Jobs, Prosperity and Competitiveness (JPC) initiative is "what would it take to improve Zambia’s competitiveness?" This analysis proceeds from the assumption that economic explanations alone and the prescriptions that emerge from them are incomplete.

25

Title of review Sector and year Description

Zambia irrigation and subsector study

Agriculture (2010)

The objectives of this sector study were to: (i) provide an analysis of a representative sample of smallholder and commercial irrigation schemes in Zambia, from technical, financial, economic, commercial, institutional, social and environmental perspectives; (ii) analyse the factors that determine the levels of technical and economic performance of irrigation schemes; and, based on this, (iii) identify and quantify additional irrigation potential that could be developed in the medium term and proposed interventions for Bank’s funding.

Zambia Development effectiveness review

Multi-Sector (2013)

This study provides an overview of Zambia’s development progress and of how the African Development Bank (the Bank or AfDB) is contributing to this progress. This report takes inclusive and sustainable growth as its guiding theme.

3.4 A Theory of Change for the Bank's Activities in Zambia (2002-2014) In order to structure and inform the methodology of the evaluation, IDEV has identified a Theory of Change (ToC) for the Bank's operations. This Theory brings together five sources of information to help explain the Bank's performance in Zambia:

(i) the 2002-2004 CSP (ii) the 2007-2010 JASZ (iii) the 2011-2015 CSP (iv) the Bank's corporate policies and strategies (v) a file review of the project portfolio. The ToC will be used to help structure the evaluation questions, methodology, indicators and data collection activities. The data collected will be triangulated against the evaluation matrix to test the ToC and assess the relevance, effectiveness, efficiency and sustainability of the Bank's intervention and assess the management of the Bank's portfolio in Zambia.

This ToC illustrates the main challenges, activities and outcomes associated with each of the major sectors of the Bank's intervention along with an indication of how each of the results levels contribute to higher level results and changes. Contributions and linkages in the results chain are identified moving from the level of activities to ultimate outcomes, but it is also possible to identify horizontal relationships among the outcomes as well as feedback loops. The ToC envisions an intervention rooted in inclusive growth and poverty reduction through economic growth and diversification and support for transparent, accountable and inclusive governance. The earliest CSP under examination possessed a slightly different focus, emphasizing poverty reduction through more equitable access to basic infrastructure and services including water, sanitation, health and education. Particular emphasis was also placed on ensuring child welfare. More recent CSPs, however, have placed more emphasis on promoting economic development through infrastructure development and government reform, an approach which better reflects the Bank's corporate strategies. The proposed ToC is presented in Figure 5, below.

26

Based on a review of the Bank's Country Strategies and Program documents, it is hypothesized that the ultimate outcome of the Bank's intervention is poverty reduction through inclusive economic growth. Inclusive economic growth is to be achieved through addressing the challenges faced with respect to key drivers of growth, namely poor infrastructure, limited institutional and public financial management capacity and limited access to basic services. Of particular importance is promoting improved regional integration among Zambia and neighbouring countries, both in terms of infrastructure and regional trade. While this emphasis on promoting Zambia’s regional integration through infrastructure development has an obvious economic rationale, and further resonates with the Bank’s priority setting on regional infrastructure development, it will be observed that the newest arrangement, being the COMESA-EAC-SADC Tripartite Free Trade Area, might hold more challenges than initially thought. The next steps of the evaluation would appraise this aspect so as to inform the ToC if the Bank’s support to infrastructure development in Zambia in a context of regional integration bears credible value-addition or whether the démarche should be situated or augmented otherwise. In any case it is taken that some of the immediate and intermediate outcomes include: (i) for infrastructure, enhanced access to markets and services and reduced cost for producers and consumers, (ii) improved access to market for poor and marginalized farmers, including women, (iii) reduced cost of doing business, and (iv) improved access to social services, regional connectivity, and fiscal space. A key assumption is that the growth will be inclusive and environmentally friendly, being cognizant to the cross-cutting imperatives of improved gender relations and adequate resource flow (both external and domestic). Gaps for evaluative information in the ToC approach In the mapping pf the ToC below, as per Fig. 5, elements which are not taken into consideration are: 1) the longer term impact , MDG considerations, and lessons for post-2015 sustainable development

goals, all of which would eventually inform the subsequent formulation of the bank’s support 2) risk assessment in terms of match or mismatch with the drivers of change for Zambia ; actually the

Theory of Change would itself justify its use as driving the valuation if convergence shown with recognised drivers of change for Zambia

27

Figure 5: A theory of change for the Bank’s assistance to Zambia 2002-2014

28

4.0 Evaluation Scope and Methodology

4.1 Evaluation Scope and Objectives The purpose of the Zambia Country Strategy and Program Evaluation is two-fold:

1) Identify findings, recommendations and lesson learned to inform the development of the Bank's next Country Strategy for Zambia; and

2) Provide necessary information to support the final synthesis of the Comprehensive Evaluation of the Bank's Development Results (CEDR).

This evaluation will cover the Bank's lending and non-lending operations in Zambia over the period of 2002-14. The lending assistance to be assessed will include both public and private sector loans and grants, while the non-lending activities will include the development of strategies, policy dialogue and advice, knowledge management, and outreach and partnership with the Zambian government.

In total, this evaluation will cover the Bank's interventions under three Country Strategy Papers (CSPs), namely:

The 2002-2006 CSP;

The 2007-2010 CSP/JASZ; and

The 2011-15 CSP. The universe for this evaluation will include all projects approved between 2002 and 2014 which were not terminated, cancelled or abandoned. As noted above, this includes a total of 32 domestic operations and 7 multinational operations targeting regional integration. A full list of projects to be assessed as part of the evaluation is presented in Annex A. These projects will be examined via a two-step approach:

1. First, the full portfolio will be subject to a document review which assesses: (i) project design quality; (ii) quality of project implementation; and (iii) the achievement of results – especially outputs and outcomes.

2. Second, selected completed or near-completed projects will be subject to more detailed

assessments, including interviews with key stakeholders and case studies.

In addition to these projects, the evaluation will consider the quality of non-project activities, including policy dialogue and economic and sector work (knowledge and analytical work).

4.1.2 The Evaluation Issues and Questions The Evaluation will primarily be guided by the OECD-DAC Criteria for International Development Evaluation, focusing on the issues of: (i) relevance; (ii) effectiveness; (iii) efficiency; and (iv) sustainability.25 In order to allow for an assessment of the Bank's organizational effectiveness, this

25 It is not foreseen that sufficient time of resources will be available to fully assess the impact of the Bank's operations.

29

evaluation will also assess the extent to which the Bank's management and operating practices, including the adoption of cross-cutting themes, design and delivery mechanisms and management for results has contributed to the achievement of results. The evaluation issues and questions are identified in table 9, below. Table 9. Evaluation Questions - Zambia CSPE

Evaluation Issue Specific Evaluation Questions

1. Relevance

EQ#1.1: To what extent are the Bank's Country Strategies and operations for Zambia aligned with the development needs, challenges and priorities of the country as well as the needs of the expected beneficiaries?

EQ#1.2: To what extent are the Bank's Country Strategies for Zambia aligned with the Bank's own strategies, priorities and policies?

EQ#1.3: To what extent are the Bank's interventions in Zambia aligned with the Bank's Country Strategies for Zambia?

2. Effectiveness

EQ#2.1: To what extent have the Bank's interventions achieved their objectives is terms of the delivery of outputs?

EQ#2.2: To what extent have the Bank's interventions contributed to the achievement of development results in terms of expected outcomes?

EQ#2.3: To what extent have the Bank's interventions contributed to the achievement of development results for the country, including intended and unintended impacts?

EQ#2.4: To what extent have the Bank's interventions benefitted target group members?

3. Efficiency

EQ#3.1: To what extent were the Bank's interventions delivered in an efficient and economical manner (were resources economically converted into outputs?)?

EQ#3.2: To what extent are the Bank's interventions implemented in a timely manner which is consistent with best practices?

EQ#3.3: To what extent did the Bank's intervention achieve good value for money (cost-benefit)?

4. Sustainability

EQ#4.1: To what extent are the results achieved likely to continue once the Bank's interventions are completed?

EQ#4.2: To what extent has the Bank sought to mitigate risks to the sustainability of results as part of the project design process?

5. Cross-cutting Themes

EQ#5.1: To what extent do the Bank's interventions inclusive in terms of geography, age and gender?

EQ#5.2: To what extent are the Bank's interventions environmentally sustainable and support the transition to green growth?

6. Design and Delivery

EQ#6.1: To what extent has the quality of CSPs been satisfactory? Are the Bank's interventions coherent and well-coordinated internally?

EQ#6.2: To what extent have the Bank's interventions been selective and strategic?

EQ#6.3: How has the Bank adapted its interventions to implementation challenges within the country?

EQ#6.4: To what extent has the Bank's engagement been informed by policy dialogue with national actors, development partners and other interested stakeholders?

EQ#6.5: How well has the Bank leveraged its financial resources within the country through mechanisms such as productive partnerships and co-financing?

EQ#6.6: To what extent has the Bank acted as a knowledge broker, advisor and convener? To what extent have the Bank's interventions been informed by the Bank's analytic work as well as that of other development partners?

EQ#6.7: To what extent has the Bank cooperated with other development partners to ensure complementarity and reduce overlaps?

7. Management EQ#7.1: To what extent has the Bank's supervision of its interventions been effective?

30

Evaluation Issue Specific Evaluation Questions

for Results EQ#7.2: How has the Bank used supervision and results information to guide the management of its interventions?

EQ#7.3: To what extent has the Bank successfully implemented a performance measurement strategy which focuses on the achievement of outputs, outcomes and impacts?

EQ#7.4: To what extent have lessons learned been identified and incorporated into subsequent CSPs?

EQ#7.5: How has the Bank worked with the Government of Zambia to enhance its own systems of Results-Based Measurement?

EQ#7.6: To what extent has the Bank identified and monitored potential implementation risks?

4.2 Proposed Methodological Approach The conduct of the CSPE for Zambia will involve the analysis of primary and secondary sources of information, both qualitative and quantitative, which will be triangulated to address the evaluation questions. Secondary sources of evidence will include:

documentary evidence, including Bank policies, strategies and country development plans;

project approval documents;

official statistics available at country level;

project supervision and completion documents; and

relevant literature, including evaluations conducted by other development partners and structural development indicators compiled by other organizations (e.g. CPIA assessments).

Primary sources of evidence will include:

interviews with key stakeholders both within the Bank as well as country counterparts;

Site visits and a Project Results Assessment of selected projects; and

To the extent possible, focus groups or surveys of beneficiaries for selected projects. This evidence will be mapped to the indicators and criteria identified in the evaluation matrix provided in section 4.2.4. This triangulated data will be cross-references against the available literature to confirm the findings and address any inconsistencies in the data. This triangulated information will be used to identify ratings at both the project and country levels for each of the evaluation criteria using a six-point scale ranging from "highly satisfactory" to "highly unsatisfactory." This approach is not completely consistent with the Bank's approach to rating the performance of projects, which is typically conducted on a 4-point scale. Accordingly, where necessary, any existing performance ratings or quantitative targets assessed on a 4-point rating scale will be converted to a 6-point rating as provided in Table 10, below.

31

Table 10. Proposed Conversion of Rating Scales

6-Point Rating System

4-Point Rating System Achievement of Quantitative

Targets (where relevant)

6 Highly Satisfactory 1 Highly Satisfactory More than 90%

5 Satisfactory 2 Satisfactory Between 65 and 89%

4 Moderately Satisfactory

3 Moderately

Unsatisfactory 3 Unsatisfactory Between 40 and 64% 2 Unsatisfactory

1 Highly Unsatisfactory 4 Highly Unsatisfactory Less than 39%

4.2.1 Data Collection Activities Completed to Date

Preliminary Scoping Mission

Prior to the engagement of the AGRER consulting Team that from now on will conduct the Zambia Country Strategies Evaluation, (working in close cooperation with the AFDB IDEV staff), to date, the IDEV evaluation Team has conducted a preliminary scoping mission to Zambia to:

(i) build buy-in and support for the evaluation (ii) identify key stakeholder groups as well as the evaluation issues that are relevant to them (iii) explore the country context and opportunities for data collection (iv) gather background documents (v) validate the existing portfolio.

During the mission, the evaluation team met with and interviewed key government officials in Ministries, Departments and Agencies associated with AfDB-supported activities, Civil Society Organizations, academics, AfDB-supported project implementing units, field office staff and other key stakeholders.

In addition to providing confirmation of the Bank's lending and non-lending activities, these stakeholders also provided feedback on issues they felt were particularly important to address through the evaluation, including:

the relevance and use of feasibility studies and Economic and Social Work (ESW) before and after initiation of the project cycle;

the role of limited capacity, particularly procurement capacity and availability of resources for ongoing project management;

the costs of delayed implementation due to Bank approval processes and the frequency of Bank missions;

the extent to which the Bank is adequately capitalizing on opportunities for joint and co-financed projects and PPPs throughout the project cycle; and

the appropriate role of the field office in supervision, project implementation and promotion of partnerships.

32

To the extent possible, these issues have been incorporated into the evaluation matrices prepared by the IDEV staff and will be addressed in the conduct of the evaluation that will be performed by the AGRER Team

Review of CSPs and Project Documents

In order to identify a Theory of Change for the Bank's intervention in Zambia as well as a matrix of appropriate indicators to assess the performance of the Bank's interventions across various sectors, the IDEV evaluation team conducted a preliminary review of each CSP as well as the appraisal reports for the range of projects under examination.

During this review of project appraisal reports, particular attention was paid to the results measurement frameworks provided in these documents in order to gain an understanding of:

(i) the context in which Bank interventions have been implemented (ii) the specific development needs they are meant to address (iii) the intervention logic of each project (iv) indicators identified to track project implementation and results.

This information was used to develop sector-level results chains which will frame IDEV's assessment of the effectiveness of the Bank's interventions at the sector and country levels.

4.2.2 Proposed Data Collection Methods and Analysis The data collection and analysis will be conducted using a phased approach through which subsets of the portfolio will be subjected to gradually deeper levels of analysis. These phases include:

(i) a review of key policies, strategies and project documents; (ii) interviews with key stakeholders covering a sub-set of projects

In-depth case studies and project results assessment for a further sub-set of completed or nearly-completed projects. The proposed approach for each phase is described in greater detail in the sections below.

Document Review

The Document Review will be used to identify, verify and analyse all the qualitative and quantitative information that can be drawn from secondary sources. These documents will include:

project approval documents and loan agreements (including background studies);

project supervision and completion reports;

procedural and governance documents, including policies and strategies;

official statistical data sets;

international scoring schemes; and

past evaluations produced by other development partners.

The document review will support all elements of the evaluation. First, the document review will help establish the context of the Bank's intervention as well as the key factors thought to influence the achievement of results. Furthermore, the review of corporate and project documents will be used to

33

establish the Bank's intervention logic at the project and country level, which will then be tested through further analysis.

With respect to the Bank's portfolio, the document review will serve as a first level of analysis applied to the entire range of projects. This phase will be particularly important for examining projects which have recently been approved or which are still in the early stages of implementation.

To the extent possible, the document review will be conducted using structured templates that allow for the coding of relevant information against the evaluation issues, questions and indicators.

The anticipated scope of the document review is presented in table 11, below.

Table 11. Proposed scope of the Document Review

Type of Document Document Examples

Project Documents

Appraisal Reports

Loan Agreements

ADOA Assessments

Supervision Reports

Project Completion Reports and Extended Supervision Reports

Operational Policies, Strategies and Assessments

Corporate Strategies (e.g. Ten Year Strategy)

Country Strategies

Regional Strategies

Sector Policies and Strategies

Readiness Reviews

Quality at Entry Assessments

National Development Strategies and cooperation frameworks

National Development Strategies and Frameworks

Sectoral Strategies and Program Documents

Development Partner Cooperation Frameworks

Annual National Policy Dialogue Reports

Development Strategies of partner organizations

Evaluation documents and studies

Previous AfDB country and sector evaluations relevant to Zambia

Evaluations conducted by other development partners

Government of Zambia evaluation and monitoring reports

Available studies and Economic and Social Work

Development Research and Literature

Data sets, statistics and rating schemes

National Datasets

Sectoral and Regional-level statistics

MDG indicators

International rating schemes (CPIA, Doing Business, Perceived Corruption Index, Mo Ibrahim)

Interviews with Key Stakeholders

Interviews with key stakeholders will be used to gain a deeper understanding of the context and rationale underpinning the Bank's intervention. It will also be used to collect and assess information about the perspectives, expectations and experiences of key informants and stakeholders within the Bank, the Government of Zambia and other development institutions.

34

The stakeholder interviews will be conducted using semi-structured protocol which addresses the issues, questions and indicators identified in the evaluation matrix. The interview notes will then be transcribed and coded against the evaluation indicators to contribute to the analysis.

With respect the Bank's portfolio, key informant interviews will be used to gain additional information about projects for which implementation is ongoing, but for which it is not yet possible to assess the achievement of results.

The scope of key stakeholders and key informants to be interviewed is presented in table 12, below. Draft interview protocol is available in Annex C. A preliminary list of stakeholders’ interviews is available in Annex D.

Table 12. Potential Stakeholder Groups to be interviewed

Stakeholder Group Potential Interviewees

Bank Staff

Task Managers

Sector Managers

Field Office Staff

Country Economists

Country Team Members

Public Sector Representatives

Ministry of Finance and National Planning

Ministry of Agriculture and Cooperatives

Ministry of Commerce, Trade and Industry

Ministry of Energy and Water Development

Ministry of Gender and Development

Ministry of Local Government and Housing

Ministry of Community Development and Social Services

Ministry of Communication and Transport

Private Sector Representatives

SME and Microfinance Organizations

NGOs and CSOs

Farmer's associations

business organizations

Local Communities local service providers

community organizations

Other Donors Multilateral and bilateral donors

IFIs and other development partners

Project Implementation Units

Zambia Association of Manufacturers (ZAM)

Zambia Chamber of Commerce and Industry (ZACCI)

Investrust Bank

CETZAM Financial Services

Zambia Development Agency - PPP Unit

Itezhi-Tezhi Power Corporation (ITPC)

Zambia Road Development Agency

Nkana Water and Sewerage Company

Case Studies and Project Results Assessments

Case studies will comprise in-depth assessments of the development results achieved for projects which are completed, closed or ongoing, but approaching full disbursement. The case studies will involve field

35

work, including in-depth interviews with stakeholders and focus groups or surveys with the project beneficiaries.

In addition, all closed and completed projects approved after 2014 will be subject to an in-depth assessment of the results chain called a Project Results Assessment (PRA). These PRAs will be conducted in accordance with the approach recommended in the Evaluation Cooperation Group's "Big Book on Evaluation Good Practice Standards," including:

An assessment of the causal chain in relation to the needs of the target population in collaboration with stakeholders and experts;

Examination of the critical risks and assumptions and expectations inherent in the project's design;

Use of available research evidence and practical experience to compare the project with projects based on similar concepts; and

Observation of the project in operation, focusing on interactions that were expected to produce the intended outcomes.

As part of this process, the AGRER evaluation Team will complete a Project Results Assessment (PRA) for each project. This template is meant to collect information about the Bank's development results from completed projects to support a synthesis of the Bank's development results as part of the CEDR. This assessment will involve the identification of ratings on a six-point scale of the relevance, effectiveness, efficiency and sustainability of each project against a series of set criteria. The issues, questions and indicators identified as part of the CSPE are complementary to this process such that this assessment will contribute to the identification of overall ratings for these evaluation issues. The PRA template is provided in Annex E.

Review of Country Mechanisms

In order to support the CEDR synthesis, evidence from the CSPE will be used to support an assessment of the extent to which various operating mechanisms are contributing to the Bank's development effectiveness. The evaluation team will use the CSPE evidence to complete a template which identifies the various contextual factors which have contributed to the Bank's performance as well as the extent to which various operating mechanisms have been applied, including, but not limited to: (i) selectivity; (ii) leveraging; (iii) engagement and dialogue; and (iv) strategic partnerships.

This template is still under development, but will complement the CSPE issues, questions and indicators. However, unlike the PRAs, this assessment will not be used to identify ratings for each of the evaluation criteria.

4.2.2 The Project Sample Although the entire scope of the Bank's portfolio between 2002 and 2014 will be assessed as part of the evaluation, each project will be subject to a different depth of analysis depending on their characteristics. The first group of projects will be subject only to a review of project documents, including the appraisal report, loan agreement and any supervision information. The second group of projects will be subject to both a review of project documents and interviews with key stakeholders. The final group of projects will be subject to a review of project documents, interviews with key stakeholders and site visits. This final group includes those projects which will be subject to a Project Results Assessment.

36

The projects were divided among the three groups using the following rationale:

Group 1 (Document Review Only) includes projects which are ongoing or approved with net loans amounting to less than 1,000,000 UA;

Group 2 (Document Review + Stakeholder Interviews) includes projects which are ongoing or approved with net loans amounting to more than 1,000,000 UA, as well as closed and completed projects which are less than 50% disbursed; and

Group 3 (Document Review + Stakeholder Interviews + Site Visits) includes projects which are closed or completed and greater than 50% disbursed. This final group includes projects which will be subject to a Project Results Assessment as part of the CEDR.

In identifying projects for inclusion in group 3, an exception was made to include some large transport and energy sector projects which are not yet closed or completed. These projects were included in the third group due to their scale as well as the importance of transport and energy sector projects to both the current and subsequent CSPs. Regional projects which involve specific outputs and outcomes for Zambia were included in group 2 regardless of their implementation status, disbursement rate and size due to the likely limited scale of funds and activities specific to Zambia. In addition, two large completed projects in the Water Supply and Sanitation Sector which were eligible for inclusion in group 3 will be examined under group 2 because site visits will be conducted as part of a concurrent review of the Bank's activities in the Water Supply and Sanitation sector. The IDEV evaluation team will ensure coordination with this concurrent project to ensure that the necessary information is shared and that the project assessments are conducted in a coordinated manner. The final project sample is presented in table 12, below. Table 12. Zambia CSPE Project Sample

Zambia CSPE Project Sampling

Group (PRA)

Project Name Sector Approval

Date Net Loan '000 UA

Status Disbursement

Ratio

1 HUMANITARIAN

EMERGENCY ASSISTANCE Agriculture 02/12/2005 354.20 Closed 0.00

1 ZAMBIA EMERGENCY ASSIST. FLOOD VIC.

Agriculture 05/06/2008 354.20 Completed 0.00

1 FINNISH SUPPORTED

SMALL SCALE IRRIGATION

Agriculture 28/12/2009 0.00 Ongoing 0.00

1 NACALA ROAD

CORRIDOR STUDIES Transport 25/11/2009 256.01 Ongoing 0.00

1 EMERGENCY ASSISTANCE

TO 2009 FLOOD MITIGATION


1 MULTIPURPOSE SMALL

DAMS Water Supply and

Sanitation 04/09/2012 760.48 Ongoing 30.00

1 PROJECT PREPARATION

FACILITY - ZM LISP Agriculture 29/03/2013 300.00 Ongoing 65.65

2 CENTRAL PROVINCE

EIGHT CENTRES WATER Water Sup/Sanit 17/12/2003 21342.00 Closed 100

37


Group (PRA)



Status Disbursement

Ratio

SUPPLY AND SANITATION

2

STATISTICAL CAPACITY BUILDING UNDER THE

INTERNATIONAL COMPARISON

PROGRAM FOR AFRICA

Multi-Sector 12/10/2004 18,140.00 Completed 99.89

2

ENHANCING PROCUREMENT

REFORMS AND CAPACITY PROJECT

Multi-Sector 21/07/2006 5,660.00 Completed 100

2 SADC NORTH-SOUTH

TRANPORT CORRIDOR IMPROVEMENT STUDY

Transport 01/12/2006 1450.00 Closed 87

2 RURAL WATER SUPPLY & SANITATION PROGRAM

Water Supply and Sanitation

31/10/2006 15000.00 Closed 68.02

2 ZANACO FAPA TA GRANT

FOR ZAMBIAN SMES Finance 10/11/2008 642.26 Closed 100

2 PFSL- FAPA TA - ZAMBIA Finance 13/07/2009 662.48 Closed 100

2 KAZUNGULA BRIDGE

PROJECT Transport 07/12/2011 51000.00 Ongoing 0.98

2 ITEZHI-TEZHI POWER

PROJECT Power 13/06/2012 24736.00 Approved 0.00

2 FRB SUBSIDIARY IN

ZAMBIA Finance 12/12/2012 31899.94 Approved 0.00

2 LIVESTOCK

INFRASTRUCTURE SUPPORT PROJECT

Agriculture 19/06/2013 12000.00 Ongoing 7.63

2 MAAMBA COLLIERIES POWER GENERATION

PROJECT Power 02/10/2013 97706.50 Approved 0.00

2 STRENGTHENING

CLIMATE RESILIENCE KAFUE

Environment 18/10/2013 26924.57 Ongoing 4.11

2 SUPPORT FOR SCIENCE &

TECHNOLOGY EDUCATION

Social 20/11/2013 22220.00 Ongoing 3.45

2 GAFSP-AGRICULTURE PRODUCTIVITY AND

MARKET Agriculture 26/03/2014 20111.15 Ongoing 2.06

2 RURAL WATER SUPPLY

AND SANITATION II Water Supply and

Sanitation 10/09/2014 13018.10 Approved 0.00

2 LAKE TANGUANYIKA

DEVELOPMENT PROJECT Agriculture 18/12/2014 20078 Approved 0.00

3 (no)

SMALL SCALE IRRIGATION PROJECT

Agriculture 07/09/2000 6171.20 Completed 100

3 (yes)

PROJECT TO SUPPORT THE LAKE TANGANYIKA INTEGRATED REGIONAL

DEVELOPMENT PROGRAMME (Zambia)


3 (yes)

LUMWANA COPPER MINING PROJECT

Ind/Mini/Quar 27/09/2006 29903.90 Closed 100

3 POVERTY REDUCTION Multi-Sector 29/11/2006 20000.00 Closed 100

38


Group (PRA)



Status Disbursement

Ratio

(yes) BUDGET SUPPORT

3 (yes)

INVESTRUST ZAMBIA Finance 24/09/2008 2479.80 Closed 100

3 (yes)

ZANACO GUARANTEE FAC./LOC ZANACO-

ZAMBIA Finance 24/09/2008 7110.65 Closed 100

3 (yes)

POVERTY REDUCTION BUDGET SUPPORT II

Multi-Sector 29/10/2008 14901.83 Closed 100

3 (yes)

NKANA WATER SUPPLY AND SANITATION

PROJECT


27/11/2008 35000.00 Closed 89.97

3 (no)

COMMUNITY WATER MANAGEMENT IMPROVEMENT

Agriculture 12/11/2009 575.72 Completed 100

3 (yes)

POVERTY REDUCTION BUDGET SUPPORT III

Multi-Sector 30/06/2010 31900.00 Closed 100

3 (no)

MULTI-NACALA ROAD CORRIDOR PROJECT -

Phase II (ZAMBIA) Transport 27/09/2010 69369.00 Ongoing 26.83

3 (yes)

FOURTH POVERTY REDUCTION BUDGET

SUPPORT Multi-Sector 28/09/2011 15000.00 Closed 100

3 (no)

ITEZHI-TEZHI POWER PLANT AND

TRANSMISSION LINE PROJECT

Power 13/06/2012 36400.00 Approved 0.00

3 (yes)

CETZAM FINANCIAL SERVICES PLC

Finance 21/10/2013 1062.81 Ongoing 100

4.2.4 The Evaluation Framework The ratings assessed for each of the evaluation criteria will be based upon a series of indicators and judgement criteria corresponding to each evaluation question. To the extent possible, these indicators have been harmonized with those used in other IDEV CSPEs as well as the One Bank Results Measurement Framework and the requirements of the PRA process and CEDR synthesis. For each evaluation issue, the available qualitative and quantitative evidence from primary and secondary sources will be triangulated to identify findings, conclusions, lessons learned and appropriate evaluation ratings. These indicators and criteria are presented in the evaluation matrices below.

Evaluation Matrices

1. Relevance

Evaluation Question Judgement Criteria Data Sources

EQ#1.1: To what extent are the Bank's Country Strategies and operations for Zambia aligned with the development needs, challenges and priorities of the country as well as the needs of the expected beneficiaries?

Thematic consistency of the Bank's CSPs with the Government of Zambia's strategic framework for development and priorities. Degree of consistency between the Bank's

interventions at the sector level with the

National and Sector-level Development Strategies

Bank CSPs for Zambia

Interviews with RMC stakeholders

Available literature

39

1. Relevance

Evaluation Question Judgement Criteria Data Sources

Government of Zambia's sector development strategies and action plans.

Perceived consistency between Bank strategies and country development the needs among RMC stakeholders.

Degree of consistency between the Bank's strategies and development needs identified in the available literature.

EQ#1.2: To what extent are the Bank's Country Strategies for Zambia aligned with the Bank's own strategies, priorities and policies?

Degree of consistency between the Zambia Country Strategies and the Strategic priorities of the Bank.

Degree of consistency between the Bank's interventions and the Bank's strategic priorities at the sector level.

Bank Policy and Strategy documents


Interviews with Bank staff

EQ#1.3: To what extent are the Bank's interventions in Zambia aligned with the Bank's Country Strategies?

Degree to which project approvals over each CSP period reflect the Bank's Country Strategy for Zambia.

File Review of project information


Interviews with Bank staff

Rationale and Approach

The relevance of the Bank's assistance will be assessed at both the project and country levels along three dimensions: (i) alignment of projects and the CSP with demonstrable development needs; (ii) alignment of the CSP with the Bank's corporate and sector-level strategies; and (iii) alignment of projects with the CSP and with Regional Integration Strategies. Evidence from each of the evaluation questions will be triangulated to provide an overall rating on a 6-point scale from Highly Satisfactory to Highly Unsatisfactory.

2. Effectiveness

Evaluation Question Data Sources

EQ#2.1: To what extent have the Bank's interventions achieved their objectives is terms of the delivery of outputs?

Review of supervision reports and project monitoring information

PCRs, XSRs, BTORs and mid-term reviews

Available country statistics for key indicators

Interviews with project stakeholders (Bank staff, implementing partners, Government of Zambia)

Interviews and/or surveys of beneficiaries as possible

Field visits

EQ#2.2: To what extent have the Bank's interventions contributed to the achievement of development results in terms of expected outcomes?

EQ#2.3: To what extent have the Bank's interventions contributed to the achievement of development results for the country, including intended and unintended impacts?

EQ#2.4: To what extent have the Bank's interventions benefitted target group members?

Judgement Criteria and Rationale

Effectiveness will be examined at the project, sector and country levels and will centre upon the achievement of both outputs and immediate outcomes. The achievement of intermediate outcomes will be considered where credible data are available. This examination of the Bank's effectiveness will be based on a triangulation of qualitative a quantitative evidence from both primary and secondary sources to inform a robust and credible assessment. First, each project will be rated on a six-point scale ranging from Highly Satisfactory to Highly Unsatisfactory. Next, a rating will be provided for each sector based on the extent to which the collection of projects are found to have

40

contributed to sector-level outcomes, identified below. Finally, a global rating will be provided based on the extent to which the projects are found to have contributed to country level outcomes identified in the underlying Theory of Change for the CSP.

For completed projects, the AGRER Team will conduct an in-depth assessment of the results chain consistent with the approach recommended in the Evaluation Cooperation Group's "Big Book on Evaluation Good Practice Standards," including:


Examination of the critical assumptions and expectations inherent in the project's design;



Special note will be taken of challenges and enabling factors within the implementation context for each project in order to test the Bank's Theory of Change for its operations in Zambia.

Sector Sector Objective Results Chain and Potential Indicators

Agriculture and Environment

To contribute to the reduction of rural poverty and food

insecurity

Ultimate outcomes

Economic growth and diversification

Reduced poverty and enhance food security in rural communities Intermediate outcomes

Increase income levels for farmers

Increased resilience to climate change and exogenous shocks

Creation of enabling environment for smallholder self-supply investments

Increased number of smallholder farmers investing in self-supply solutions

Immediate outcomes

Increased farm land under cultivation and agricultural productivity

Increased employment of farm helpers

Improved climate resilience infrastructure

Increased availability of irrigation technology and equipment

Increased promotion and use of on-farm water resource management methods

Credit access and investment promotion

Increased agro-processing Outputs

Development of new farm blocks

Provision of credit to farmers

Guaranteed market and storage facilities

Public expenditure review in Agriculture sector

Capacity building for farmers and extension service

Strategic rural roads climate proofed Indicators

Number of cultivated Hectares

Increase in Food production

Number of jobs created

Growth rate in the agriculture sector

Contribution of Agriculture to GDP

41

Proportion of the population assessed as food secure

Reduction in number of districts that are food Insecure

Number of farm blocks created

Amount of credit extended to farmers

Storage facilities created

Number of capacity building activities benefitting men and women

Number of people in areas at risk whose livelihoods have improved

Reduction in damage / losses from extreme climate events

Percentage primary crops processed by men and women Challenges

Lack of access to credit, particularly for MSMEs

Low entrepreneurship

Lack of knowledge on opportunities and constraints of women entrepreneurs

Very low skills level / inadequate skilled human resources

Limited access to formal education

Increasing labour force without skills (300.000/ year leaving the school system)

Land tenure system not favourable to the poor (97% of the poor have no title to land

Lack of infrastructure and irrigation/agro processing equipment

Environmental degradation and climate change


Improved health and well-being of Zambians through

sustainable, reliable and

affordable water and sanitation

Ultimate outcomes

Improved health and standard of living

Economic development

Poverty reduction Intermediate outcomes

Improved quality and delivery of water and sanitation services

Improved functionality of rural water supply facilities

Reduced infant and maternal mortality

Reduced incidence of diarrhoea diseases and worm infestation in infants and children in rural communities

enhanced educational enrolment and completion rates for girls Immediate outcomes

Increased supply of water

Improved access to water and sanitation;

Improve management of water and sanitation services

Improved awareness and attitudes on environmental cleanliness and personal hygienic practices

Outputs

WSS facilities constructed or rehabilitated

Sanitation facilities in public institutions (schools, health centre and markets)

Increased proportion of the rural population with adequate and convenient supply of clean (potable) water;

Sanitation and hygiene promotion

Capacity building; Indicators

Number of people with access to safe water and sanitation

42

facilities

Number of cubic meters of water supplied annually

Water supply and sanitation facilities constructed

Reduction in the incidence of water borne diseases

Pupil per latrine stance ratio for boys and girls

Number of water points rehabilitated

Number of meters installed

Proportion of water tests meeting standards

Average water consumption per user

Average distance between homes and water sources

Water pricing and rate of non-payment by area

Number of hours of water service per day

Number of persons with access to safe individual/communal toilets

Percentage of people observing good health and hygiene practices

Sewage treatment rate by area Challenges

Poor state of infrastructure

limited access to safe water and sanitation facilities

Distance to water sources

Multi-Sector

Contribute to economic growth

through strengthening economic and

financial governance and improving the

investment climate

Ultimate goal

Inclusive economic growth

Transparent, accountable governance

Poverty reduction Intermediate outcomes

Improved living standards and equitable access to service delivery.

Improved financial management capacity

Sustainable domestic and external debt.

Improved business enabling environment.

Legal and Regulatory framework for public procurement strengthened

Public Procurement practices and procedures enhanced Immediate outcomes

Increased clarity of roles and responsibilities and improved efficiencies and effectiveness of debt management

Efficient budget execution and public expenditure management environment

Reduced perceived corruption

Increased share of population has access to social services.

Increased decentralization of resources and capacity for service delivery

Revenue base is increased to predictably finance social services Outputs

Revised PAF taking into account the FNDP monitoring framework

Procurement Act and Regulations

Draft report detailing Public Expenditure Management and Financial Accountability reforms

Implementation of all the twelve components of PEMFA

Debt Policy and Management Strategy agreed

43

New Public Procurement Act enacted

Effective and efficient business management

Improved and effective public finance management Indicators

Unemployment/underemployment rates

Percentage of discretionary budget allocation to direct poverty reduction programmes

Poverty incidence

Clearance of domestic arrears and reduction in domestic borrowing

Improved public procurement system

Improved service delivery and accountability

Number of days to register a business

Number of days to import and export products

Expenditure variance

Percentage of budget released to District Boards

Percentage of population with access to safe reliable water

Debt assessment

Global Competitiveness Index

Resource Development Index

Country Policy and Institutional Assessment Score

Real GDP growth

IIAG score

"Ease of Doing Business" Indicators

Public Expenditure and Financial Accountability (PEFA) Assessment Indicators

Challenges

Weak implementation capacity

High fiduciary risks which could lead to weak accountability

increased public expenditure without corresponding revenue

high oil and commodity prices

weak capacity within government

weak PFM system

weak M/E system to support quality data collection

High election year public expenditure

Private Sector / Finance

Support economic growth and diversification, job creation and the development of an inclusive financial system in Zambia

Ultimate Outcomes

economic growth and diversification

poverty reduction

increased regional and global trade Intermediate Outcomes

job creation in the formal sector (particularly local jobs and jobs for women)

increased number and diversity of SMEs (particularly SMEs owned by women)

increased bankarization levels

increased tax revenues Immediate Outcomes

increased availability of SME financing products and services

increased SME lending

improved SME lending terms

enhanced capacity for compliance with environmental and social

44

standards and monitoring

increased industrial capacity (including mining and manufacturing) Outputs

provision of training and technical assistance

provision of gender training

provision of risk sharing instruments

faster and more transparent loan approval processes

improved lending terms for SMEs

provision of financial resources

rehabilitation and development of mining and other industrial sites

Indicators

number of jobs created (local jobs and jobs for women)

% increase in SME lending portfolio among target institutions

% increase in SME loan approvals (proportion for women)

time required for loan approvals

# of new SMEs financed

% reduction in collateral requirements

% of projects in compliance with Environmental and Social standards

government tax revenues as a share of GDP/Expenditure

% of ESMP projects implemented successfully. Challenges

low bankarization levels and access to finance

cumbersome lending process and collateral requirements

low capacity to provide lending services among local banks

low productivity in the agricultural and manufacturing sectors

Education

Expansion of higher technical and vocational education

Ultimate Outcomes

improved productivity

increased global competitiveness of the Zambian economy Intermediate Outcomes

reduced unemployment/underemployment

reduced job skills mismatched

Increased number of science and technology graduates

Increased enrolment in science and technology as well as entrepreneurship training

Increased innovation Immediate Outcomes

Increased access to science and technology higher education and entrepreneurship training

Increased quality and relevance and quality of teaching in target institutions

Enhanced gender-responsive science and technology instruction Outputs

Higher learning institutions renovated, upgraded and/or supplied with new equipment

Training of science and technology lecturers at the PhD level

Provision to entrepreneurship training to youth

Development of new gender-responsive curricula for science and technology and entrepreneurship

Indicators

45

% new S/T grads with skills demanded by industry

%/# of science and technology grads employed in key industries

% of science and technology graduates who become youth entrepreneurs

enrolment in S/T higher education

number of lecturers in targeted masters and PhD programs

number of youths trained in work-based and entrepreneurship training

pass rate for STE graduates

number of rehabilitated learning institutions

number of rehabilitated learning institutions with improved water supply and sanitation facilities

Challenges

Skill mismatch with the industrial sector, which limits opportunities for new industries to generate employment

Need for additional high quality instruction and educational opportunities in science and technology

Transport

Improved transport system

which links centres of

economic activity, enhances

socioeconomic development, and promotes poverty

reduction and regional

integration

Ultimate Outcomes

Increased economic growth

increased regional and international trade

increased global competitiveness of Zambian and regional economy

Intermediate Outcomes

increased access to markets and social services

creation of jobs during construction and within surrounding communities

increased trade volumes Immediate Outcomes

reduced vehicle operating costs and transport costs

improved road safety

reduced travel and border transit time

increased traffic throughput

improved project coordination Outputs

Delivery of project design plans and specifications, estimates and tender documents for planned infrastructure installations

Provision of consultancy services

Provision of HIV/AIDS sensitization training

number and percentage of ESMP projects successfully implemented

Construction of border posts and bridges

Rehabilitation of key roads

Delivery of procurement capacity development training Indicators

annual value of regional and international trade

quantity of traffic and value of goods transported through border posts

border transit time

vehicle operating costs

# of job created during construction and operation

annual number of road accidents and fatalities along corridor

46

number of people trained

length of roads rehabilitated

length of roads built

number of border stations built

Energy

Increased and sustainable access

to affordable electricity services

to support economic growth

and reduce poverty

Ultimate Outcomes

Reduced Poverty and improved standard of living

Increased economic development and promotion of a business enabling environment

Intermediate Outcomes

increased government revenues

increased access to and use of electricity

reduced carbon emissions

improved sustainability of power supply Immediate Outcomes

increased electricity supply

increased electricity generation capacity

increased job creation (for women and local population) Outputs

implementation of proposed works

transmission lines built

sub-stations built and/or extended

implementation of proposed ESMP/RAP activities

provision of capacity development assistance

Improved tariff structure Indicators

power supply capacity in MW

national electrification rate

number of jobs created during construction and operation (for women, youths and local population)

total up-front and annual concession fees accrued

length of transmission line built

number of substations built or extended

number of ESMP/RAP activities implemented

net tons of carbon emissions equivalents avoided each year Challenges

Existing power deficit limits access to electricity for business and domestic purposes

Energy deficit is identified as a major barrier to economic development

Existing energy infrastructure is insufficient to address energy needs and leverage existing natural resources

High power tariff and inability to pay by poor segment of population

3. Efficiency

Evaluation Question Indicators Data Sources

EQ#3.1: To what extent were the Bank's interventions delivered in an efficient and economical manner (were resources economically converted into outputs)?

Implementation Progress Ratings (IPRs)

Perceived timeliness and economy of project implementation

Project Completion Reports

Interviews with Task Managers and implementation partners

47

EQ#3.2: To what extent are the Bank's interventions implemented in a timely manner which is consistent with best practices?

Ratio of planned versus actual implementation time at the project level

Project Completion Reports

Interviews with Task Managers and implementation partners

EQ#3.3: To what extent did the Bank's intervention achieve good value for money (cost-benefit)?

Extent to which the chosen project approach has yielded results at less cost relative to other potential approaches.

Economic Internal Rate of Return (where available)

Financial Internal Rate of Return (where available)

File review of project documents

Project completion reports

Interviews with task managers and implementation partners

Review of available literature


The assessment of efficiency will involve three components, namely: (i) the effective use of resources for the delivery of planned outputs; (ii) implementation timeliness; and (iii) extent of the results achieved relative to cost and implementation time (cost-benefit). The assessment will be informed by a triangulation of qualitative and quantitative evidence from project supervision and completion reports, as well as a review of the available literature. A rating will be provided for each dimension using a six-point rating scale from Highly Satisfactory to Highly Unsatisfactory, depending on the availability of adequate data. In addition, a global rating will be provided for efficiency of the Bank's operations in Zambia over the evaluation period.

4. Sustainability


EQ#4.1: To what extent has the Bank sought to mitigate risks to the sustainability of results as part of the project design process?


Readiness Reviews

Interviews with Bank staff, government and local stakeholders and implementation units

Site visits

Criteria Indicators

Technical Soundness

Extent to which the technical design of the project was supported by extensive analytics.

Extent to which maintenance requirements and operational risks have been identified in the project design.

Extent to which the technical design of the project is based on mechanisms and methods which have a demonstrated track record of success.

Financial and Economic Viability

Extent to which adequate public resources or user fees are available to support the continued operation and maintenance of the investment.

Extent to which private sector operations demonstrate long-term profitability

Extent to which long-term financial and technical support has been secured from partners and co-financers.

Institutional Sustainability

Use of country systems for implementation (where feasible).

Identification and implementation of necessary institutional reforms and mechanisms.

Extent of policy dialogue and national ownership in implementing necessary policy and institutional reforms.

Ownership and Partnerships

Extent to which interventions have been implemented in consultation and partnership with local authorities, CSOs, other donors and the private sector.

Resilience Degree of project resilience to exogenous factors, including climate and economic shocks

EQ#4.2: To what extent are the results achieved likely to continue once the Bank's interventions are completed?


Interviews with Bank staff, government and local stakeholders and implementation units

48

National Statistics (where available)

Site visits

Sector Indicators

Agriculture and Environment

Evidence of regular maintenance conducted for infrastructure investments (e.g. rural roads, irrigation schemes, storage facilities)

Implementation of identified agricultural sector reforms

Village and district development plans funded and implemented


Share of infrastructure investments appropriately maintained and still operational

Share of water-supply schemes which are financially self-sustainable

Technical and financial capacity for maintenance and operation among local officials.

Transport Evidence of regular maintenance for infrastructure investments

Long-term trends is travel times and Vehicle Operating Costs

Long-term financial sustainability of maintenance schemes

Implementation of identified policy reforms

Education Share of targeted institutions and training facilities operational and maintained

Retention of specialized faculty and trained staff

Financial sustainability of targeted training and educational facilities

Finance / Private Sector Development

Economic and financial performance of selected projects (profitability)

Retention rate of trained staff

Repayment rate for financial support to SMEs

Growth of loans to MSMEs

Multi-Sector Extent of implementation of targeted reforms

Extent of reforms appropriately supported by public funds

Retention of trained staff

Demonstrated use and maintenance of improved systems

Power Evidence of regular maintenance for infrastructure investments

Share of electrical distribution systems and transmission lines properly functioning

Long-term financial sustainability of maintenance schemes

Technical and financial capacity for maintenance and operation among local officials

Implementation of identified policy reforms

The extent of line losses (technical and non-technical)

Deviation between unite cost and average tariff


The assessment of sustainability will be conducted via two streams of analysis: (i) the extent to which the project design has adequately examined and addressed critical elements to promote the long-term sustainability of the project; and (ii) the demonstrated long-term viability and sustainability of project outputs and outcomes. The assessment will primarily focus on the first stream of analysis. The second stream of analysis will be used to inform sector-level and global ratings depending on the availability and credibility of relevant data. A rating for sustainability will be provided at both the project and program levels against a six-point scale from Highly Satisfactory to Highly Unsatisfactory.

5. Cross-cutting themes


EQ#5.1: To what extent do the Bank's interventions inclusive in terms of geography, age and gender?

Degree and depth of gender analysis in the design and delivery of projects

Share of interventions with specific components addressing gender inequality

Share of projects utilizing gender checklists and gender-informed design

Extent of institutional capacity development support provided

File review of project documentation

Review of available PCRs/XSRs

Interviews with task managers,

49

for the identification and management of gender disparities.

Geographical distribution of operations

Share of bank interventions which explicitly consider or address regional disparities

local government stakeholders and implementing partners

site visits EQ#5.2: To what extent are the Bank's interventions environmentally sustainable and support the transition to green growth?

Share of infrastructure projects which integrate environmental protection measures

Share of new projects designed using climate checklists, environmental experts and climate-informed design

Existence of mechanisms for monitoring and addressing environmental impact

Extent of support provided for institutional capacity development surrounding environmental management and climate change


The assessment of cross-cutting themes will involve an examination of the extent to which the themes of inclusive and green growth have been incorporated into the design and delivery of each project. A global assessment will be provided for the Bank's portfolio in Zambia against a six-point scale ranging from Highly Satisfactory to Highly Unsatisfactory.

6. Design and Delivery


EQ#6.1: To what extent has the quality of CSPs been satisfactory? To what extent are the Bank's interventions coherent and well-coordinated internally?

Extent to which CSPs have been identified as Satisfactory through reviews of Quality at Entry

Extent of linkages among operations in different sectors.

CSP documents

Government Development Strategies

Available literature

Reviews of Quality at Entry

Readiness Reviews

Review of ESWs and knowledge products


Interviews with Country Team, Government of Zambia stakeholders, partners and co-financers

EQ#6.2: To what extent have the Bank's interventions been selective and strategic?

Degree of concentration of the Bank's activities in selected focus areas

Evidence of demonstrated expertise and comparative advantage within selected focus areas

Extent of complementarity with the activities of other donors and partners

EQ#6.3: How has the Bank adapted its interventions to implementation challenges within the country?

Evidence of ongoing analysis of implementation challenges and limitations

Share of projects utilizing innovative design approaches and instruments

Share of projects addressing emerging development needs

EQ#6.4: To what extent has the Bank's engagement been informed by policy dialogue with national actors, development partners and other interested stakeholders?

Evidence of ongoing and strategic policy dialogue with the Government of Zambia

Evidence of appropriate stakeholder mapping and engagement with potential partners in the design of programs and projects

EQ#6.5: How well has the Bank leveraged its financial resources within the country through mechanisms such as productive partnerships and co-financing?

Share of projects which utilize co-financing

Evidence of strategic engagement with partners to identify opportunities for co-financing

Evidence of strategic selection of projects as candidates for partnership and co-financing

EQ#6.6: To what extent has the Extent and quality of sector and thematic analyses

50

Bank acted as a knowledge broker, advisor and convener? To what extent are the Bank's interventions coherent and well-coordinated internally?

Evidence of ongoing research and ESW to inform program and project design

Evidence of technical assistance provided to inform necessary policy and institutional reforms

Evidence of ongoing policy dialogue with government stakeholders to identify opportunities for analytical support and partnership.

EQ#6.7: To what extent has the Bank cooperated with other development partners to ensure complementarity and reduce overlaps?

Evidence of donor mapping to inform the Bank's strategic positioning

Extent to which the Bank's interventions overlap with those of other development institutions

Share of the Bank's interventions co-financed by other development partners

Degree of collaboration with other development partners in project design, analytical work, monitoring and reviews

Extent of participation and leadership in donor partnership groupings


This assessment will involve consideration of the extent to which the Bank has successfully implemented strategic operating mechanisms to promote the achievement of sustainable development results and maximize its value as a development partner. These mechanisms, identified in corporate strategies such as the TYS and the MTS, include:

implementing high-quality strategies which are informed by dialogue and consultations with RMCs;

ensuring selectivity in operations;

providing innovative solutions to development challenges;

providing knowledge and technical advice;

leveraging financial resources through convening co-financers, including other development partners and the private sector; and

Maintaining effective partnerships with other development partners. A global rating will be provided for management of the Bank's portfolio in Zambia against a six point scale ranging from Highly Satisfactory to Highly Unsatisfactory. The rating will be made based on a triangulation of evidence from project documents, interviews with key stakeholders and site visits.

7. Management for Results


EQ#7.1: To what extent has the Bank's supervision of its interventions been effective?

Extent to which projects have been supervised on an annual basis

Extent to which supervision has been supported by field offices

Share of projects which underwent a mid-term review of progress


Review of PCRs, XSRs, mid-term reviews and BTORs

Interviews with Task Managers, Field Office Staff, implementation units and Government of Zambia

EQ#7.2: How has the Bank used supervision and results information to guide the management of its interventions?

Evidence that supervision reports have been used to inform ongoing implementation

Evidence of implementation and design changes made in response to ongoing supervision and monitoring

EQ#7.3: To what extent has the Bank successfully implemented a performance measurement

Share of projects with an appropriate results measurement framework, including clear targets

share of projects possessing appropriate baseline

51

strategy which focuses on the achievement of outputs, outcomes and impacts?

data

Evidence of regular data collection against established results indicators

stakeholders

Site visits

EQ#7.4: To what extent have lessons learned been identified and incorporated into subsequent projects and CSPs?

Extent to which evidence from PCRs/XSRs, Mid-term reviews and evaluations have been incorporated into CSPs

Evidence of incorporation of lessons learned into the design of new projects

EQ#7.5: How has the Bank worked with the Government of Zambia to enhance its own systems of Results-Based Measurement?

Conduct of a needs assessment to identify capacity challenges for results based management

Implementation of regular dialogue or capacity development support to promote ongoing management for development results.

EQ#7.6: To what extent has the Bank identified, monitored and addressed potential implementation risks?

Evidence of identified implementation risks and mitigation measures

Evidence of ongoing risk monitoring

Use of risk monitoring information to inform project design and implementation


A final rating will be provided at the country level based on an assessment of the extent to which the Bank has successfully implemented a rigorous performance measurement regime and used this information to manage risks, guide implementation and inform the development of new projects and country strategies. In addition, this assessment will involve an examination of the Bank's efforts to develop capacity for results based management within Zambia. A global rating will be provided based on a 6-point scale ranging from Highly Satisfactory to Highly Unsatisfactory based on a triangulation of evidence from project documents, interviews and site visits.

4.3 Special Considerations for Case Studies and Project Results Assessments

4.3.1 Contribution Analysis - Methodology and Approach Case studies and Project Results Assessments will use contribution analysis to the greatest extent possible in order to help ensure that credible evaluative information is available about the achievement of results at the level of outcomes to inform the CEDR. All projects identified under group 3 of the project sample will be subject to a contribution analysis of the achievement of results, implemented via the following five stages:26

1. Identification of a project logic which identifies how the project is intended to work and details: (i) the project context; (ii) the anticipated activities, outputs and outcomes; (iii) the intended beneficiaries. The initial project logic will be identified through a review of project documents.

26 This approach is consistent with that outlined in Mayne (2001) "Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly," The Canadian Journal of Program Evaluation, Vol 16, No 1, Pages 1-24.

52

2. Gathering of additional secondary evidence as well as interviews with key stakeholders in order to: (i) confirm or adjust the project logic; (ii) identify necessary assumptions underlying the project logic; and (iii) identify alternative explanations or contributing factors for the achievement of desired outcomes.

3. Implement interviews and focus groups with beneficiaries and key informants in the field and observe the project in operation in order to: (i) assess the extent to which activities have been implemented and outputs have been delivered; (ii) assess the extent to which expected outcomes have been achieved; (iii) test the underlying assumptions and of the project logic; and (iv) determine the extent to which alternative explanations and factors have contributed to the achievement of results.

4. Create a performance story based on the available evidence and identify potential limitations and data gaps.

5. Seek out additional evidence as necessary to modify and strengthen the performance story. The means through which the contribution analysis is implemented will depend on the availability of credible comparative information against which to assess the achievement of results.27 Where possible, it is expected that the assessment of results will proceed by assessing the results achieved against an appropriate counterfactual, such as a similar region which did not benefit from the intervention under consideration. However, where an appropriate counterfactual does not exist or where such a comparison would be impractical (e.g. where there is no other feasible factor which could contribute to the achievement of results), the evaluation team may use a before-and-after assessment to assess the extent to which results have been achieved. Use of this approach is subject to the availability of appropriate baseline data. Should neither appropriate baseline data nor an appropriate counterfactual be available, the evaluation team will rely on in-depth key informant interviews and focus groups to assess the achievement of results as well as the contribution of alternative explanations. In each case, the methodology chosen will be justified based on the availability of credible data.

4.3.2 Implementation Given the high number of projects to be assessed at this level of detail and the limited amount of time available to conduct the evaluation, the AGRER evaluation Team will need to ensure that any data collection undertaken in the field is focused and targeted to the requirements of the PRA process. Accordingly, the assessment of projects identified for group 3 will proceed in three stages designed to identify and address critical data gaps prior to data collection in the field. First, the evaluation team will conduct a thorough review of all project documents and available supervision and completion reports.

27 This graduated approach is consistent with the methods outlined in White, Sinha and Flanagan (2006) "A Review of the State of Impact Evaluation." http://www.oecd.org/dac/evaluation/dcdndep/37634226.pdf

53

As stated above, this information will be used to identify an initial project logic to be examined further. The evaluation team will also plug this information into the PRA template (provided in Annex E) in order to identify the extent of existing data gaps. Subsequent to the document review, the evaluation team will conduct interviews with relevant stakeholders at headquarters to verify and complement the information collected from the document review. Information from the interviews will be used to test the project logic and will also be coded and be inserted into the PRA template. The AGRER evaluation Team will again assess the availability, depth and quality of the information and address any remaining data gaps. This assessment will be used to complete a data request for stakeholders in the field, adjust interview protocol and establish data collection criteria for site visits. The data requests and any available interview protocol will be sent to interlocutors in advance to help ensure that the required information is readily available when the evaluation team is in the field. Following data collection in the field, the evaluation team will analyse the available data and insert it into the PRA template. An attempt will then be made to address any remaining data gaps using available literature and external sources of evidence before finalizing the Project Results Assessment. The methodological requirements for group 3 projects are further elaborated in the Guidance Note for the PRA, provided in Annex F.

4.4 Challenges, Limitations and Risks IDEV anticipates a number of challenges and limitations in the conduct of the evaluation which may frustrate timely delivery or limit the quality of the report. An effort has been made to identify these issues in advance and identify measures for mitigating their impact. These challenges, limitations and mitigation measures are discussed in table 13, below. Upon review of the Table below, the AGRER Team concurs with IDEV. Table 13. Challenges and mitigation measures

Challenge / Risk / Limitation

Description and Mitigation Strategy

Data Quality and Availability

Each of the data sources identified for the evaluation are subject to data quality concerns. With respect to project documents, logical frameworks and data provided may be of poor quality and supervision data may not be available. Stakeholder interviews are subject to the availability of the appropriate interlocutors as well as recollection bias. In order to address these concerns, the evaluation team will triangulate all available data to identify evaluation findings and conclusions and mitigate the impact of data gaps and quality issues. In addition, all evaluation findings will be verified against the available literature subject to being finalized.

Available Time and Resources

This evaluation must be implemented utilizing minimal time and resources based on the restraints associated with IDEV's work program, the CEDR and the CSP renewal process. As such, the evaluation team will be hard pressed to deliver the evaluation in a timely manner while meeting internal quality requirements. In order to address this challenge, the evaluation team will identify opportunities to complete

54

parts of the evaluation in-house, while leveraging the consulting expertise of the contracted AGRER Team to ensure rigorous data collection in the field. In addition, it is hoped that the coordination of data requests in advance of field missions will help ensure that time in the field is used efficiently and that data collection is targeted to the gaps identified.

Lack of Available Baseline Data

A review of project documents indicates that, although performance targets have been established for many projects, baseline data is not consistently available. This may affect the ability of the AGRER evaluation Team to implement rigorous contribution analysis. To the extent possible, any lack of baseline data will be addressed through the use of appropriate counterfactuals and the identification of alternative secondary data sources, assuming that they are available. Should these alternatives not be available, the AGRER evaluation Team will attempt to address this gap through additional key informant interviews.

Shift in Program Philosophy

As previously indicated, the Bank's assistance to Zambia has undergone significant thematic changes since the beginning of the evaluation period. In particular, large commitments have been made in the energy and transport sectors which are still far from completion. Despite the importance of these interventions to future CSPs, there may be limited information available to contribute to an assessment of effectiveness. In order to address this challenge, the evaluation team has been directed to include certain infrastructure investments in the third project sample group, despite lower level of disbursements. This approach is intended to maximize the amount of relevant information available to inform the development of the next Country Strategy Paper.

Generalizability of the Project Sample

Because the evaluation utilizes a purposive sample to assess the results achieved, strictly speaking, none of the analysis conducted at any stage can be generalized to the remainder of the portfolio. This is particularly relevant in the case of projects sampled into group 3, which will be subject to an in-depth analysis. Despite the resulting limitation, this approach was necessary to ensure that resources are targeted to projects for which it is more likely that results information is available. Appropriate coverage has been ensured by subjecting all completed or closed projects covered by the evaluation period to an in-depth assessment.

5.0 Implementation Plan and Next Steps

The evaluation will be conducted in four phases:

(i) Inception; (ii) Data collection and analysis; (iii) Reporting; and (iv) Dissemination.

This inception report drafted by the IDEV staff and updated by the AGRER evaluation Team deals with the framework for the implementation of the first three phases of the evaluation. The dissemination activities for this evaluation, once completed, will be planned in greater detail by IDEV's Knowledge Management, Dissemination and Capacity Development division.

These phases are described in greater detail below. Indicative timelines for these activities are provided in table 14.

55

Table 14. Proposed Implementation Timelines for the Zambia CSPE

Implementation Phase

Activity Indicative Timeline

Inception Phase Finalization of the Inception Report September 25th

Data Collection and Analysis

Document Review, coding and preparation of data requests

September 21

Interviews with Bank Stakeholders September 29th to October 30th

Field missions (site visits and interviews) September 29th to October 30th

Data Analysis Need Guidance on all subsequent dates & milestones

Completion of PRA Reports and Country Synthesis Reports

Reporting Phase

Preparation and presentation of preliminary findings

Preparation and finalization of technical report

Preparation of draft synthesis report

Finalization of synthesis report

Evaluation process, timelines, deliverables, and allocation of tasks

Summary of the proposed tasks list

First Phase - Inception Phase Task 1.1. Document and literature review Task 1.2. Assessment of Existing Data Task 1.3. Finalisation of the inception report Task 1.4. Soliciting comments on the inception report Second Phase - Data Collection and Analysis Phase Task 2.1. Document and Literature Review and Project portfolio assessment Task 2.2. Interviews with key informants including Electronic Surveys Task 2.3. Case studies (site visits; will be determined based on the size of the final sample) Task 2.4. Data analysis and Elaboration of Sector Reports Third Phase of study - Reporting Phase Task 3.1. Completion of Project Results Assessment Templates Task 3.2. Completion of Country-level templates for CEDR Synthesis Task 3.3. Presentation of preliminary findings to stakeholders Task 3.4. Production of draft technical report Task 3.5. Soliciting comments on the draft technical report (incl. reference group, peer review) Task 3.6. Finalisation of technical report

5.1. Detailed Tasks description

56

The tables below detail each task and sub-tasks, and focus on answering the following questions, (i) Why? (Objective) (ii) What? (Deliverables), and (iii) How? (Resources), within the timeframe of 4 months, as defined in the ToRs for the Evaluation

With the understanding that IDEV staff and the Consultant will work closely at all stages of the implementation of this evaluation, our work plan includes all activities required to complete successfully this evaluation, even activities under the sole responsibility of the IDEV team.

Task 1 Inception Phase

Objective: To assess and verify the existing data for the purposes of finalising the draft evaluation inception report and enable the mission to start in an agreed framework

Deliverables: Inception Report

1.1 Document and literature review

Prior to the first field phase, the Team Leader (TL) will start reviewing all necessary documents needed to understand the contextual and operational frameworks of this evaluation. He will gather electronic copies of various studies, reports and all useful documents and data. This includes all key documents related to the Zambian Government's Development Priorities and the Bank's Strategy, commitments, and operations in Zambia, technical and financial documents and reports (notably M&E reports, mid-term and end-of-term evaluation reports) that the Bank will have gathered. A complete set of documents was sent by the IDEV team to the AGRER TL so that the review process may begin.

The Team Leader will be supported by the Research Officer, in particular for documentary review regarding regional strategies and operations. During this document review, the experts will also focus on identifying and listing the key stakeholders to be met by the evaluation team.

1.2 Assessment of Existing Data

As indicated in the ToR, the objective of the Inception Phase will be to assess and verify the existing data. This includes mapping, reviewing and verifying existing evidence from scoping missions, project documents and available evaluative information.

1.3 Finalisation of the inception report

After this data assessment, AGRER will communicate with to discuss their approach, methodology and the work schedule for undertaking the Evaluation. This will be done on the basis of the draft inception report elaborated by the IDEV team and the findings and suggestions of the TL.

This exchange with the IDEV team will also be the opportunity to discuss administrative matters and logistics, arrange domestic counterparts, set the arrival of the sectoral experts, planning joint field trips between IDEV and the experts, etc.

The TL will then assist in the finalisation of the inception report for the evaluation.

Agreeing on the scope of the evaluation is a key issue. As indicated in point 2.2 referring to a two-step approach, it is crucial to clearly clarify in the Inception report which of the national and multi-national projects will be subjected to a more detailed assessment.

1.4 Soliciting comments on the inception report

57

The Team Leader will then submit the inception report for final comments and suggestions. It is understood that the report will undergo an internal peer-review as well as external expert review. The peer-reviewers will focus on methodology and on the quality and soundness of the conclusions and recommendations. In addition, the inception report will be reviewed by the Stakeholder Reference Group (SRG) which will provide comments and suggestions on the inception report. After this process, the inception report will be finalised addressing the comments and suggestions received, and submitted to the IDEV Task Manager/Evaluation Team for final approval.

Resources TL: 7 WD

KE2: 3 WD

IDEV Evaluator General IDEV Manager IDEV Task Manager IDEV Evaluation Officer IDEV Junior Consultant Stakeholder Reference Group (SRG) members

Task 2 Data Collection and Analysis Phase

Objective: To collect primary data to complement the review of project documents and fill identified data gaps

Deliverables: Sector Reports compiling findings on identified data gaps and recovered data

2.1 Overview

The objective of the Data Collection and Analysis Phase will be to implement the finalised inception report and collection of primary data to complement the review of project documents and to fill identified data gaps. Subsequently, all available data will be triangulated against the approved evaluation matrix to identify evaluation findings and conclusions.

Activities to be implemented as part of this phase include:

A document and literature review of CSPs, quality-at-entry assessments, project documentation, Bank-wide and Sector Strategies and relevant evaluations;

A review of available country statistics as well as data from relevant international assessments (e.g. CPIA scores);

Mapping, reviewing and verifying existing evidence from project documents and available evaluative information;

Interviews with key informants within the Bank;

Interviews with country stakeholders, including government officials, field office staff, civil society organizations, project implementation units and local beneficiaries; and

site visits for selected projects.

This phase will begin with a Workshop to be held in Pretoria on September 29 in which there will be a presentation and discussion of the Theory of Change elaborated by the AfDB.

58

This initial phase will culminate in the production of Project Results Assessment Reports for selected projects, and Country Synthesis Reports to support the CEDR. In addition, a data matrix will be created to map the available data to the evaluation framework and identify findings, conclusions and recommendations.

2.2 Document and Literature Review and Project portfolio assessment -

Each sectoral expert (including the TL in charge of Private sector and Finance portfolios) will have 1 day to review documentation and project portfolio specific to their sectors. This will enable them to have a broad view and help them to familiarize with the institutional and operational environment of these projects. It will also enable them to distinguish which completed or near-completed projects that will be subject to more detailed assessments in the next phase of interviews and site visits, as well as to identify and plan interviews with the key stakeholders to be met by the evaluation team.

It is important to note that the document review is expected to cover the full portfolio as indicated in point 2.2 referring to the two-step approach. It also includes multi-national projects, as this evaluation will cover those operations which are not already being addressed by an ongoing Regional Integration Strategy Evaluation.

2.3 Interviews with key informants

The TL and all sectoral experts will arrive in Lusaka to prepare for interviews and site visits. Due to the limited number of days for this task, experts will focus on key stakeholders for their interviews. These interviews will follow the format prepared by the IDEV team. It is estimated around half a day per interview including transcribing notes.

2.4 Case studies (site visits; will be determined based on the size of the final sample)

All field missions will be conducted in cooperation with, and participation of the IDEV Task Manager and the IDEV evaluation Team. Therefore sectoral experts will agree with IDEV on the mission schedule and the sample of stakeholders and project sites to be visited. Indeed, due to the limited number of days for this task, not all projects sites will be visited but only a sample. It is foreseen around a full day per site visit, including transcribing notes.

2.5 Data analysis and Elaboration of Sector Reports

Following interviews and site visits, sectoral experts will have a few days to continue in-depth project assessments on their portfolio. This phase will also include data coding and analysis. The aim of this task is to collect primary data to complement the review of project documents and fill identified data gaps. Subsequently, all available data will be triangulated against the approved evaluation matrix to identify evaluation findings and conclusions. Sector Reports compiling findings on identified data gaps and recovered data will be issued by each expert. These reports are not officially requested but are preliminary reports with information necessary to complete the Project Results Assessment Templates (PRA) and Country-level templates for CEDR Synthesis.

Resources TL: 13 WD KE3: 13 WD KE4 : 17 WD KE5 : 14 WD KE6 : 6 WD

59

IDEV Evaluator General IDEV Manager IDEV Task Manager IDEV Evaluation Officer IDEV Junior Consultant.

Task 3 Reporting Phase

Objective: To consult key stakeholders on evaluation findings and obtain validation on evaluation recommendations to be incorporated in the final technical report

Deliverables:

a) PRA templates and Country-Level Template for CEDR Synthesis completed b) Draft Technical Report c) Final Technical Report

3.1 Overview

The reporting phase will proceed in three stages with the objective of: (a) consulting key stakeholders on the preliminary evaluation findings; (b) validating the preliminary findings and identifying evaluation recommendations; and (c) preparing the technical and synthesis reports.

The sequence will be as follows: First, preliminary findings will be presented to IDEV staff and key stakeholders for discussion and the identification of recommendations. Next, detailed technical reports will be drafted to present the available evidence and analysis to support each evaluation finding. Finally, the IDEV evaluation team will produce a synthesis report which summarises the evaluation findings, conclusions and recommendations for presentation to Bank management.

The activities to be carried out at this stage include:

Production of a presentation of preliminary findings and analysis;

Implementation of a workshop with the evaluation reference group to solicit comments on preliminary findings and identified recommendations;

Production of a technical report which outlines all findings, conclusions, recommendations and supporting evidence; and

production of a draft and finalised synthesis report based on the comments of the stakeholder reference group, expert reviewers, IDEV peers and management.

3.2 Completion of PRA templates

Following the data collection and analysis phase, findings and information compiled in several sector reports will now be transcript in the Project Results Assessment (PRA) templates. In fact, the resulting data will be used to complete a results template wherein a rating is provided for various dimensions of relevance, effectiveness, efficiency and sustainability and a detailed justification is provided.

The PRA process aims to enhance the assessment of development results achieved at the outcome level. All completed and closed projects as well as ongoing projects approved after 2004 which are at least

60

75% disbursed will be subject to an in-depth assessment of the results chain consistent with the approach recommended in the Evaluation Cooperation Group's "Big Book on Evaluation Good Practice Standards," including:


Examination of the critical assumptions and expectations inherent in the project's design;



This template is currently in the process of being finalised by the IDEV evaluation team. It could be necessary to evaluate workload in terms of working days necessary for one expert to fill in one of these templates. According to the number of projects under their supervision, the number of working days could be adapted accordingly.

3.3 Completion of Country-Level Template for CEDR Synthesis

After completing the project-level assessments, sectoral experts will provide country-level information to support the CEDR synthesis. This will contribute to an overall assessment of the program as a whole along the lines of the evaluation questions listed in an earlier section of this Inception report. These Country-level templates will go beyond the project evaluation completed through PRA to give an overall view on the programmes at country-level. The Information gathered during the documentary review, the data collection phase and the PRA above will fully inform the country information requirement for the CEDR. It is understood that no additional effort is needed in terms of interviews or site visits.

This assessment will account for the management of the Bank's portfolio, including the selectivity of the Bank's assistance, the extent of engagement and policy dialogue with RMCs and the quality of risk management. Furthermore, this assessment will involve consideration of contextual factors within the country which may impact the implementation of the Bank's interventions, such as public procurement capacity.

Similar to the PRA, this assessment will be based on data already collected to address the evaluation questions and will follow a standardised template which IDEV is currently in the process of finalising. It could be necessary to evaluate workload in terms of working days necessary for one expert to fill in one of these templates. According to the number of projects under their supervision, the number of working days could be adapted accordingly.

3.4 Presentation of preliminary findings to stakeholders

Subsequent to data collection and template completion, the consultant team will produce a presentation of emerging findings to support a consultative workshop with IDEV staff, management, peer reviewers and a Bank reference group. This will foster discussion and the identification of recommendations. A one-day workshop will be organized in Lusaka under the joint coordination of the Team Leader and the IDEV Task Manager. Each sectoral expert will present a PowerPoint on findings regarding his sector that will be followed by discussions. This will lead to preliminary recommendations to be further developed by the consultants at the next stage. Once finalised, this presentation may be used for outreach purposes with the field office and regional department.

61

3.5 Production of draft technical report

The AGRER consulting Team will then incorporate comments from these stakeholders and propose evaluation recommendations taking also into account the feedback received. Each sectoral expert will produce a sectoral report under the instructions of the TL who will be in charge of compiling all reports into 1 draft technical report. The TL will be supported by the Research Expert who will have 5 working days to support this compilation and to proceed to a quality control on the overall document thanks to his specific experience in analysis and formulation of complex multi-sector documents within East African regional programmes.

3.6 Soliciting comments on the draft technical report (incl. reference group, peer review)

The draft technical report will then undergo an internal peer-review as well as external expert review. The peer-reviewers will focus on methodology and on the quality and soundness of the conclusions and recommendations. The feedback provided by the Peer-reviewers (both internal and/or external) will be used to refine the planning documentation and final draft of the evaluation, as required. In addition, the evaluation will be reviewed by a Stakeholder Reference Group (SRG).

3.7 Finalisation of technical report

The Team Leader will submit a Final Technical Report consolidated from the revisions following the comments on the Draft Technical Report. The Consultant will request the Client’s approval. This will coincide with the conclusion of the Consultants’ mission.

Finally, the IDEV evaluation team will produce a finalised synthesis report which will summarise the evaluation findings, conclusions and recommendations for presentation to Bank management. This will also include the production of a draft and final summary report. It will also be the Bank’s task to proceed with the dissemination and communication activities of all outputs, as per the Bank’s requirements.

Resources TL: 13 WD KE3: 13 WD KE4: 17 WD KE5: 14 WD KE6: 6 WD

IDEV Evaluator General IDEV Manager IDEV Task Manager IDEV Evaluation Officer IDEV Junior Consultant. Stakeholder Reference Group (SRG) members

Mission Plan – Zambia CSPE - Preliminary Agenda for visits to South Africa and Zambia

Part 1: South Africa (September 29 to October 2)

Dates Team Members Activities

September 29-30 Madhu, Erika, Interview task managers at SARC (9)

62

Paolo Workshop on ToC with SARC (TBD)

October 1-2 Madhu, Erika, Paolo

Interviews with partners:

Development Bank of South Africa (3)

ATMS.AMSCO African Management Services (request contact)

Ausenco International (request contact)

Bateman Engineering (request contact)

European Investment Bank (2)

AFD (French Development Agency) (1)

ABSA Capital (Johannesburg)

Part 2: Zambia (October 5 to October 30)

Dates Team Members

Activities / Interviews

October 5 and 6

Madhu, Erika, Jim & AGRER team

Theory of Change Workshop with ZMFO

Task Managers and contacts at ZMFO (6)

October 6 to 9

Madhu, Erika, Jim & AGRER Team

Zambia Institute for Policy Research

Zambia Chamber of Commerce and Industry

Zambia Association of Manufacturers

International Growth Centre

Jesuit Centre for Theological Reflection

Develop Assistance from Person to Person

Anti-Corruption Commission

Southern Africa Development Community Focal Point

JICA

COMESA

IFAD

European Union

World Bank

DANIDA

DfID

IMF

USAID

NORAD

Embassy of Finland

October 12-23

Team 1 - Finance, Private Sector and Power (Erika, Paolo and Power Expert)

Interviews

Cetzam Financial Services (Lusaka)

Zanaco (Lusaka)

Lumwana Mining Company / Barrick Gold (Solwezi)

Pulse Financial Services (Lusaka)

ZESCO (Lusaka)

Golder Associates (Lusaka)

African Mining Consultants (Kitwe)

63

Zambia Environmental Management Agency (Lusaka)

Office for the Promotion of Private Power Investment (tbd)

TATA Construction (Lusaka)

Zambia Industrial Mining Company (ZCCM; Lusaka)

SEPCO Construction

Lunsemfwa Hydro Power Company (Kabwe)

Copperbelt Energy Company (Kitwe)

Energy Regulation Board (Lusaka) Site visits

Investrust Zambia (Lusaka)

ZANACO Guarantee Facility / LOC (Lusaka)

Cetzam Financial Services (Lusaka)

Lumwana Mine (Solwezi)

Itezhi-Tezhi Power and Transmission Project (Itezhi-tezhi Dam, Mumbwa, Lusaka)

October 12-23

Team 2 - Agriculture (Agriculture Expert and Jim/Madhu)

Interviews

CETZAM (Lusaka)

DAPP (Lusaka)

IFAD (Lusaka)

Lake Tanganyika Authority (Ngagara)

Environmental Council of Zambia (Lusaka)

National Livestock and Epidemiological Information Centre (Lusaka)

Site Visits

Small Scale Irrigation Project (Chongwe, Mazabuka, Sinazongwe)

Tanganyika Integrated Regional Development Program (Mpulungu, Nsama)

Community Water Management Improvement (Chingola, Masaiti, Mkushi, Kapiri Mposhi)

October 12-23

Team 3 - Water and Transport (Infrastructure Expert and Jim/Madhu)

Interviews


Kitwe City Council (Kitwe)

Kalulushi City Council (Kalulushi)

National Water Supply and Sanitation Council (Lusaka)

Nkana Water and Sewerage Company (Kitwe)

Road Sector Investment Program (ROADSIP; Lusaka)

Kazungula Bridge Authority (Kazungula) Site Visits

Nkana Water Supply and Sanitation Project (Chambishi, Kitwe, Kalulushi)

Multi-Nacala Road Corridor Project (Luangwa-Chipata-Mwami)

October Ministry Interviews (Lusaka)

64

24-30 Madhu, Jim, Erika Governance and Ministry Interviews

Office of the President - Cabinet Office


Ministry of Agriculture, Food and Fisheries

Ministry of Lands, Natural Resources and Environmental Protection


Ministry of Education, Science, Vocational Training and Early Education

Ministry of Mines, Energy and Water Development

Ministry of Environment and Natural Resources

National Road Fund Agency

Central Statistics Office

Private Sector Development Reform Program

Zambia Development Agency

National Climate Change Secretariat

Zambia Revenue Authority

Zambia Public Procurement Authority

Office of the Auditor General

Ministry of Gender and Child Development

Zambia Privatisation Agency

Ministry of Works and Supply

Zambia National Tender Board

Communication & Outreach Strategy In addition to the Bank's Board, the target audience for this evaluation will include:

a) The governments of both Zambia and non-regional member countries who may use these evaluations to account to their constituents and inform their future interactions with the Bank;

b) (ii) Bank management and staff, who may utilize the findings, recommendations and lessons learned identified in the report to inform their work; and (iii) beneficiaries in Zambia, who may use the evaluation to hold their government to account.

Given that the target audience for this evaluation includes both internal and external stakeholder groups, dissemination and communication tools will be tailored to the needs of these groups, as necessary. Proposed means of dissemination for each stakeholder group are described in greater detail below.

Primary Internal Stakeholders

The primary internal stakeholders of the Zambia CSPE include the Bank's Committee on Operations and Development Effectiveness (CODE) as well as the Bank's Board and Executive Directors. It is anticipated that these stakeholders will use the evaluation to inform a range of executive decisions, including the approval of the next CSP as well as new projects to be implemented in Zambia.

65

The results of the evaluation will be communicated to these stakeholders through targeted meetings, consultations and workshops as well as print and electronic distribution of the final report.

Secondary Internal Stakeholders

The secondary internal stakeholders of the CSPE include Bank operational managers and staff. It is anticipated that these stakeholders will use the report for learning purposes. For example, key lessons from the report may be used to inform the development of operational policies and strategies as well as projects.

The evaluation will be disseminated to this stakeholder group through learning events such as Evaluation Community of Practice (ECOP) meetings and "Evaluation Week," workshops, online tools such as the Evaluation Results Database (EVRD) as well as print and electronic distribution of the final report.

External Stakeholders Groups

External stakeholder groups for this evaluation include beneficiaries in Regional Member Countries, academics, media and the general public. It is anticipated that this stakeholder group will use the evaluation for both accountability and educational purposes. For example, the evaluation may be used to hold both regional and non-regional member governments to account for development activities and funding decisions and to learn more about evaluation and development activities in Africa.

The evaluation will be disseminated to these stakeholders through print and online distribution of the final and inception reports as well as through press releases and academic conferences. In 2015-16, IDEV will also identify new ways of leveraging social media to reach these stakeholders, including the use of twitter and Facebook to share informative infographics about the evaluation and draw attention to its webpage.

Management and Quality Control The Zambia CSPE will be carried out under the direction of Rakesh NANGIA, Evaluator General of the AfDB's Independent Development Evaluation function, and the guidance and supervision of Samer HACHEM, Manager, high level evaluations. Mr. Nangia and Mr. Hachem will retain responsibility for the final quality review and approval of all deliverables. The evaluation will be subject to a series of other quality control mechanisms. The final inception report and draft synthesis report will be reviewed by a team of internal IDEV peer-reviewers as well as an external expert.

These reviewers will comment on the quality and rigour of the report and the soundness of the conclusions and recommendations in accordance with the OECD-DAC Quality Standards for Development Evaluation (www.oecd.org/dac/evaluation) and, ECG Good Practice Standards for Country Strategy and Program Evaluation and the ECG guidance on evaluating private sector operations.28

In addition, the inception and synthesis reports will be reviewed by a Stakeholder Reference Group (SRG). The SRG for this evaluation will comprise:

28 ECG Big Book on Evaluation Good Practice Standards

http://www.oecd.org/dac/evaluation

66

IDEV peer reviewers and management;

Bank Management and stakeholders from relevant AfDB sector departments and country offices; and

External stakeholders within RMCs as appropriate.

This group will be asked to provide comments and suggestions on the inception report, preliminary findings and draft synthesis reports as well as to correct any factual inaccuracies.

List of Annexes

Annex A - Zambia's progress against MDG Indicators

Annex B - The Bank's project portfolio in Zambia (2002-2014)

Annex C - List of Interviewees

Annex D - Interview Protocol

Annex E - Project Results Assessment Template

Annex F - Project Results Assessment Guidance

67

Annex A - Zambia's Progress Against MDG Indicators

Progress Against Selected MDG Indicators

Indicator 2006-2008 2010-2013 Target 2015 Likelihood

Goal 1 - Eradicate Extreme Hunger and Poverty

Proportion of population below USD 1 (PPP) per day (%) 51.0 42.3 29.0 Unlikely

Prevalence of underweight children (%) 14.6 13.3 12.5 Likely

Goal 2 - Achieve Universal Primary Education

Net enrolment in primary education (%) 97.0 93.7 100 Attention Required

Primary Six Completion Rate (%) N/A 90.9 100 Attention Required

Literacy Rate 15-24 year-olds (%) 70 89 100 Attention Required

Goal 3 - Promote Gender Equality and Empower Women

Ratio of females to males primary enrolment (girls per 100 boys)

0.97 0.99 1 Likely

Ratio of females to males secondary enrolment (girls per 100 boys)

0.73 0.89 1 Attention Required

Proportion of literate females to males (15-24) 0.8 0.87 1 Likely

Goal 4 - Reduce Child Mortality

Infant mortality rate (per 1000 live births) 119 138 63.6 Unlikely

Under-five mortality rate (per 1000 live births) 70 76 35.7 Unlikely

Goal 5 - Improve Maternal Health

Maternal Mortality Rate (per 100,000 live births) 449 483 162 Unlikely

Proportion of births attended by skilled health personnel (%)

46 46.5 100 Unlikely

Goal 6 - Combat HIV/AIDS, Malaria and other Diseases

HIV Prevalence rate (%) N/A 14.3 15.6 Met

Population with advanced HIV infection with access to ARVs

N/A 79 80 Likely

New malaria cases per 1000 individuals N/A 330 Less than 225 Attention Required

Malaria fatality rate per 1000 individuals N/A 34 11 Attention Required

Goal 7 - Ensure Environmental Sustainability

Proportion of the population using an improved drinking water source (%)

40 36.9 25.5 Unlikely

Proportion of the population using an improved sanitation facility (%)

36.1% 32.7% 87% Unlikely

Goal 8 - Develop a Global Partnership for Development

ODA (millions US$) 415 480 N/A

1 Latest data available during the specified period.

From African Development Bank 2012-2016 CSP and the 2010 UNDP / Government of Zambia Millennium Development Goals Report

68

Annex B - The Bank's Portfolio in Zambia (2002-2014) Project Code Project Name Status Sector Net Loan

(UA'000) Approval date

Agriculture & Environment

P-ZM-AAC-001 SMALL SCALE IRRIGATION PROJECT*** COMP Agriculture 5,788.57 9/7/2000

P-ZM-AA0-021 PROJECT TO SUPPORT THE LAKE TANGANYIKA INTEGRATED REGIONAL DEVELOPMENT PROGRAMME

COMP Agriculture 20,078.50 6/23/2004

P-ZM-AZ0-001 HUMANITARIAN EMERGENCY ASSISTANCE CLSD Agriculture 325.47 12/2/2005

P-ZM-AAZ-001 ZAMBIA EMERGENCY ASSIST. FLOOD VIC. COMP Agriculture 325.47 6/5/2008

P-ZM-AAC-002 COMMUNITY WATER MANAGEMENT IMPROVEMENT COMP Agriculture 635.46 11/12/2009

P-ZM-AAZ-003 EMERGENCY ASSISTANCE TO 2009 FLOOD MITIGATION COMP Agriculture 650.94 12/6/2010

P-ZM-AA0-015 FINISH SUPPORTED SMALL SCALE IRRIGATION OnGo Agriculture 6,171.27 12/28/2009

P-ZM-AAE-002 PROJECT PREPARATION FACILITY - ZM LISP OnGo Agriculture 300.00 3/29/2013

P-ZM-AAE-001 LIVESTOCK INFRASTRUCTURE SUPPORT PROJECT OnGo Agriculture 12,000.00 6/19/2013

P-ZM-CZ0-001 STRENGTHENING CLIMATE RESILIENCE KAFUE OnGo Environment 24,736.00 10/18/2013

P-ZM-AA0-019 GAFSP-AGRICULTURE PRODUCTIVITY & MKT EP OnGo Agriculture 20,111.15 3/26/2014

P-Z1-AAF-001 LAKE TANGANYIKA DEVELOPMENT PROJECT (ZAMBIA) APVD Agriculture 3,260.00 12/18/2014

Private Sector

P-ZM-BAA-001 LUMWANA COPPER MINING PROJECT CLSD Ind/Mini/Qua 27,472.74 9/27/2006

P-ZM-HAZ-002 INVESTRUST ZAMBIA CLSD Finance 2,278.28 9/24/2008

P-ZM-HAZ-001 ZANACO GUARANTEE FAC./LOC ZANACO- ZAMBIA CLSD Finance 7,812.10 9/24/2008

P-ZM-HAB-001 ZANACO FAPA TA GRANT FOR ZAMBIAN SMES CLSD Finance 638.45 11/10/2008

P-ZM-HB0-002 PFSL- FAPA TA - ZAMBIA CLSD Finance 608.62 7/13/2009

P-ZM-HB0-003 CETZAM FINANCIAL SERVICES PLC OnGo Finance 997.07 10/21/2013

P-ZM-HAB-022 FRB SUBSIDIARY IN ZAMBIA APVD Finance 31,899.94 12/12/2012

Multi-Sector

P-ZM-K00-002 POVERTY REDUCTION BUDGET SUPPORT CLSD Multi-Sector 20,000.00 11/29/2006

P-ZM-K00-003 POVERTY REDUCTION BUDGET SUPPORT II CLSD Multi-Sector 14,901.83 10/29/2008

P-ZM-KA0-004 POVERTY REDUCTION BUDGET SUPPORT III CLSD Multi-Sector 31,900.00 6/30/2010

P-ZM-IZ0-002 FOURTH POVERTY REDUCTION BUDGET SUPPORT CLSD Multi-Sector 15,000.00 9/28/2011

Power

P-ZM-FA0-003 ITEZHI-TEZHI POWER PLANT AND TRANSMISSION LINE PROJECT

APVD Power 36,400.00 6/13/2012

P-ZM-FAB-004 ITEZHI-TEZHI POWER PROJECT APVD Power 24,736.00 6/13/2012

P-ZM-F00-001 MAAMBA COLLIERIES POWER GENERATION PROJECT APVD Power 97,706.50 10/2/2013

Social

P-ZM-IA0-005 SUPPORT FOR SCIENCE & TECHNOLOGY EDUCATION OnGo Social 22,220.00 11/20/2013

Transport

P-Z1-DB0-028 SADC NORTH-SOUTH TRANPORT CORRIDOR IMPROVEMENT STUDY

CLSD Transport 1,450.00 12/1/2006

P-Z1-DB0-080 NACALA ROAD CORRIDOR STUDIES OnGo Transport 256.01 11/25/2009

P-Z1-DB0-063 MULTI-NACALA ROAD CORRIDOR PROJECT - Phase II (ZAMBIA)

OnGo Transport 69,369.00 9/27/2010

P-Z1-DB0-031 KAZUNGULA BRIDGE PROJECT OnGo Transport 51,000.00 12/7/2011

P-Z1-DB0-137 NACALA ROAD CORRIDOR DEVELOPMENT PROJECT– Phase IV

APVD Transport 5,000.00 12/3/2013


P-ZM-E00-005 CENTRAL PROVINCE EIGHT CENTRES WATER SUPPLY AND SANITATION

CLSD Water Sup/Sanit

21,342.00 12/17/2003

P-ZM-E00-009 RURAL WATER SUPPLY & SANITATION PROGRAM CLSD Water Sup/Sanit

15,000.00 10/31/2006

69

P-ZM-E00-008 NKANA WATER SUPPLY AND SANITATION PROJ. CLSD Water Sup/Sanit

35,000.00 11/27/2008

P-ZM-EAZ-002 MULTIPURPOSE SMALL DAMS OnGo Water Sup/Sanit

839.40 9/4/2012

P-ZM-E00-028 RURAL WATER SUPPLY AND SANITATION II APVD Water Sup/Sanit

13,018.10 9/10/2014

REGIONAL PROJECTS

Agriculture

P-Z1-AAZ-002 COMESA AGRICULTURAL MARKETING PROMOTION AND REGIONAL INTEGRATION PROJECT

COMP Agriculture 3,687.00 07/01/04

P-Z1-AAE-007 SADC- STRENGTHENING OF INSITUTIONS FOR COMP Agriculture 13,710.00 05/07/2006

AFRICA FOOD CRISIS RESPONSE COMP Agriculture 20,000.00 24/07/2008

Finance

P-ZA-HAA-009 FIFTH LINE OF CREDIT TO THE DEVELOPMENT BANK OF SOUTHERN AFRICA LIMITED

COMP Finance 192,068.84 03/02/2011

P-ZI-HAA-043 BUSINESS PARTNERS INTERNATIONAL SOUTHERN AFRICA FUND

COMP Finance 4612.70 11/07/2012

P-Z1-HAA-016 PTA BANK 2ND LINE OF CREDIT & TA SUPPORT OnGo Finance 649.81 08/08/2008

P-Z1-HAA-050 PTA BANK LOC-EQUITY 2013 APVD Finance 32,490.90 02/10/2013

Multi-Sector

P-Z1-K00-012 STATISTICAL CAPACITY BUILDING UNDER THE INTERNATIONAL COMPARISON PROGRAM FOR AFRICA***

COMP Multisector 18,140.00 12/10/2004

P-Z1-K00-014 ENHANCING PROCUREMENT REFORMS AND CAPACITY PROJECT***

COMP Multisector 5,660.00 21/07/2006

P-Z1-KF0-033 CAPACITY STRENGTHENING FOR GS- IOC APVD Multisector 616.91 15/07/2013

P-Z1-KF0-021 PIDA CAPACITY BUILDING APDV Multisector 5,600.00 12/12/2013

Social

P-Z1-IAZ-002 AFRICAN VIRTUAL UNIVERSITY PROJECT COMP Social 5,000.00 11/01/04

P-Z1-IB0-007 SADC SUPPORT TO THE CONTROL OF COMMUNICABLE DISEASES

OnGo Social 20,000.00 31/05/2006

P-Z1-IAH-001 SADC CAPACITY BUILDING FOR OPEN AND DISTANCE LEARNING

OnGo Social 15,000.00 20/12/2006

P-Z1-IAZ-006 AFRICAN VIRTUAL UNIVERSITY SUPPORT PROJECT - Phase II (AVU II)

OnGo Social 10,000.00 16/12/2011

Transport

P-Z1-KE0-002 PROGRAMME FOR INFRASTRUCTURE DEVELOPMENT COMP Transport 1679.00 19/10/2005

P-Z1-DB0-041 SADC TECHNICAL ASSISTANCE - TRANSPORT COMP Transport 350.00 12/01/06

P-Z1-DA0-003 COMESA AIRSPACE INTEGRATION PROGRAM ONGO Transport 5,750.00 15/09/2010

P-Z1-DZ0-018 SADC ROAD SUB-SECTOR HARMONIZATION STUDY Aban. Transport 234.62 12/26/07


P-Z1-EAZ-018 SUPPORT TO SADC WSS PROGRAMME OnGo Water sup & sanit

18,231.28 04/06/2009

***To be covered under the current evaluation.

70

Annex C - List of Key Stakeholders

CSPE Zambia - Potential Stakeholder Interviews

Stakeholder Group Stakeholders

Field Office Staff

Peter Rasmussen, Principal Country Economist

Ernest Mdaniso Sakala, Private Sector Investment

Nathan Jere, Procurement Specialist

Phillip Boahen, Country Program Manager

Herbert Chinokoro, Senior Water and Sanitation Manager

Lewis Bangwe, Agricultural Expert

SARC

Mecuria Assefaw, Chief Financial Analyst

Johannes Mtchera Chirwa, Chief PPP Infrastructure Specialist

Benedict Kunene, Principal Education Specialist

Matila Mothobi, Principal Macroeconomist (Governance)

Richard Malinga, Senior Transport Engineer

Elisabeth Muguti, Senior Power Engineer

Kennedy Mbekeani, Director

Joseph Coompson, Chief Agricultural Economist

Owusu Mensha Agyei, Principal Financial Management Officer

Headquarters Staff

Tatsuo Harada, Principal Transport Engineer, OITC2

Rhoda Limbiani Mshana, Senior Investment Officer, EARC

Angela Nalikka, Division Manager, OPSD3

Amadou Bamba Diop, Principal Environmental Specialist, OSAN3

Serring Jallow, Senior Advisor to the Vice President, OSVP

Gabriel Negatu, Director, EARC

Marlene Kanga, Director, ORCE

Alex Rugamba, Director, ONEC

Aymen Ali, Chief Transport Engineer, OITC2

Noel Kulmeka, Chief Socio-Economist, ONEC3

Patrick Opoku-Darkwa, Principal Transport Engineer, ORNG

Justin Murara, Chief Poverty Alleviation Officer, OSHD1

Themba Bhebhe, Chief Operations Officer

Jacob Mukete, Director, OSGE

Kelello Ntoampe, Principal Environmentalist, ONEC3

Moses Ayiemba, Chief Regional Procurement Officer, GHFO

Orision Amu, Chief Regional Program Officer, GHFO

Sunita Pitamber, Acting Director, OSHD

Mohamed El Azizi, Director, OWAS

Government of Zambia

Office of the President - Cabinet Office


Ministry of Agriculture, Food and Fisheries

Ministry of Lands, Natural Resources and Environmental Protection


Ministry of Education, Science and Vocational Training and Early Education

Ministry of Energy and Water Development

Ministry of Environment and Natural Resources


Central Statistical Office

Private Sector Development Reform Program

Zambia Development Agency

71



National Climate change secretariat

Zambia Environmental Management Agency

ROADSIP (Road Sector Investment Program)

Zambia Revenue Authority

Zambia Public Procurement Authority

Office of the Auditor General

Zambian Wildlife Authority

Office for Promoting Private Power Investment

Gender in Development Division (Cabinet office)

Zambia Privatization Agency

Ministry of Works and Supply

Ministry of Gender and Child Development

Project Implementation Units



Nkana Water and Sewerage Company

Department of Infrastructure and Support Services

TATA Consulting Engineers

Ministry of Youths and Sports

Ministry of Education Project Management Unit

Civil Society Organizations

Zambia Institute for Policy Analysis and Research

Zambia Chamber of Commerce and Industry

Zambian Association of Manufacturers

Zambia Private Sector Development Association

International Growth Centre

Jesuit Centre for Theological Reflection

Civil Society for Poverty Reduction

Energy Regulation Board

Coffee Growers Association of Zambia

Zambia National Farmers Union

Credit Union and Savings Association

Water Users' Association

Development Assistance from Person to Person (DAPP)

Environmental Council of Zambia

National Water Supply and Sanitation Council

Kitwe City Council

Kalulushi Municipal Council

Anti-Corruption Commission

Zambia National Tender Board

Women Entrepreneurs' Development Association of Zambia

National Livestock and Epidemiological Information Centre

Kazungula Bridge Authority

Zambia Water Partnership

University of Zambia

Coffee Growers' Association of Zambia

Private Sector Organizations

Cetzam Financial Services

ZANACO

Itezhi-Tezhi Power Corporation Ltd.

Investrust Bank

72



Lumwana Mining Company

Pulse Financial Services Limited

ATMS.AMSCO African Management Services

Zambian Electricity Supply Corporation (ZESCO) Ltd.

Golden Associates Ltd

African Mining Consultants

Ausenco International Pty

Bateman Engineering

Equinox Copper Venture Ltd. (since acquired by Barrick Gold)

Zambia Industrial and Mining Company

SEPCO Construction Corporation

Lake Tanganyika Authority

Development Partners (Partners who have worked with the Bank on specific projects in Zambia)

Japan International Cooperation Agency

Common Market for Eastern and Southern Africa (COMESA)

Embassy of Finland - Lusaka

European Union

Food and Agriculture Organization

World Bank

Danish International Development Agency (DANIDA)

United Nations Development Programme (UNDP)

Norwegian Agency for Development Cooperation (NORAD)

United Nations Children's Fund (UNICEF)

World Health Organization (WHO)

European Investment Bank (EIB)

Dutch Development Agency (FMO)

International Fund for Agricultural Development (IFAD)

Department for International Development (DfID)

Arab Bank for Economic Development for Africa (BADEA)

Kuwait Fund for Arab Economic Development

OPEC Fund for International Development

UN Commission on International Trade Law (UNCITRAL)

International Monetary Fund (IMF)

French Development Agency (AFD)

United States Agency for International Development (USAID)

Development Bank of Southern Africa (DBSA)

Southern Africa Development Community (SADC)

73

Annex D - Interview Protocol

Interview Protocol - Task Managers

Issue #1 - Relevance

1. To what extent does this project reflect the development priorities of the Zambian government and the needs of target beneficiaries?

Probes

How were the needs of beneficiaries assessed in order to inform the project design?

How was the Government of Zambia engaged to ensure alignment with their development priorities and inform the project design?

2. To what extent does this project reflect the Bank's key strategies and policies, including the CSP? How were these strategies and policies considered in the design and implementation of the project?

Issue #2 - Effectiveness

3. To what extent have the project outputs been delivered as planned? What challenges have emerged in this regard and how were they addressed?

4. To what extent did the project achieve its intended outcomes? What are some of the factors that contributed to or hindered the achievement of outcomes and how were they addressed? Probes

To what extent did the project reach the target beneficiaries?

5. What were the development impacts of the project? Did the project result in any unintended impacts?

Issue #3 - Efficiency

6. What were some of the challenges that were encountered in the design and implementation of this project? How were these challenges mitigated?

Probes

What were the key sources of delay?

How could these challenges be better addressed in future?

7. In your opinion, were the results of this projects appropriate given the resources and time devoted to its implementation (cost)? Are there opportunities to increase the achievement of results at the same cost?

74

Issue #4 - Sustainability

8. How were issues of sustainability identified and addressed in the design and implementation of this project? In your view, what is the likelihood that the results of this project will be sustainable?

Probes

What role has policy dialogue and engagement with RMC governments and other development partners played in promoting the sustainability of projects?

Issue #5 - Cross-cutting themes

9. How were gender and regional disparities considered in the design of this project? How have these issues been monitored and addressed in implementation?

Probes

How could these issues be better addressed for future projects?

To what extent has institutional support been provided to help address these challenges?

10. How were environmental sustainability issues considered in the design of this project? How were these issues monitored and addressed in implementation?

Probes

How could these issues be better addressed for future projects?


Issue #6 - Design and Delivery

11. To what extent did you encounter implementation delays related to administrative, technical or implementation capacity of the RMC Government or implementation unit? How did you adapt or modify the project design or implementation as a result?

Probes

How is the Bank addressing these issues through technical assistance or capacity development? Do you feel this approach is sufficient?

Issue #7 - Management for Results

12. How has supervision contributed to project implementation? How is information from supervision missions used to guide project management? Probes

How regularly has the project been supervised?

Do you feel that the current level of supervision is adequate?

75

13. How were implementation risks considered in the design and delivery of the project? How are implementation risks being tracked?

Probes

What support have you received from the Bank in this regard?

14. What were some of the challenges you have faced in tracking the results of this project? To what extent have you been able to implement a performance measurement framework? Probes

Were you able to find appropriate baseline data?

How regularly is performance tracked against the PMF?

How has this information been used to guide project management and implementation?


15. To what extent did you incorporate lessons learned from previous projects into the implementation of this project? How are you harvesting lessons from this project for future use? Probes

Are the available "lessons" from past projects sufficient to promote the design of good projects?

Interview Protocol - Country Team


1. To what extent does the CSP reflect the development priorities of the Zambian government and the needs of target beneficiaries in the country?

Probes

How have you assessed the needs of beneficiaries to inform the design of the CSP?

How have you engaged the Government of Zambia to ensure alignment with their development priorities and inform the design of the CSP?

2. To what extent has the Bank's interventions reflected the Bank's key strategies and policies,

including the CSP? How have these strategies and policies been considered in the design and implementation of projects?


3. To what extent were the outputs of the Bank's interventions delivered as planned? What were some of the challenges were encountered in this regard and how were they addressed?

76

4. To what extent have the intended outcomes of the Bank's CSPs been achieved? What are some of the factors that contributed to or hindered the achievement of outcomes and how were they addressed? Probes

To what extent did the Bank's interventions reach target beneficiaries?


5. What were some of the challenges that have been encounter in the implementation of the Bank's Country Strategy in Zambia? How have these challenges been mitigated?

Probes

To what extent have outputs been delivered on time and on budget?




6. How were issues of sustainability identified and addressed in the design and implementation of the Bank's Country Strategy? In your view what is the likelihood that the results of the portfolio will be sustainable?

Probes

To what extent has the Bank sought to engage and build partnerships with beneficiaries, civil society and local government in order to promote the sustainability of results?

How are risks to sustainability being monitored?


7. How were gender and regional disparities and environmental sustainability considered in the design of the Bank's Country Strategies? How have these issues been monitored and addressed in project design and implementation?

Probes

How could these issues be better addressed in future?



8. What measures were undertaken to promote Quality at Entry of the Bank's Country Strategy for Zambia? To what extent have these measures been successful?

Probes

How strategic has the Bank been in identifying areas of intervention in Zambia which are aligned with its strengths and expertise?

77

How has the Bank sought to ensure that its interventions complement those implemented by other development partners?

9. To what extent has the Bank maintained its policy dialogue and engagement with the Government of Zambia and other development actors within the country? How has this engagement added to the implementation of the Bank's country program?

Probes

How could the Bank's policy dialogue be better implemented?

To what extent has the Bank sought to build partnerships with other development actors and civil society in implementing its program and promoting key reforms?

10. To what extent has the Bank been proactive in sharing knowledge, providing technical assistance and conducting Economic and Sector Work? Probes

How could the Bank's research and technical assistance be made more strategic in future?

11. How has the Bank sought to leverage the financial resources of potential partners, including other development partners and the private sector?

Probes

To what extent has the Bank been strategic in identifying these opportunities for partnerships? How could the current approach be improved?

Issue #7: Management for Results

12. What role has the field office played in supervising the implementation of projects? How is information from supervision missions used to guide project management? Probes

How regularly are project supervised?

What are some of the challenges you face in ensuring appropriate supervision?

13. How were implementation risks considered in the design and implementation of the Country Strategy? How are implementation risks being tracked?

14. What role does the country office play in measuring the results of projects? What were some of the challenges you have faced in tracking the results of the Bank's interventions? Probes

Are suitable baseline data available?


78


15. To what extent have lessons learned from the Bank's previous CSPs been incorporated into the design of the current and upcoming CSPs? In general, how are lessons being identified and harvested for future use?

Interview Protocol - Headquarters Staff

1. To what extent does the Bank's intervention in Zambia / this project reflect the development priorities of the Zambian government and the needs of target beneficiaries?

Probes

How were the needs of beneficiaries assessed to inform these interventions?

How did the Bank engage the Government of Zambia to ensure alignment with their development priorities and inform sector-specific interventions?

2. To what extent does the Bank's intervention in Zambia reflect the Bank's key sector strategies and policies? How have these strategies and policies been considered in the design and implementation of projects?


3. To what extent have project outputs been delivered as planned? What challenges have been faced in this regard and how were they addressed?

4. To what extent have the intended outcomes of the Bank's interventions been achieved? What are some of the factors which contributed to or hindered the achievement of outcomes and how were they addressed?

Issue 3# - Efficiency

5. What were some of the challenges that have been encountered in the design and implementation of projects in Zambia? How were these challenges mitigated?

Probes




6. In your opinion, are the results of the Bank's interventions appropriate given the resources and time devoted to its implementation (cost)? Are there opportunities to increase the achievement of results at the same cost?


79

7. Overall, how has the Bank performed in addressing sustainability issues across your sector? In your view what is the likelihood that the results of these projects will be sustainable?

Probes

To what extent has the Bank team seek to engage and build partnerships with beneficiaries, civil society and local government in order to promote the sustainability of results at the sector level?

How are risks to sustainability being monitored?


8. What measures were undertaken to promote Quality at Entry of the Bank's Country Strategy for Zambia? To what extent have these measures been successful?

Probes

How strategic has the Bank been in identifying areas of intervention in Zambia which are aligned with its strengths and expertise?


9. To what extent have your team encountered implementation delays related to administrative, technical or implementation capacity of the RMC Government or implementation unit? How has project design or implementation been modified as a result?

Probes


10. How has the Bank sought to leverage the financial resources of potential partners in your sector, including other development partners and the private sector?

Probes



11. What were some of the challenges you have faced in tracking the results of projects in your sector? To what extent has your team been able to implement rigorous performance measurement regimes? Probes


12. To what extent are lessons learned from previous projects being incorporated into the design of new projects and the Country Strategy? How are lessons being harvested for future use?

80

Probes

Are the available "lessons" from past projects sufficient to promote the design of good projects?

Interview Protocol - Government of Zambia


1. To what extent were the Bank intervention aligned with the priorities of the Zambian Government and the needs of the target beneficiaries?

Probes

To what extent did the Bank engage in consultation and dialogue with the Zambian Government to ensure alignment both at program and project level?

How could alignment and Bank practices be improved in future?

To what extent did the Bank engage in needs assessments and preparatory work during project/program design? How could this be improved in the future?


2. How well did the Bank deliver the intended outputs of the strategy?

Probes

What were the challenges or contributing factors to the achievement of the intended outputs?

How can the Bank improve the delivery of intended outputs in the future?

3. To what extent did the Bank contribute to the intended outcomes of the Country Strategy?

Probes

What factors contributed or hindered the achievement of these outcomes?

What steps were taken to tackle the challenges?

To what extent did the Bank's interventions reach the target beneficiaries?

4. What has been the overall development impact of the Bank’s interventions? Did the Bank's interventions result in any unintended impacts?


5. What were some of the challenges that have been encountered in the implementation of the Bank's Country Strategy in Zambia? How have these challenges been mitigated?

Probes

81




6. In your view, what value has the Bank added to these operations in terms of the achievement of results? To what extent might the results be different if these operations were implemented by other donors or exclusively by the Government of Zambia?


7. How were issues of sustainability identified and addressed in the design and implementation of the Bank's interventions? In your view, what is the likelihood that the results of the portfolio will be sustainable?

Probes

To what extent has the Bank sought to engage and build partnerships with beneficiaries, civil society and local government in order to promote the sustainability of results?

How are risks to sustainability being monitored? How proactive and responsive has Bank staff been in identifying and addressing sustainability issues?

Issue #5 - Cross-cutting Themes

8. How were gender and regional disparities and environmental sustainability considered in the design of the Bank's Country Strategies? How have these issues been monitored and addressed in project design and implementation?

Probes




9. How strategic and selective has the Bank been in identifying areas of intervention in Zambia which are aligned with its strengths and expertise?

Probes


10. To what extent has the Bank maintained its policy dialogue and engagement with the

Government of Zambia and other development actors within the country? How has this engagement added to the implementation of the Bank's country program?

11. To what extent has the Bank shared knowledge and provided technical assistance to support policy development and the implementation of projects? How have these initiatives added value to the Bank's interventions?

Probes

82

How could the Bank's advisory and knowledge work be made more strategic in future?

12. How has the Bank sought to leverage the financial resources of potential partners, including other development partners and the private sector?

Probes



13. To what extent is monitoring and performance measurement being undertaken by RMC actors? Probes

What challenges do you face with regards to collecting performance information for Bank-supported projects?

14. What has been the role of the Bank in providing technical support for performance

measurement? Probes

Do you think that this support is sufficient and adequate? How could this support be made more strategic in future?

Interview Protocol - Project Implementation Units


1. To what extent does this project reflect the development priorities of the Zambian government and the needs of target beneficiaries?

Probes

How were the needs of beneficiaries assessed in order to inform the project design?

How was the Government of Zambia engaged to ensure alignment with their development priorities and inform the project design?

2. To what extent does this project reflect the Government of Zambia's key sector policies and development strategies? How were these strategies and policies considered in the design and implementation of the project?


83

1. To what extent were the outputs of the project delivered as planned? What challenges did you face in this regard and how were they addressed?

Probes

What support did the Bank provide in the implementation of this project? Was this support adequate considering the implementation challenges?

2. To what extent did the project achieve its intended outcomes? What are some of the factors

which contributed to or hindered the achievement of outcomes and how were they addressed? Probes

To what extent did the project reach the target beneficiaries?

3. What were the development impacts of the project? Did the project result in any unintended impacts?


4. What were some of the challenges you encountered in the design and implementation of this project? How were these challenges mitigated?

Probes



5. In your opinion, were the results of this project appropriate given the resources and time devoted to its implementation (cost)? Are there opportunities to increase the achievement of results at the same cost?


6. How were issues of sustainability identified and addressed in the design and implementation of this project? In your view, what is the likelihood that the results of this project will be sustainable?

Probes

What role has policy dialogue and engagement and the promotion of partnerships played in promoting the sustainability of projects?'

How has the Bank supported you in identifying and addressing sustainability issues related to this project? Has this support been sufficient?


7. To what extent do you feel that sufficient consideration was given to gender and regional disparities as well as environmental sustainability in the design of the project? How could these issues be better addressed in future?

84

Probes

How has institutional capacity building been used to address these issues? Do you feel that the institutional support provided has been sufficient?

8. What are some of the challenges you have encountered pertaining to gender and regional

disparities as well as environmental sustainability in the course of implementation? How have these issues been monitored and addressed?


9. To what extent did you encounter implementation delays? How did you adapt or modify the project design or implementation as a result?

Probes

What were the key sources of delay for this project?



10. How has supervision contributed to project implementation? How is information from supervision missions used to guide project management? Probes

How regularly has the project been supervised?

Do you feel that the current level of supervision is adequate?

11. How were implementation risks considered in the design and delivery of the project? How are implementation risks being tracked?

Probes


12. What were some of the challenges you have faced in tracking the results of this project? To what extent have you been able to implement a performance measurement framework? Probes




13. To what extent did you incorporate lessons learned from previous projects into the implementation of this project? How are you harvesting lessons from this project for future use?

85

Interview Protocol - Other Development Partners


1. In your view, what are the key development needs and challenges that Zambia is facing at this time? How has your organization sought to assess and respond to these challenges through your operations?

Probes

What, in your view, are best practices for ensuring the alignment of interventions with development needs?

2. To what extent do you think that the Bank’s program and interventions are aligned to Zambia’s

development priorities and the needs of beneficiaries?

Probes

What do you think should be done in the future to improve alignment?


3. What are some of the major challenges you face in terms of the delivery of outputs and the achievement of outcomes in Zambia? Probes

What are some of the factors which hinder or contribute to the achievement of outcomes and how are they best addressed?

To what extent has your partnership and cooperation with the Bank contributed to the achievement of results?

4. In your experience, to what extent do you feel the Bank has been effective in delivering its

planned outputs and achieving planned outcomes?

Probes

What could the Bank do differently to achieve better results? Issue #3 - Efficiency

5. What are some of the major challenges you have faced in promoting the efficient delivery of projects in Zambia? What are some of the key best practices that have been applied for ensuring the efficiency and value of your operations? Probes

What could be done differently in future to ensure the efficient delivery of programs and projects?

86

6. In your experience, the what extent do you feel that the Bank has delivered its projects efficiently? What could the Bank do differently to promote efficient project delivery?

Probes

In light of the results achieved, to what extent do you feel that the Bank's projects have demonstrated good value for money?


7. How has your organization addressed risks to the sustainability of your interventions through modifications to project design and delivery? What challenges have you faced in this regard?

Probes

How has your organization sought to build institutional capacity through policy dialogue, technical assistance and targeted capacity building? To what extent have these initiatives been successful?

8. In your experience, how effective has the Bank been in managing the risks associated with the sustainability of its projects?

Probes

In your view, how could the Bank better address risks to the sustainability of its operations? Issue #5 - Cross-Cutting Issues

9. How has your organization addressed issues pertaining to gender, regional disparities and environmental sustainability through your operations? What challenges have you faced in this regard?

Probes

How have these issues been monitored and used to inform project management?

In your experience, how effective has the Bank been in addressing these issues?


10. To what extent do you feel that the Bank's interventions complement your own initiatives? How proactive has the Bank been in engaging your organization to promote complementarity in project design?

11. To what extent has the Bank been proactive in engaging your organization to identify potential partnerships? In your view, what could be done to better promote partnerships in future?

87

12. To what extent have you encountered implementation delays related to the administrative, technical or implementation capacity of the Government of Zambia or the implementation unit? How did you address these challenges? Probes

To what extent has your organization provided institutional support to help address these challenges?

In your experience, how has the Bank performed in addressing these challenges? Issue #7 - Management for Results

13. To what extent have you coordinated with other development partners to identify and share lessons from the implementation of projects?

Probes

How proactive has the Bank been in sharing and discussing lessons for the improvement of future projects?

Interview Protocol - Civil Society


1. How well do you think that the Bank’s program and interventions were aligned to Zambia’s development priorities and the needs of beneficiaries?

Probes

What do you think should be done in the future to improve alignment?

To what extent did the Bank engage in needs assessments and preparatory work to support the design of its interventions? How could this preparatory work be strengthened in future?


2. In your view, to what extent did the Bank contribute to its intended development objectives? How could the Bank improve its performance in this regard?

Probes

What were some of key factors which contributed to or hindered the achievement of outcomes? What measures could have been taken to address these factors?

To what extent did the Bank's interventions reach the target beneficiaries?

3. In your view, what were the development impacts of the Bank's interventions, if any? Were there any unintended impacts?


88

4. In your view, what value has the Bank added to these operations in terms of the achievement of results? To what extent might the results be different if these operations were implemented by other donors or exclusively by the Government of Zambia?


5. Overall, how has the Bank performed in identifying and addressing sustainability issues in the design and implementation of its interventions? Probes

To what extent has the Bank been successful in engaging government as required and promoting partnerships and ongoing engagement?

6. Overall, to what extent do you feel that the results achieved through Bank support will be

sustainable? Why or why not?

Probes

To what extent do you feel that appropriate institutional support has been provided to promote the sustainability of results? How could this support be better provided?

How proactive and responsive have Bank staff been in identifying and addressing risks to sustainability?

Issue #5 - Cross-cutting Themes

7. To what extent did the Bank address gender and regional disparities as well as issues of environmental sustainability in its interventions?

Probes


To what extent did the Bank consult and work with your organization and other CSOs to address gender and regional disparities and well as environmental sustainability issues?


8. How strategic and selective has the Bank been in identifying areas of intervention in Zambia which are aligned with its strengths and expertise? How could the selectivity of the Bank's interventions be improved?

9. In your opinion, how active has the Bank been in implementing and maintaining a constructive policy dialogue?

Probes

What opportunities exist to strengthen this dialogue?

How involved were the CSOs in the consultation process during the design of the Bank’s program in Zambia?

89

10. To what extent do you feel that the Bank's activities are complementary to those of other donors and partners within the country?

Probes

How could the complementarity of the Bank's activities with other DPs be improved?

90

Annex E - PRA Assessment Template

PROJECT PERFORMANCE ASSESSEMENT TEMPLATE

(For public sector projects) Name of project Insert here

Country Insert here

Year approved Insert here

Year Closed (if still open, insert that open and also level of disbursement) Insert here

Sector Insert here

PCR available Yes/No

1.1 RELEVANCE

1.1 Relevance of project objectives Rating (1 - 6) The relevance of objectives assesses to what extend the project purpose as specified in the RLF was aligned with the Bank’s CSP and the applicable sector strategies, the

country’s development strategies and the beneficiary needs from design/approval to completion (including any adjustments that were made to the project in view of changes in the applicable policy environment, such as project restructuring).

Insert your evidence here Insert your rating here

1.2 Relevance of project design to achieve those objective The relevance of project design should consider: (i) the extent to which the project’s objectives are clearly stated and focused on outcomes as opposed to outputs; (ii) The realism of intended outcomes in the country’s current circumstances also is assessed; (iii) the extent to which project design adopted the appropriate solutions to the identified problems; (iv) the modifications to project design; (v) the circumstances prevailing at the time of the evaluation.


Overall Rating for Relevance

Insert your rating here

91

2. EFFECTIVENESS The assessment of Effectiveness tests the validity of the anticipated links between the project’s activities, outputs, and intended outcomes (the results chain). Actual, expected and

unintended results of an operation are included in the assessment of Effectiveness. For PBOs the assessment should not only review the extent to which outputs were delivered (i.e. agreed-upon policy reforms took place), but also the degree to which complementary measures necessary for their implementation occurred (e.g. public awareness, policy dialogue and institutional arrangements).

2.1 Achievement of outputs Rating (1 - 6) The assessment of outputs is based on the output execution ratio. It should consider the planned (targets) and actual output or those who are considered on track to be

reached. In determining the final rating, no formula based on a pre-determined weight applied to individual outputs is undertaken. It is considered good practice to select no more than 10 output indicators in the RLF and to take into account the relative importance of the various components of the project in their selection. The overall output rating is based on the percentage of outputs (output execution ratio) that reached or are on track to meet the end of project target.


2.2 Achievement of outcomes The assessment of outcome puts is based on the direct and intermediate outcomes stated in the retrospective project logic model. It should cover at least the core sector indicators.


2.2 Unintended outcomes (if any) The assessment will cover all unintended outcome which came out during the project cycle Insert your evidence here Insert

your rating here

Overall Rating for Effectiveness


92

3. EFFICIENCY The Efficiency assessment attempts to answer two questions: (i) Did the benefits of the project (achieved or expected to be achieved) exceed project costs; and (ii) Were the benefits of the project achieved at least cost? Cost-benefits analysis helps to address the first question. To address the second question a cost-effectiveness analysis is carried out. Good practices suggest also the, In addition to the traditional measures of efficiency (cost-benefit analysis and cost-effectiveness analysis), the Efficiency assessment considers aspects of project design and implementation that either contributed to or reduced efficiency (Timeless and Implementation progress) to the extent they are not already captured in the evaluation’s cost-benefit or cost-effectiveness analysis.

3.1 Cost-benefits analysis Rating (1 - 6) Cost-benefit analysis is carried out to the extent that data is available. The validity of the cost-benefit analysis conducted at appraisal/mid-term review is re-assessed at completion It is a

recommended to use the same model that was developed at appraisal. For PBOs a quantitative assessment will be done if an Economic Rate of Return (ERR) was calculated at appraisal, otherwise an assessment could be done with regards to the contribution of policy reforms to economic growth (if not applicable, indicate N/A for this criterion). Insert your evidence here Insert your

rating here

3.2 Cost-Effectiveness Rating (1 - 6) The analysis considers the cost of alternative ways to achieve project objectives, unit costs for comparable activities, sector or industry standards, and/or other available evidence of the

efficient use of project resources. Insert your evidence here Insert your

rating here

3.3 Timeliness Rating (1 - 6) The timeliness of project implementation is based on a comparison between the planned and the actual period of implementation from the date of effectiveness. For PBOs, the timely releases

of the tranche(s) are assessed through this criterion. The following rating scale applies Insert your evidence here Insert your

rating here

3.4 Implementation progress (IP) Rating (1 - 6) The IP rating will be derived from the IPR that shall be updated in tandem with the PCR preparation. The IP rating takes into account all applicable IP criteria assessed under each of the three

main categories: i) compliance with covenants (project covenants, environmental and social safeguards and audit compliance), ii) project systems and procedures (procurement, financial management and monitoring and evaluation), and iii) project execution and financing (disbursement, budget commitments, counterpart funding and co-financing). The simple arithmetic average of the individual ratings is calculated to derive the final rating. The overall IP rating is provided as follows Insert your evidence here Insert your

rating here

Overall Rating for Efficiency Insert your rating here

93

4. SUSTAINABILITY

The assessment of sustainability considers the extent to which the project has addressed risks during implementation and put in place mechanisms to ensure the continued flow of benefits after completion. It should also evaluate risks to the sustainability of development outcomes and/or the project’s benefits, including the resilience to exogenous factors. The overall rating of the sustainability outcome is the mean of the rating of the following four criteria: i) technical sustainability; ii) financial sustainability, iii) institutional sustainability and strengthening of capacities, iv) ownership and sustainability of partnerships and v) environmental and social sustainability.

4.1 Technical Soundness Rating (1 - 6) The criterion assesses the extent to which the project achievements rely on sound technology using inputs efficiently and providing productivity gains. It

includes operation and maintenance (O&M) facilitation, availability of recurrent funding, spare parts, workshop facilities etc.) Insert your evidence here


4.2 Economic and Financial viability Rating (1 - 6) This criterion assesses the extent to which funding mechanisms and modalities (e.g. tariffs, user fees, maintenance fees, budgetary allocations, other

stakeholder contributions, aid flows, etc.) have been put in place to ensure the continued flow of benefits after project completion, with particular emphasis on financial sustainability. For PBOs the assessment should focus on the financial sustainability of the reforms, as well as the Bank’s policy dialogue to promote financial sustainability of the reforms.

Insert your evidence here


4.3 Institutional sustainability and strengthening of capacities Rating (1 - 6) The criterion assesses the extent to which the project has contributed to strengthen institutional capacities - including for example through the use of

country systems - that will facilitate the continued flow of benefits associated with the project. An appreciation should be made with regards to whether or not improved governance practices or improved skills, procedures, incentives, structures, or institutional mechanisms came into effect as a result of the operation. For PBOs this should include an assessment on the contributions made to building the capacity to lead and manage the policy reform process; as well as the extent to which the political economy of decision making was conducive to reform, the Government’s commitment to reform and how the design reinforced national ownership



4.4 Political and governance environment

94

This criteria assesses the extent political and governance developments that could impact the government’s priorities with respect to the project. This includes (but is not limited to) upcoming elections or an impending change in government; and other factors that could impact the political commitment to the operation or operational engagement and the political decisions required for sustainability of project results (including laws and the provision of counterpart financing). Special attention should be paid to fraud, corruption and other unethical practices resulting from governance failures.


4.5 ownership and sustainability of partnerships Rating (1 - 6) The assessment determines whether the project has effectively involved relevant stakeholders, promoted a sense of ownership amongst the beneficiaries

(both men and women) and put in place effective partnerships with relevant stakeholders (e.g. local authorities, civil society organizations, private sector, donors) as required for the continued maintenance of the project outputs. For PBOs, the assessment should include the extent to which the Government conducted extensive consultations during the preparation and implementation of the PRSP and the extent to which the Bank supported the Government in deepening the consultation processes



4.6 environmental and social sustainability Rating (1 - 6) This criterion would normally only apply to Environmental Category I and II projects. It assesses the extent to which the environmental and social

mitigation/enhancement measures of the project were implemented, the capacity of country institutions and systems and the availabil²ity of funding to ensure the environmental and social sustainability of the operation. The Environmental and Social Safeguards rating in the IPR should be used as guidance.



4.7 Resilience to exogenous factors and risk management Rating (1 - 6) This criterion assesses the extent to which the achievements depend on exogenous factors, such as the terms of trade, the world market prices or the

political situation in neighbouring countries. Insert your evidence here

Overall Rating for Sustainability Insert your rating here

95

Annex F - Project Results Assessment Guidance

PROJECT PERFORMANCE ASSESSMENT FOR PUBLIC SECTOR PROJECT

Rating Guidance Note

Draft of 12 June 2015

96

1. RELEVANCE This section should cover both: (i) the relevance of project objectives; and (ii) the relevance of project design. The relevance assess the degree to which the projects objectives and design remain valid and pertinent as originally planned or as subsequently modified owing to changing circumstances within the immediate context and external environment of that project. Accordingly, the relevance of the objectives and design are assessed at the time of evaluation – not project appraisal. While the evaluation may note that the project objectives and design were relevant at project appraisal, it should assess and rate the extent to which project objectives and design remained relevant towards the evaluation year.

a.) Relevance of project objectives

The relevance of objectives assesses to what extend the project purpose as specified in the RLF was aligned with the Bank’s CSP and the applicable sector strategies, the country’s development strategies and the beneficiary needs from design/approval to completion (including any adjustments that were made to the project in view of changes in the applicable policy environment, such as project restructuring).

Any inconsistencies between the country, Bank and beneficiary priorities should also be reflected upon and reported. The assessment also considers the extent to which the project’s development objective was clearly stated and focused on outcomes and the realism of the intended outcomes in the project setting. For PBOs, it will be particularly important to ensure alignment with the Poverty Reduction Strategy paper (PRSP), Performance Assessment Framework (PAF) or applicable country and Bank sector strategies.

To assess alignment to the national development strategy/plan and to other relevant national sector and thematic strategies it is expected that AfDB project documentation sets out how the proposed Bank operation will contribute to specific national development objective(s), usually time bound and quantified.

For multinational and regional operations, regional development strategies, regional integration roadmaps, etc. should be used as reference for alignment. The preparation and adoption status of the source of alignment should be specified, as well as the eventual endorsement by partners, the period covered by the strategy, its priorities, pillars and key objectives with quantitative targets if available, and the main implementation issues highlighted in the latest review or progress report. Fragile states, often countries coming out of conflict, may not have a PRSP or national development plan in place or such development planning tools may be under preparation at a time when a project is prepared and expected to be presented to the Board for consideration. In such cases, alignment to country transitional results frameworks or interim strategies at the national or sector levels, often available, should be demonstrated together with appropriate references to the Bank’s interim strategy or country dialogue note.

The following rating scale applies:

6 – Highly Satisfactory: It is demonstrated that the project objectives doesn’t have any shortcoming in their alignment with: i) the Bank’s CSP, ii) applicable Bank sector strategies, iii) the country’s development strategies, and iv) the beneficiary needs.

5 – Satisfactory: It is demonstrated that the project objectives have minor shortcomings in the alignment with: i) the Bank’s CSP, ii) applicable Bank sector strategies, iii) the country’s development strategies, and iv) the beneficiary needs.

97

4 – Moderately Satisfactory: It is demonstrated that the project objectives have moderate shortcomings in the alignment with: i) the Bank’s CSP, ii) applicable Bank sector strategies, iii) the country’s development strategies, and iv) the beneficiary needs

3 – Moderately Unsatisfactory: It is demonstrated that the project objectives have significant shortcomings in the alignment with one of the following: i) the Bank’s CSP, ii) applicable Bank sector strategies, iii) the country’s development strategies, and iv) the beneficiary needs.

2 – Unsatisfactory: It is demonstrated that the project objectives have major shortcomings in the alignment with two of the following: i) the Bank’s CSP, ii) applicable Bank sector strategies, iii) the country’s development strategies, and iv) the beneficiary needs.

1 – Highly Unsatisfactory: It is demonstrated that the project objectives have severe shortcomings in the alignment with all of the following: i) the Bank’s CSP, ii) applicable Bank sector strategies, iii) the country’s development strategies, and iv) the beneficiary needs.

b.) Relevance of project design to achieve those objective

This criterion assesses the soundness of the project design and the timing of eventual adjustments that were made during implementation in the scope, implementation arrangements, or technical solutions, to ensure the achievement of the intended results (outcomes and outputs). For PBOs an assessment will be made on the relevance of the prior actions, the policy dialogue and the extent to which the operation could have been more pro-poor in its design and implementation.

The relevance of project design should consider: (i) the extent to which the project’s objectives are clearly stated and focused on outcomes as opposed to outputs; (ii) The realism of intended outcomes in the country’s current circumstances also is assessed; (iii) the quality of risk assessment (assumptions made in the logic model); (iv) the extent to which project design adopted the appropriate solutions to the identified problems; (iv) the relevance of modifications made to project design; (v) the circumstances prevailing at the time of the evaluation. Evaluator should assess to what extent potential negative impacts were identified, their likelihood of occurring and how they might be avoided.

The rating is based on the following scale:

6 – Highly Satisfactory: The project design was fully conducive to achieving the project results. The original design was solid and remained appropriate throughout implementation; no adjustments to the scope, implementation arrangements or technical solutions were required to ensure the achievement of the intended outcomes and outputs.

5 – Satisfactory: The project design was largely conducive to achieving the project results. The original design was solid and remained appropriate throughout implementation; minor adjustments to the scope, implementation arrangements or technical solutions were required to ensure the achievement of the intended outcomes and outputs.

4 – Moderately Satisfactory: The project design was moderately conducive to achieving the project results. The original design was to some extent, sound and remained appropriate throughout implementation; adjustments to the scope, minor implementation arrangements or technical solutions were required and they were carried out in a timely manner to ensure the achievement of the intended outcomes and outputs.

3 – Moderately Unsatisfactory: The design was somewhat conducive to achieving the project results. The original design was either weak or lost its relevance during implementation; major adjustments to the scope, implementation arrangements or technical solutions were required during implementation, but these were done with substantial delays which negatively affected the achievement of the intended outcomes and outputs.

2 –Unsatisfactory. From approval to closure, the design was marginally conducive to achieving the project results. The original design was weak and remained irrelevant. Major adjustments to the scope,

98

implementation arrangements or technical solutions were required during implementation, but these were not done which negatively affected the achievement of the intended outcomes and outputs.

1 – Highly Unsatisfactory. The project design was fully not conducive to achieving the project results. The original design was weak and remained irrelevant during implementation. Major adjustments to the scope, implementation arrangements or technical solutions were required during implementation, but these were not done which negatively affected the achievement of the intended outcomes and outputs.

2. EFFECTIVNESS

The assessment of Effectiveness tests the validity of the anticipated links between the project’s activities, outputs, and intended outcomes (the results chain). Actual, expected and unintended results of an operation are included in the assessment of Effectiveness. For PBOs the assessment should not only review the extent to which outputs were delivered (i.e. agreed-upon policy reforms took place), but also the degree to which complementary measures necessary for their implementation occurred (e.g. public awareness, policy dialogue and institutional arrangements).

a.) Achievement of outputs

The assessment of outputs is based on the output execution ratio (see table below) and the quality of outputs. It should consider the planned (targets) and actual output or those who are considered on track to be reached. In determining the final rating, no formula based on a pre-determined weight applied to individual outputs is undertaken. If possible, select no more than 10 output indicators in the RLF and to take into account the relative importance of the various components of the project in their selection. Any selection needs to be clearly justified.

The overall output rating is based on the percentage of outputs (output execution ratio) that reached or are on track to meet the end of project target.

The following table demonstrates what is expected:

Major Activities Expected Outputs Actual Outputs Outputs execution rate

Outputs quality assessment


6 – Highly Satisfactory: Based on the output execution ratio all the project output targets were reached or are considered on track to be reached by the end of the project in accordance with quality standards.

5 – Satisfactory: Based on the output execution ratio between 90% and 99% of the project output targets were reached or are considered on track to be reached by the end of the project. Corrective actions for off track indicators were implemented in a timely manner to ensure that the end of project targets could be achieved in accordance with quality standards.

4 – Moderately Satisfactory: Based on the output execution ratio between 75% and 89% of the project output targets were reached or are considered on track to be reached by the end of the project. Corrective actions for off track indicators were implemented in a timely manner to ensure that the end of project targets could be achieved in accordance with quality standards.

3 – Moderately Unsatisfactory: Based on the output execution ratio between 50% and 74% of the project output targets were reached or are considered on track to be reached by the end of the project. Corrective actions for off track indicators were not implemented in a timely manner to ensure that the end of project targets could be achieved.

99

2 – Unsatisfactory: Based on the output execution ratio between 35% and 49% of the project output targets were reached or are considered on track to be reached by the end of the project. Corrective actions were not implemented and closely monitored for off track indicators. Poor performance jeopardized the achievement of one or more outcomes of the project.

1 – Highly Unsatisfactory: Based on the output execution ratio less than 35% of the project output targets were reached or are considered on track to be reached by the end of the project. Poor performance jeopardized the achievement of most expected outcomes and the possibility of stopping or suspending the project was considered.

b.) Achievement of outcomes

Outcomes are assessed against the project’s objectives, as contained in the Project Appraisal Report (PAR). The assessment of outcome puts is based on the direct and intermediate outcomes stated in the retrospective project logic model. If the statement of project objectives in the appraisal documents is unclear or is focused on outputs rather than outcomes, the evaluator reconstructs an outcome-oriented statement of objectives using the project’s results chain, performance indicators and targets, and other information including country strategies and interviews with government officials and AfDB staff. The anticipated links between the project’s activities, outputs, and intended outcomes are summarized in the project’s results chain. The results chain is taken from the PAR. If the results chain is absent or poorly defined, the evaluator constructs a retrospective results chain from the project’s objectives, components, and key performance indicators.

The evaluator should make sure that data collection remains open to unintended results that have not anticipated by including some open-ended questions in interviews and questionnaires, and by encouraging reporting of unexpected results. Unexpected benefits, once known about, can be designed into future interventions.

The ECG's 'Big Book on Evaluation Good Practice Standards'29 recommends the following approach in this area:

i. Assess the causal chain in relation to the needs of the target population, collaborating with stakeholders and experts.

ii. Examine the critical assumptions and expectations inherent in the project’s design, reviewing the logic and plausibility of the results chain. Again, this is done in collaboration with stakeholders.

iii. Use available research evidence and practical experience elsewhere, comparing the project with projects based on similar concepts.

iv. Observe the project in operation, focusing on interactions that were expected to produce the intended outcomes.

For PBOs the assessment should not only review the extent to which outputs were delivered (i.e. agreed-upon policy reforms took place), but also the degree to which complementary measures necessary for their implementation occurred (e.g. public awareness, policy dialogue and institutional arrangements). Since PBOs are typically joint with other donors and RMC governments and are implemented through country systems, it may be difficult to attribute a direct link between the specific inputs of the Bank Group (and those of other partners) and the expected results. Therefore, the progress will be measured in terms of the collective efforts of the RMC and other partners, where applicable, while taking into account other external factors.


29 https://www.ecgnet.org/document/ecg-big-book-good-practice-standards

100

6 – Highly Satisfactory: Taking into account the latest value of the outcome indicators and the analysis of other relevant exogenous risks/factors and assumptions, it is plausible to expect that all intended project outcomes were achieved or are likely to be achieved.

5 – Satisfactory: Taking into account the latest value of the outcome indicators and the analysis of other relevant exogenous risks/factors and assumptions, it is plausible to expect that most (75%) intended project outcomes were achieved or are likely to be achieved.

4 – Moderately Satisfactory: Taking into account the latest value of the outcome indicators and the analysis of other relevant exogenous risks/factors and assumptions, it is plausible to expect that a substantial (50%-74%) intended project outcomes were achieved or are likely to be achieved.

3 – Moderately Unsatisfactory: Taking into account the latest value of the outcome indicators and the analysis of other relevant exogenous risks/factors and assumptions, it is plausible to expect that few (25-49%) intended project outcomes were achieved or are likely to be achieved.

2 –Unsatisfactory: Taking into account the latest value of the outcome indicators and the analysis of other relevant exogenous risks/factors and assumptions, it is plausible to expect that few (5-24%) intended project outcomes were achieved or are likely to be achieved.

1 – Highly Unsatisfactory: Taking into account the latest value of the outcome indicators and the analysis of

other relevant exogenous risks/factors and assumptions, it is plausible to expect that very few (<5%) of the

intended project outcomes were achieved or are likely to be achieved.

3. EFFICIENCY

The Efficiency assessment attempts to answer two questions: (i) Did the benefits of the project (achieved or expected to be achieved) exceed project costs; and (ii) Were the benefits of the project achieved at least cost? Cost-benefits analysis helps to address the first question. To address the second question a cost-effectiveness analysis is carried out. Good practices suggest also the, In addition to the traditional measures of efficiency (cost-benefit analysis and cost-effectiveness analysis), the Efficiency assessment considers aspects of project design and implementation that either contributed to or reduced efficiency (Timeless and Implementation progress) to the extent they are not already captured in the evaluation’s cost-benefit or cost-effectiveness analysis.

a.) Cost-Benefit Analysis

Cost-benefit analysis is carried out to the extent that data is available. Where financial and/or economic internal rates of return can be re-estimated at post-evaluation, the details would be presented in appropriate appendixes. These appendixes would present a comprehensive evaluation of the rates of return and the underlying assumptions and the methodology adopted in sufficient detail. The sensitivity analysis indicates the key variables used and the underlying rationale and assumptions for using them. If time series data are used in the projections, such statistical data should be included in the appendix as table(s). Sensitivity tests on EIRR based on possible changes in key assumptions should also be carried out.

The validity of the cost-benefit analysis conducted at appraisal/mid-term review and completion is re-assessed at post-evaluation. It is a recommended to use the same model that was developed at appraisal.

For PBOs a quantitative assessment will be done if an Economic Rate of Return (ERR) was calculated at appraisal, if an independent assessment has been done (for example in existing joint evaluations) with regards to the contribution of policy reforms to economic growth this can be used (if not applicable, indicate N/A for this criterion).

For Economic Performance

101


6 – Highly Satisfactory: If EIRR is equal or above the opportunity cost of capital.

5 – Satisfactory: If (90% of the opportunity cost of capital ≤ EIRR < the opportunity cost of capital)

4 – Moderately Satisfactory: If (80% of the opportunity cost of capital ≤ EIRR < 90% of the opportunity cost of capital).

3 – Moderately Unsatisfactory: If (60% of the opportunity cost of capital ≤ EIRR < 80% of the opportunity cost of capital).

2 – Unsatisfactory: If (40% of the opportunity cost of capital ≤ EIRR < 60% of the opportunity cost of capital).

1 – Highly Unsatisfactory: If EIRR is less than 40 % of the opportunity cost of capital.

For Financial Performance

Projects, which involve an entity that operates on a commercial basis or otherwise depends on cost recovery for sustainability, should undertake financial analysis to establish that, under a set of plausible assumptions, the entity will eventually be able to self-finance its activities.

The financial viability of revenue earning projects is determined on the basis of the project itself, not on the basis of the operations of the entity that owns or operates the project. The principal comparison is between the Financial Internal Rate of Return (FIRR) which represents the rate of return earned on the project and the Weighted Average Cost of Capital (WACC) for the project. If the rate of return exceeds the cost of capital to finance the project it meets the test of financial viability. The following rating scale applies:

6 – Highly Satisfactory: If (FIRR ≥ 1.25 x WACC).

5 – Satisfactory: If (1.1 x WCC ≤ FIRR < 1.25 x WACC)

4 – Moderately Satisfactory: If (WCC ≤ FIRR < 1.1 x WACC)

3 – Moderately Unsatisfactory: If (0.85 x WCC ≤ FIRR < WACC)

2 – Unsatisfactory: (If (0.7 x WCC ≤ FIRR < 0.85 x WACC)

1 – Highly Unsatisfactory: If FIRR < 0.7 x WACC.

b.) Cost-Effectiveness

The analysis considers the cost of alternative ways to achieve project objectives, unit costs for comparable activities, sector or industry standards, and/or other available evidence of the efficient use of project resources. Since one objective of providing funds as PBO rather than project is to reduce transaction costs, evidence can be collected to assess whether this has happened.

c.) Timeliness

The timeliness of project implementation is based on a comparison between the planned and the actual period of implementation from the date of signature. For PBOs, the timely releases of the tranche(s) are assessed through this criterion. The following rating scale applies:

6 – Highly Satisfactory: The ratio of planned implementation time (as per PAR) and actual implementation time from the date of effectiveness is expected to be >1.

5 – Highly Satisfactory: The ratio of planned implementation time (as per PAR) and actual implementation time from the date of effectiveness is expected to be 1.

4 – Satisfactory: The ratio of planned implementation time (as per PAR) from the date of effectiveness and actual implementation time from the date of effectiveness is expected to be <1 and ≥0.90.

102

3 – Moderately Unsatisfactory: The ratio of planned implementation time (as per PAR) from the date of effectiveness and actual implementation time from the date of effectiveness is expected to be <0.90 and ≥0.80.

2 – Unsatisfactory: The ratio of planned implementation time (as per PAR) from the date of effectiveness and actual implementation time from the date of effectiveness is expected to be <0.75 and ≥0.80.

1 – Highly Unsatisfactory: The ratio of planned implementation time (as per PAR) from the date of effectiveness and actual project implementation time from the date of effectiveness is expected to be <0.75.

d.) Implementation progress (IP)

The IP rating will be derived from the IPR that shall be updated in tandem with the PCR preparation. The IP rating takes into account all applicable IP criteria assessed under each of the three main categories: i) compliance with covenants (project covenants, environmental and social safeguards and audit compliance), ii) project systems and procedures (procurement, financial management and monitoring and evaluation), and iii) project execution and financing (disbursement, budget commitments, counterpart funding and co-financing). The simple arithmetic average of the individual ratings is calculated to derive the final rating. The overall IP rating is provided as follows:

6 - Highly Satisfactory: The average rating of applicable IP criteria ratings is comprised between 3.5 and 4. The implementation processes have for the most part been highly satisfactory and has to lead to the anticipated results.

5 - Highly Satisfactory: The average rating of applicable IP criteria ratings is comprised between 3 and 3.49. The implementation processes have for the most part been highly satisfactory and has to lead to the anticipated results.

4 - Satisfactory: The average rating of applicable IP criteria ratings is comprised between 2.5 and 2.95. The implementation processes has for the most part been satisfactory and has for the most part lead to the anticipated results.

3 - Satisfactory: The average rating of applicable IP criteria ratings is comprised between 2 and 2.49. The implementation processes has for the most part been satisfactory and has for the most part lead to the anticipated results.

2 - Unsatisfactory: The average rating of applicable IP criteria ratings is comprised between 1.5 and 1.95. Several dimensions of implementation processes have not been satisfactory which has jeopardized the achievement of some project results.

1 - Highly Unsatisfactory: The average rating of applicable IP criteria ratings is comprised between 1.0 and 1.49. Most dimensions of implementation processes have not been satisfactory which has jeopardized the achievement of project results.

4. SUSTAINABILITY

The assessment of sustainability considers the extent to which the project has addressed risks during implementation and put in place mechanisms to ensure the continued flow of benefits after completion. It should also evaluate risks to the sustainability of development outcomes and/or the project’s benefits, including the resilience to exogenous factors. The overall rating of the sustainability outcome is the mean of the rating of the following four criteria: i) technical viability; ii) financial and economic sustainability, iii) institutional sustainability and strengthening of capacities, iv) Political and governance environment, (v) ownership and sustainability of partnerships vi) environmental and social sustainability and (vii) Resilience to exogenous factors and risk management.

103

a.) Technical Soundness

The criterion assesses the extent to which the project achievements rely on sound technology using inputs efficiently and providing productivity gains. It includes operation and maintenance (O&M) facilitation, availability of recurrent funding, spare parts, workshop facilities etc.)

6 – Highly Satisfactory: There is a low likelihood that the achievement of the project results is adversely affected by factors related to the technical design of the project. Reasons for this could include the following: the operation is technically simple; it was informed by extensive analytical work; the client and the Bank have extensive experience with similar projects or programs; and its economic benefits depend almost entirely on operation specific factors that can be effectively addressed in the operational design.

5 – Satisfactory: There is a likelihood that factors related to the technical design of the program or project may adversely impact the achievement of the project results. Such factors could include the following: the project is technically moderately complex; it was informed by adequate analytical work; it has a small number of components and sub-components; the client or the Bank has some experience with similar operations; and the technologies and processes used in the design have been successfully used elsewhere. The operation’s economic benefits depend primarily on factors that can be adequately addressed in the design.

4 – Moderately Satisfactory: There is a moderate likelihood that factors related to the technical design of the program or project may adversely impact the achievement of the project results. Such factors could include the following: the project is technically moderately complex; it was informed by adequate analytical work; it has a small number of components and sub-components; the client or the Bank has some experience with similar operations; and the technologies and processes used in the design have been to some extent successfully used elsewhere. The operation’s economic benefits depend primarily on factors that can be adequately addressed in the design.

3 – Moderately Unsatisfactory: There is a moderately substantial likelihood that factors related to the technical design of the program or project may adversely impact the achievement of the project results. Such factors could include the following: the project is technically complex; it was informed by limited analytical work; it has several components and subcomponents; the client or the Bank has limited experience with similar operations; and the design incorporates or relies on relatively new technologies and processes, which do not yet have a sufficient record. It may also be the case that r the project’s economic benefits to some extent depend on external factors that cannot be controlled through the operational design.

2 – Unsatisfactory: There is a substantial likelihood that factors related to the technical design of the program or project may adversely impact the achievement of the project results. Such factors could include the following: the project is technically complex; it was informed by limited analytical work; it has several components and subcomponents; the client or the Bank has limited experience with similar operations; and the design incorporates or relies on relatively new technologies and processes, which do not yet have a track record. It may also be the case that r the project’s economic benefits significantly depend on external factors that cannot be controlled through the operational design.

1 – Highly Unsatisfactory: There is a high likelihood that factors related to the technical design of the program or project may severely impact the achievement of the project results. Such factors could include the following: the project is of high technical complexity; it was not informed by strong analytical work; it has a large number of components and subcomponents; the client or the Bank has no experience designing similar operations; the design incorporates or relies on untested or unfamiliar technologies and processes; and making adjustments to the operation’s design would be very difficult and costly. It may also be the case that the project’s economic benefits are largely dependent on factors project activities may not be realistic or cannot be properly calibrated.

b.) Financial and Economic Viability

This criterion assesses the extent to which funding mechanisms and modalities (e.g. tariffs, user fees, maintenance fees, budgetary allocations, other stakeholder contributions, aid flows, etc.) have been put in place to ensure the continued flow of benefits after project completion, with particular emphasis on financial sustainability. For PBOs the assessment should focus on the financial

104

sustainability of the reforms, as well as the Bank’s policy dialogue to promote financial sustainability of the reforms. The rating is essentially qualitative using the following scale:

6 – Highly Satisfactory: The project has put in place robust mechanisms for economic and financial sustainability that are very likely to ensure the continued flow of benefits associated with the project after completion.

5 – Satisfactory: The project has put in place sufficient mechanisms for economic and financial sustainability that are deemed sufficient to ensure the continued flow of benefits associated with the project after completion.

4 – Moderately Satisfactory: The project has put in place moderate mechanisms for economic and financial sustainability that are deemed sufficient to ensure the continued flow of benefits associated with the project after completion.

3 – Moderately Unsatisfactory: The project has put in place few mechanisms for economic and financial sustainability, but they are not expected to be sufficient to ensure the continued flow of benefits associated with the project after completion.

2 – Unsatisfactory: The project has put in place very few mechanisms for economic and financial sustainability, but they are not expected to be sufficient to ensure the continued flow of benefits associated with the project after completion

1 – Highly Unsatisfactory: The project has not put in place any mechanisms for economic and financial sustainability, and the flow of benefits associated with the project are not expected to continue after completion.

c.) Institutional sustainability and strengthening of capacities

The criterion assesses the extent to which the project has contributed to strengthen institutional capacities - including for example through the use of country systems - that will facilitate the continued flow of benefits associated with the project. An appreciation should be made with regards to whether or not improved governance practices or improved skills, procedures, incentives, structures, or institutional mechanisms came into effect as a result of the operation. For PBOs this should include an assessment on the contributions made to building the capacity to lead and manage the policy reform process; as well as the extent to which the political economy of decision making was conducive to reform, the Government’s commitment to reform and how the design reinforced national ownership. The rating is based on the following scale:

6 – Highly Satisfactory: The project was critical in building or strengthening institutional capacities in the concerned sector / area of intervention. Country systems and capacities are excellent and sufficient to ensure the continued flow of benefits associated with the project after completion.

5 – Satisfactory: The project significantly contributed to strengthening institutional capacities in the concerned sector / area of intervention. Country systems and capacities are very good and deemed sufficient to ensure the continued flow of benefits associated with the project after completion.

4 – Moderately Satisfactory: The project moderately contributed to strengthening institutional capacities in the concerned sector / area of intervention. Country systems and capacities are good and deemed sufficient to ensure the continued flow of benefits associated with the project after completion.

3 – Moderately Unsatisfactory: The project marginally contributed to strengthening institutional capacities in the concerned sector / area of intervention and/or parallel systems had to be used. Country systems and capacities remain moderately weak and are deemed insufficient to ensure the continued flow of benefits associated with the project after completion.

2 – Unsatisfactory: The project very marginally contributed to strengthening institutional capacities in the concerned sector / area of intervention and/or parallel systems had to be used. Country systems and capacities remain weak and are deemed insufficient to ensure the continued flow of benefits associated with the project after completion.

105

1 – Highly Unsatisfactory: The project did not contribute to strengthening institutional capacities in the concerned sector / area of intervention and or parallel systems had to be used intensively. Country systems and capacities are very weak and not able to ensure the continued flow of benefits associated with the project after completion.

d.) Political and governance environment

This criteria assesses the extent political and governance developments that could impact the government’s priorities with respect to the project. This includes (but is not limited to) upcoming elections or an impending change in government; and other factors that could impact the political commitment to the operation or operational engagement and the political decisions required for sustainability of project results (including laws and the provision of counterpart financing). Special attention should be paid to fraud, corruption and other unethical practices resulting from governance failures.

6 – Highly Satisfactory: There is a low likelihood that political factors could adversely impact the project results. The political and governance situation does not represent a risk to the project results thanks to political stability, consensus on development priorities, a strong anti-corruption and ethics environment and high levels of transparency, accountability and participation. All relevant political decisions (including approval of laws and regulations) have been taken and cannot be reversed easily.

5 – Satisfactory: There is a likelihood that political and governance factors could adversely impact the project results. At the program level, the political context is relatively stable and not likely to significantly affect the project results. The government has a clear set of development priorities, which are generally supported across the political spectrum and are consistent with the program. Adequate anti-corruption and public sector ethics regulations exist and are generally enforced.

4 – Moderately Satisfactory: There is a moderate likelihood that political and governance factors could adversely impact the project results. The political context is relatively stable and not likely to significantly affect the project results. The government has a clear set of development priorities, which are generally supported across the political spectrum and are consistent with the program. Adequate anti-corruption and public sector ethics regulations exist and are to some extent enforced.

3 – Moderately Unsatisfactory: There is a substantial likelihood that political and governance factors could significantly impact the project results achievement. The project results could be impacted by significant political uncertainty or transition. This may include post-conflict countries that have achieved some level of political stability; or countries that enjoy a period of relative stability but have a history of endemic political upheaval with negative effects on the operational engagement. Likewise, the government has taken initial steps to improve transparency, accountability and participation, but with limited impact.

2 – Unsatisfactory: There is a major likelihood that political and governance factors could significantly impact the project results achievement. The project results could be impacted by significant political uncertainty or transition. This may include post-conflict countries that have achieved some level of political stability; or countries that enjoy a period of relative stability but have a history of endemic political upheaval with negative effects on the operational engagement. Likewise, the government has taken initial steps to improve transparency, accountability and participation, but with limited impact.

1 – Highly Unsatisfactory: There is a high likelihood that political and governance factors could severely impact the project results. The project results could be derailed by a high degree of political instability, fragility, uncertainty or transition. The country may be undergoing conflict or may have recently emerged from conflict, and the political context is fragile. The government’s development priorities are unclear. Anti-corruption and public sector ethics regulations do not exist or are not enforced.

e.) Ownership and sustainability of partnerships

The assessment determines whether the project has effectively involved relevant stakeholders, promoted a sense of ownership amongst the beneficiaries (both men and women) and put in place effective partnerships with relevant stakeholders (e.g. local authorities, civil society organizations,

106

private sector, donors) as required for the continued maintenance of the project outputs. For PBOs, the assessment should include the extent to which the Government conducted extensive consultations during the preparation and implementation of the PRSP and the extent to which the Bank supported the Government in deepening the consultation processes. The rating is largely qualitative using the following scale:

6 – Highly Satisfactory: The project has been highly effective at involving all the relevant stakeholders and there is a strong sense of ownership amongst the beneficiaries. Effective partnerships with relevant stakeholders (e.g. local authorities, civil society organizations, private sector) have been put in place to ensure the continued maintenance and management of project outputs.

5 – Satisfactory: The project has been effective at involving most stakeholders and promoting a sense of ownership amongst the beneficiaries. Partnerships with relevant stakeholders have been put in place and are deemed sufficient to ensure the continued maintenance and management of project outputs.

4 – Moderately Satisfactory: The project has been to a moderate extent effective at involving some stakeholders and promoting a sense of ownership amongst the beneficiaries. Partnerships with relevant stakeholders have been put in place and are deemed somewhat sufficient to ensure the continued maintenance and management of project outputs.

3 – Moderately Unsatisfactory: The project has involved only a small number of stakeholders and there is limited ownership amongst the beneficiaries. No or marginally effective partnerships with relevant stakeholders have been put in place and are not considered sufficient to ensure the continued maintenance and management of project outputs.

2 – Unsatisfactory: The project has involved only a small number of stakeholders and there is limited ownership amongst the beneficiaries. No or marginally effective partnerships with relevant stakeholders have been put in place and are not considered sufficient to ensure the continued maintenance and management of project outputs.

1 – Highly Unsatisfactory: The project has not been effective at involving the relevant stakeholders and there is no sense of ownership amongst the beneficiaries. No partnerships with relevant stakeholders have been established to ensure the continued maintenance and management of project outputs.

f.) environmental and social sustainability

This criterion would normally only apply to Environmental Category I and II projects. It assesses the extent to which the environmental and social mitigation/enhancement measures of the project were implemented, the capacity of country institutions and systems and the availability of funding to ensure the environmental and social sustainability of the operation. The Environmental and Social Safeguards rating in the IPR should be used as guidance. The rating is largely qualitative using the following scale:

6 - Highly satisfactory. The ESMP has been implemented in a timely and satisfactory manner; institutional capacity is strong and there is sufficient funding to ensure the environmental and social sustainability of the operation.

5 – Satisfactory. The ESMP has largely been implemented in a timely and satisfactory manner; institutional capacity and funding are deemed sufficient to ensure the environmental and social sustainability of the operation.

4 – Moderately Satisfactory. The ESMP has largely been implemented in a timely and satisfactory manner; institutional capacity and funding are deemed moderately sufficient to ensure the environmental and social sustainability of the operation.

3 – Moderately Unsatisfactory. The ESMP has been implemented with major delays or somewhat in an unsatisfactory manner; institutional capacity and funding are deemed moderately insufficient to ensure the environmental and social sustainability of the operation.

107

2 – Unsatisfactory. The ESMP has been implemented with major delays or in an unsatisfactory manner; institutional capacity and funding are deemed insufficient to ensure the environmental and social sustainability of the operation.

1 – Highly unsatisfactory. The ESMP has not been implemented; institutional capacity and funding are not available to ensure the environmental and social sustainability of the operation.

g.) Resilience to exogenous factors and risk management

This criteria assesses the extent to which the achievements depend on exogenous factors, such as the terms of trade, the world market prices or the political situation in neighbouring countries.

6 – Highly Satisfactory: The project achievements did not depend on any exogenous factors or/and have low risks to achieving the intended results.

5 – Satisfactory: The project achievements very marginally depend on exogenous factors or/and have very marginal risks to achieving the intended results.

4 – Moderately Satisfactory: The project achievements marginally depend on exogenous factors or/and have marginal risks to achieving the intended results.

3 – Moderately Unsatisfactory: The project achievements to some extend depend on exogenous factors or/and have sufficient risks to achieving the intended results.

2 – Unsatisfactory: The project achievements substantially depend on exogenous factors on exogenous factors or/and have major risks to achieving the intended results.

1 – Highly Unsatisfactory: The project achievements highly depend on exogenous factors on exogenous factors or/and have very significant high risks to achieving the intended results.

108

Annex G – CEDR Synthesis – Country Template Guidance

CEDR Synthesis

Country template

Guidance Note

21 September 2015

109

1. INTRODUCTION

The Comprehensive Evaluation of the Bank’s Development Results (CEDR) is designed as a theory based evaluation with the objective of responding to three main questions:

1. What difference has the Bank made in Africa? What has the Bank achieved in Africa between 2004 and 2013?

2. How is the Bank adding value? How is the Bank maximizing its value as a development partner? 3. Is the Bank learning from what it is doing?

14 CSPEs commissioned in 2014/2015 by IDEV will serve as the main source of evidence for the final CEDR synthesis, each constituting a case study. They will be complemented by project level results assessments. The 14 CSPEs are conducted using a common set of questions listed below.

Table 1: CSPE common evaluation questions

Relevance

EQ1 - To what extent are the country strategy and Bank operations aligned with: (a) RMC development needs; (b) RMC development strategies and priorities; and (c) the needs of beneficiaries? EQ2 - To what extent are the interventions in the country aligned with the Bank’s strategy and priorities?

Effectiveness

EQ3 - To what extent have the Bank’s interventions achieved their expected results? EQ4 - To what extent have the Bank’s interventions benefited target group members? EQ5 - To what extent have the Bank’s interventions contributed to the achievement of development objectives and expected development results of the country, including impacts (both intended and unintended)?

Sustainability EQ6 - To what extent have achieved benefits continued or will be likely to continue once the Bank’s interventions are completed?

Cross-cutting issues

EQ7 - To what extent are the Bank’s interventions inclusive (i.e., bringing prosperity by expanding the economic base across the barriers of age, gender and geography) in terms of gender equality and regional disparity? EQ8 - To what extent are the Bank’s interventions environmentally sustainable and support the transition to green growth?

Lessons EQ9 - What are the key factors positively and negatively influencing the achievement of sustainable development results?

Efficiency

EQ10 - To what extent are the Bank’s interventions delivered in an efficient manner (i.e., whether resources and inputs are economically converted to results)? EQ11 - To what extent are the Bank’s interventions implemented in a timely manner and in compliance with operational standards?

Design

EQ12 - To what extent is the quality of the CSP satisfactory? EQ13 - To what extent has the Bank applied selectivity in designing its country portfolio and focused on areas where it brings added value? EQ14 - To what extent has the Bank been innovative in adapting its approach to the country’s context and development challenges/needs? EQ15 - To what extent are the Bank’s interventions coherent and well-coordinated internally?

Knowledge and policy advice

EQ16 - To what extent has the Bank actively engaged in and influenced policy dialogue through relevant advice? EQ17 - To what extent has the Bank delivered adequate analytical work in support of its interventions, positioning and policy advice?

Partnerships and leverage

EQ18 - To what extent are the Bank’s interventions harmonized with those of other donors (avoiding duplication, simplifying procedures etc.)? EQ19 - To what extent are the Bank’s interventions and resources bringing in other players and being leveraged for maximizing development effectiveness at country level?

110

Managing for results

EQ20 - To what extent has the Bank successfully implemented management systems that focus on results and allow learning from past experience? EQ21 - To what has the Bank supported the development of national capacities and management systems that focus on results?

To respond the CEDR questions 2 and 3, a theory of action has been elaborated (see Figure 1 below). It has identified a set of mechanisms / contextual factors for which evidence should be collected consistently across the 14 case studies so that cross case analysis can be conducted. The purpose of the country template is to provide the basis for consistent data collection and analysis. This guidance note details the requirements for filling the country template.

The list of mechanisms / country conditions analysed through the country template is the following:

Mechanisms

M1 – Knowledge and strategic advice: the Bank provides the country with relevant knowledge and appropriate advice, and participates effectively to policy dialogue;

M2 - Adapted solutions: the Bank’s strategy at country level is well designed, well integrated and proposes the right solutions to challenges;

M3 – Strategic focus: the Bank’s strategy is focused on areas of comparative advantage based on thorough analysis;

M4 - Leverage: the Bank’s strategy and program are designed so as to crowd in additional resources and with attention to scaling up;

M5 – Supervision: the Bank’s program is monitored effectively with a focus on ensuring the achievement of expected results;

M6 – Project focus: the design and implementation of Bank’s projects are focused on contributing to CSP outcomes;

M7 – Project design: the design and implementation of Bank’s projects address constraints to contributing to CSP outcomes;

M8 – Managing for results & Learning: the Bank's interventions are designed and managed for development results at country level, and informed by evidence of what works and doesn't work;

M9 – Partnership and coordination: The Bank’s intervention are designed and implemented in partnership and coordination with the government and other development partners in the spirit of international principles of engagement.

Country conditions

C1 – Country ownership: the country has included the Bank’s CSP objectives as contributing results in national plans and monitors the achievement of CSP objectives:

C2 – Country readiness to reform: The country has taken action to overcome policy and regulatory issues identified as constraints to achieving CSP objectives.

111

Figure 1: Theory of action for the Bank

Partnership and coordination

The Bank’s intervention are designed and implemented in partnership and coordination with the government and other development partners in the spirit of internal principles of engagement.

Managing for results and learning

The Bank's interventions are managed for development results at country level.

The design and management of Bank interventions at country level are informed by evidence of what works and doesn't work

Achievement of country

level results

Supervision

The Bank’s program is monitored effectively

with a focus on ensuring the achievement of

expected results.

Adapted solutions

The Bank’s strategy at country level is well designed, and proposes the right solutions to challenges.

Project design

The design of Bank’s projects address constraints to contributing to CSP outcomes.

Project focus

The design and implementation of Bank’s projects are focused on contributing to CSP outcomes.

Leverage

The Bank’s strategy and program are designed so as to crowd in additional resources and with attention to scaling up.

Strategic focus

The Bank’s strategy is focused on areas of comparative advantage based on thorough analysis and well integrated.

Knowledge & Strategic

advice

The Bank provides the country with relevant knowledge and appropriate advice, and participates effectively to policy dialogue

Country readiness for reforms

The country has taken action to overcome policy and regulatory issues identified as constraints to achieving CSP objectives.

Country ownership

The country has included the Bank’s CSP objectives as contributing results in national plans and monitors the achievement of CSP objectives.

Evidence linked to CSPE questions

Evidence linked to portfolio review

Evidence linked to context analysis

112

As indicated in the figure below, most of these mechanisms should be assessed at country level while a few others will be assessed based in portfolio analysis.

The guidance below proposes a standard set of judgment criteria to guide the rating of each mechanism, based on the initial experience gathered through ongoing CSPEs. It maps each mechanism to CSPE question when relevant. It is to be noted however this mapping is indicative and can vary according the exact evaluation matrix used by the CSPE team. The final mapping should be done on the basis of the judgement criteria and indicators of the evaluation matrix using this general mapping as guidance. For example, the effectiveness of supervision can be defined both based on a judgement criteria under EQ11 as part of compliance with operational standards and/or a judgement criteria under EQ20 as part of management systems focusing on results.

Two mechanisms will be assessed based on project level assessments using a bottom up approach as further explained in the guidance below: M6 - Project focus, and M7 - Project design.

Finally, the two country conditions should be included in the analysis of explanatory factors for the achievement of results (EQ9).

113

M1 – Knowledge and strategic advice: the Bank provides the country with relevant knowledge and appropriate advice, and participates effectively to policy dialogue

Related CSPE questions EQ16 - To what extent has the Bank actively engaged in and influenced policy dialogue through relevant advice? EQ17 - To what extent has the Bank delivered adequate analytical work in support of its interventions, positioning and policy advice?

Hypothesis and judgment criteria The assumption is that Bank's advice and knowledge products are sound and suited to the development needs of RMC governments and that the Bank's engagement with the government allows engagement in sensitive policy areas. In situations where it is clear that there are significant policy/regulatory failures, but a lack of sufficient will within Government to address these, it should be expected that the Bank either withdraw or have a defensible strategy for how it will engage in attempting to work with partners to open up policy debate. A defensible strategy is likely to include high-quality and timely diagnostics, policy dialogue, and technical assistance aimed at identifying entry points and laying the groundwork for greater influence and effective support. Examples of judgment criteria / indicators that can be found in the CSPE evaluation matrix rate this mechanism include:

Existence of an explicit commitment in the Bank’s strategy documents for its involvement in policy dialogue;

Existence and clarity of objectives set for policy dialogue;

Existence and clarity of the definition of the Bank’s comparative advantage in policy dialogue areas where the Banks intends to assume leadership;

Number of policy dialogue working groups (technical and strategic) and events participated by the Bank, and level of attendance to meetings

Evidence of Government follow-up to Bank’s identified institutional/policy needs

Extent and quality of sector and thematic (e.g. poverty) analysis

Degree of awareness of relevant ESW

Evidence of the use of Bank’s analytical work in policy-making process

Perceived effects of ESW and analytical work in the positioning of the Bank in the dialogue system

Extent of projects making explicit reference to previous analytical work

Degree of satisfaction of partners with the Bank’s contribution to policy dialogue and to knowledge work and appreciation of its influence of (i) the formulation of development needs; (ii) the definition of national strategies, policies and plans; (iii) influence on the implementation of these strategies and plans; and (iv) influence on the sustainability of results.

Rating scale 6 – Highly Satisfactory: The Bank has delivered significant knowledge work in all sectors selected for its interventions that has been recognized by all other development partners including the government, allowed the Bank for positioning itself as leader in policy dialogue in these areas, and has triggered change in the policy and/regulatory framework of the country.

5 –Satisfactory: The Bank has delivered significant knowledge work in most sectors selected for its interventions that has been recognized by all other development partners including the government, allowed the Bank for positioning itself as leader in policy dialogue in these areas, and has triggered change in the policy and/regulatory framework of the country.

4 – Moderately Satisfactory: The Bank has delivered knowledge work in few sectors selected for its interventions that has been recognized by all other development partners including the government, and allowed the Bank to contribute to policy dialogue in these areas.

3 – Moderately Unsatisfactory: The Bank has delivered limited knowledge work in a few sector selected for its interventions that has been recognized by all other development partners including the government, and is

114

contributing to policy dialogue, but with no linkage between the two aspects.

2 – Unsatisfactory: The Bank has delivered minimal knowledge work in a few sectors selected for its interventions with no recognition by / influence on other development partners and government. The Bank is routinely involved in policy dialogue.

1 – Highly Unsatisfactory: The Bank has not delivered any significant piece of knowledge work. It is seen as one actor among many in the development landscape of the country by most stakeholders.

M2 - Adapted solutions: the Bank’s strategy at country level is well designed, and proposes the right solutions to challenges

Related CSPE questions EQ12 - To what extent is the quality of the CSP satisfactory? EQ14 - To what extent has the Bank been innovative in adapting its approach to the country’s context and development challenges/needs?

Hypothesis and judgement criteria The assumption is that using its expertise, knowledge and capacity for innovation, the Bank is able to customize the best solutions for development challenges at country level. This mechanism should be distinguished from the standard relevance criteria. It does not focus as such on the objectives of the strategy as meeting needs but rather on all other design aspects that should ensure that all conditions are in place so that the strategy objectives are indeed met and, assuming relevance, contribute to meeting development challenges. A proxy used as assessment criteria for this mechanism is the quality at entry of the Bank strategies, including for example clarity of the objectives and intervention logic based on thorough analysis, consultation and participation to ensure ownership, mix of instruments and products to respond appropriately to needs, choice of implementation modalities to maximize chances of success and mitigate capacity issues, and risk identification and mitigation including safeguards. A second element is taken into account, which is the capacity of the Bank to innovate and adapt to different and evolving country situations. It is to be noted finally that the one element of the quality at entry, selectivity and strategic focus, is considered as a full separate mechanism as it is explicitly mentioned as such in Bank strategies. Examples of judgment criteria / indicators that can be included in the CSPE evaluation matrix rate this mechanism include:

Existence of a robust analysis of the country’s development context

Clarity and consistency of CSP intervention logic (outcomes, outputs, indicators, risks and assumptions)

Consistency of delivery model with country’s specific development framework, readiness and needs

Ownership of the national stakeholders of the design and provision of mechanisms for participation in implementation

Analysis of capacity for implementation and proposal of solution to address identified issues

Identification of risks and mitigation actions

Degree of integration of lessons from ESW, policy dialogue, and previous evaluations in the CSP

Distribution of operations by type and instruments overtime

Appropriateness of the TA accompanying interventions

Share of Bank’s initiatives using innovative instruments

Share of Bank’s initiatives addressing new, emerging themes

Rating scale 6 – Highly Satisfactory: The Bank’s strategy is based on a compelling intervention logic demonstrating a thorough understanding of the country’s evolving context, proposing solutions fully adapted to this context in all areas of intervention, and showing substantive innovation (analysis and product mix) over time in responding to

115

challenges to achieving results.

5 –Satisfactory: The Bank’s strategy is based on a sound intervention logic demonstrating a thorough understanding of the country’s evolving context, proposing solutions variably adapted to this context depending on the area of intervention, and showing some innovation (analysis and product mix) over time in responding to challenges to achieving results.

4 – Moderately Satisfactory: The Bank’s strategy is based on a sound intervention logic demonstrating a good understanding of the country’s evolving context, proposing solutions variably adapted to this context depending on the area of intervention, and not showing much innovation (analysis and product mix) over time in responding to challenges to achieving results.

3 – Moderately Unsatisfactory: The Bank’s strategy is based on an intervention logic showing partial understanding of the country’s evolving context, proposing solutions variably adapted to this context depending on the area of intervention, and not showing much innovation (analysis and product mix) over time in responding to challenges to achieving results.

2 – Unsatisfactory: The Bank’s strategy is based on an unclear intervention logic showing low understanding of the country’s evolving context, proposing “business as usual” solutions variably adapted to this context depending on the area of intervention.

1 – Highly Unsatisfactory: The Bank’s strategy is disconnected from the evolving context of the country and used as justification for the ongoing portfolio of operations that is designed and implemented as a compilation of projects agreed with direct national counterparts in the area covered.

M3 - Strategic focus: the Bank’s strategy is focused on areas of comparative advantage, and well integrated.

Related CSPE questions EQ13 - To what extent has the Bank applied selectivity in designing its country portfolio and focused on areas where it brings added value? EQ15 - To what extent are the Bank’s interventions coherent and well-coordinated internally?

Hypothesis and judgement criteria The assumptions underlying strategic focus are: (i) there is a high degree of coherence and consistency from the analytic diagnosis to the selection of areas ((sub-)sectors/thematic areas) for Bank assistance and the articulation of support strategies in and between these areas, to the identification of a suitable assistance program, and the expected results of the latter; (ii) the logic, or the storyline of the CSP, is clear, compelling and realistic; and (iii) the assistance program flows from the diagnosis and support strategies, and should not be ‘imposed’ on the CSP from the start. Strategic focus assumes that the program needs to be focused to address one or more of the following reasons that are often identified as leading to poor performance - (i) weak government capacity and commitment to reform in the considered sector, (ii) lack of appropriateness of the instruments used (for instance, involvement in budget support, SWAP, etc.), (iii) poor synergies between operations, including lending and non-lending operations, (iv) doing too little to make a significant difference or doing something that the Bank doesn't really have a comparative advantage in, and (v) no scaling up of successful initiatives.. It is important to bear in mind that a common response by operational staff in many agencies to the need to improve selectivity is to reduce the number of objectives and resort to “relabelling” or “regrouping” the objectives without a corresponding streamlining of the supporting programs. Such approaches are not a credible approach to selectivity, as they don't respond to the under-lying problem that selectivity is seeking to address. Transforming an indiscriminate program into a selective one is not easy and may take longer than expected to implement, which is especially true where the circumstances encourage expanding engagement and where development needs are overwhelming. In the same vein, support that is organized under broad ‘pillars’ is unlikely to be selective. In such cases, the main rationale for this type of approach is probably an intention to cover the entire scope of the Bank portfolio or pipeline in a given country, whereas the strategic focus of a CSP should drive the spectrum of Bank assistance, but not vice versa. It is actually often the case for Bank interventions to be organized under broad pillars just by the construction of the CSP so the analysis has to be taken at the lower level. Overall, analysis

116

should therefore start from what the Bank strategy says, and if the strategy indicates that the program has been shaped by the needs of selectivity, assess the degree to which the strategy has actually been implemented and whether it has addressed the under-lying reasons that selectivity was intended to address. Examples of judgment criteria / indicators that can be included in the CSPE evaluation matrix rate this mechanism include:

Sector and themes presenting constraints and challenges to the development of the country are identified, analysed and taken into account in the Bank strategies

Share of the Bank’s interventions clearly identifying the target population (poor, youth, etc.)

Share of the Bank’s interventions with an articulated analysis of needs of the target population

Perceived ‘gaps’ in the coverage of strategic areas within the Bank’s competence

Perceived excessive concentration of assistance in individual areas

Degree of consistency between identified needs of target groups and activities

Degree of harmonization/complementarity of CSP with other DPs country strategies

Use of donors mapping at sector level to support Bank’s selection of areas of interventions

Degree of sectoral concentration

Number of interventions with explicit linkages and synergies (cross-sector and cross-instrument)

Number of interventions with geographical linkages and synergies

Number and size of projects in areas where the Bank is perceived bringing significant added value

Rating scale 6 – Highly Satisfactory: The Bank’s strategy presents a compelling analysis of the respective positioning of development partners, areas of comparative advantage and matching of this analysis with the evolving context and challenges of the country to define priority areas of assistance for the Bank and the interconnections between those from a programmatic perspective.

5 –Satisfactory: The Bank’s strategy presents a clear analysis of the respective positioning of development partners, areas of comparative advantage and matching of this analysis with the evolving context and challenges of the country to define priority areas of assistance for the Bank but does not fully articulate interconnections and integration constraints in the program.

4 – Moderately Satisfactory: The Bank’s strategy presents an analysis of the respective positioning of development partners and areas of comparative advantage but the analysis does not fully show how this translates into priority areas of assistance for the Bank matching the evolving context and challenges of the country.

3 – Moderately Unsatisfactory: The Bank’s strategy proposes an analysis of positioning and comparative advantage. Priority areas of assistance aligned with needs are proposed without relating them to this analysis.

2 – Unsatisfactory: The Bank’s strategy proposes a basic analysis of positioning and comparative advantage but fails to articulate clearly the result in terms of areas of intervention which in practice overlap with areas of intervention of the ongoing portfolio.

1 – Highly Unsatisfactory: The Bank’s strategy replicates areas of intervention of the ongoing portfolio without any convincing analysis of positioning.

M4 - Leverage: the Bank’s strategy and program are designed so as to crowd in additional resources and with attention to scaling up

Related CSPE questions EQ19 - To what extent are the Bank’s interventions and resources bringing in other players and being leveraged for maximizing development effectiveness at country level?

Hypothesis and judgement criteria Given the small size of Bank resources relative to the economies it seeks to influence, the effectiveness of Bank interventions in helping clients increasingly depends on how it uses instruments as pilots and as catalysts to

117

leverage resources from development partners and other stakeholders. The assumption is that the Bank's brand name and convening power give the Bank a comparative advantage in this area. Strengthening the complementarity among diagnostic work, technical assistance, and lending instruments, and among policy and investment lending instruments, can help to scale up efforts and achieve more sustainable, long-term impact. This requires an important change in the mind-set, with a much stronger and explicit emphasis on scaling up: projects should be viewed as opportunities to crowd-in resources from the public and private sectors, as well as from other development partners, not simply in terms of circumscribed interventions. The impact of the Bank should be catalytic and beyond the individual intervention. This mind-set needs to be built into dialogue, planning, design, and the intelligent use of pilots and sequential planning, so the Bank can inspire action and induce policy change. Examples of judgment criteria / indicators that can be included in the CSPE evaluation matrix rate this mechanism include:

Amount of resources catalysed by Bank’s projects

Share of the Bank’s initiatives co-financed by other DPs

Influence of the Bank on the engagement of other development partners

Extent to which the use of different instruments and aid modalities by the Bank catalysed additional financing

Rating scale 6 – Highly Satisfactory: The Bank’s has demonstrated an instrumental role in bringing in additional financing in the country through dialogue and usage of relevant instruments, with a leveraging effect of at least 1 to 5 overall. The program design shows explicit and consistent attention to scaling up both at strategic and project level.

5 –Satisfactory: The Bank’s has demonstrated a significant role in bringing in additional financing in the country through dialogue and usage of relevant instruments with a leveraging effect of at least 1 to 3 overall. Most key projects show explicit attention to scaling up.

4 – Moderately Satisfactory: The Bank’s has had an opportunistic but active role in bringing in additional financing in the country through dialogue and usage of relevant instruments. A few projects show a leverage effect and explicit attention to scaling up.

3 – Moderately Unsatisfactory: Leverage is taken as an ad-hoc consideration at project level without specific attention given at the strategic level. No specific consideration is given to scaling up.

2 – Unsatisfactory: The Bank mentions leverage as an objectives and articulates some action but with no visible effect.

1 – Highly Unsatisfactory: The Bank had no leverage effect on financing in the country and does not articulate any action for that purpose.

M5 – Supervision: the Bank’s program is monitored effectively with a focus on ensuring the achievement of expected results

Related CSPE questions EQ11 - To what extent are the Bank’s interventions implemented in a timely manner and in compliance with operational standards? EQ20 - To what extent has the Bank successfully implemented management systems that focus on results and allow learning from past experience?

Hypothesis and judgement criteria The Bank assumes that its recent delegation of authority policy and movement of staff from HQ to country offices will have results in more authority to the country level and opportunities for more direct monitoring and to address concerns raised by member countries. As such, the under-lying assumption of the Bank has been that it has been the lack of appropriate levels of authority that has constrained the ability of Bank staff to identify and

118

implement effective solutions. The assessment goes obviously beyond a simple good practice of supervision. The assumption behind effective supervision is that issues are identified and acted upon with the result of keeping the portfolio healthy and projects moving. A regular portfolio level meetings examining a global list of issues and deciding / monitoring the implementation of corrective action across the portfolio would be a possible materialization of effective supervision. Effective supervision should also include monitoring of risks, which are all factors that may prevent moving from one level to another of the results chain during implementation as articulated accordingly in the RLF. Specific risks related to an operation – of a social, environmental, political, and institutional or other relevant nature - should be identified during project / program preparations and related mitigation measures should be incorporated into the design, usually on the basis of lessons learnt, and monitored throughout the lifecycle. The analysis should focus on those risks deemed critical and significant, as assessed with respect to their likelihood of occurrence and potential impact on the operation. Examples of judgment criteria / indicators that can be included in the CSPE evaluation matrix rate this mechanism include:

Time to first disbursement

Time to procure goods and services

Overall disbursement ratio

Share of the Bank’s initiatives that adhered to the original closing date (and extent of delay)

Share of the Bank’s initiatives that adhered to the original budget (and extent of slippage)

Average project duration

Satisfaction of stakeholders with the Bank’s timeliness

Share of projects’ supervision and completion report compliant with standards

Frequency of project’s supervision

Effectiveness of supervision report in clarifying the progress in results achievement, potential issues and countermeasures envisaged

Delays in publishing supervision reports after the field mission

Share of projects that underwent mid-term review

Share of Bank’s interventions with timely completed PCR

Share of PCR positively assessed and confirmation of ratings in PCRN

Extent of rapidity of follow-up on identified issues

Operations at risk

Number of projects restructured, and variation in performance before/after restructuring

Share of operations task-managed from field office

Degree of support extended by the country team to Executing Agencies to ensure smooth implementation of the Bank’s interventions

Satisfaction with the proactivity in dealing with problematic/slow moving projects

Extent to which the Bank’s is considered a preferred partner for development assistance

Rating scale 6 – Highly Satisfactory: The Bank’s has established a robust monitoring system including timely and quality supervisions, systematic follow up on actions, and review processes at all levels (operation to sector to country) including issues and risks and involving all stakeholders. The effect on the quality of the portfolio can be demonstrated.

5 –Satisfactory: The Bank’s has established a monitoring system including timely and quality supervisions, and systematic follow up on actions. Review processes include issues and risks and involve all stakeholders, but are not systematic at all levels (operations to sector to country). The effect on the quality of the portfolio can be demonstrated.

4 – Moderately Satisfactory: The Bank’s has established a monitoring system including timely and quality supervisions, and follow up on actions at the project level, with no overall review processes. It is difficult to clearly demonstrate the effect on the quality of the portfolio.

3 – Moderately Unsatisfactory: The Bank’s has established a monitoring system including timely supervisions, but the quality of supervision reports is variable as well as follow up on actions. It is difficult to clearly demonstrate the effect on the quality of the portfolio.

119

2 – Unsatisfactory: The Bank’s has established a monitoring system including timely supervisions, but the quality of supervision reports is poor and follow up on actions is not clear. There is no evidence that it is having any sort of effect on the quality of the portfolio.

1 – Highly Unsatisfactory: The supervision process is managed on an ad-hoc basis to comply with institutional requirements and there is no evidence that it is having any sort of effect on the quality of the portfolio.

M6 – Project focus: the design and implementation of Bank’s projects are focused on contributing to CSP outcomes

Related CSPE questions N/A

Hypothesis and judgement criteria The assumption is that, based on the CSP in place at the time, projects are designed and managed in order to ensure the contribution of the Bank to outcomes as defined in the CSP. The project should show a clear intervention logic making explicit the conditions of that contribution. From a technical perspective, the project design would have explored different options to achieve the intended outputs and ensure the contribution to the intended outcomes and would have selected the options that provide the higher likelihood of success. During implementation, the choices made would be reviewed and adapted as needed to ensure the conditions to contributing to outcomes remain valid, and the set of outputs technically sound to ensure that contribution. In particular, adequate attention to technical issues in the design and implementation of projects included adequate initial analysis of critical technical aspects, such as procurement, safeguards, economic analysis, fiduciary arrangements, and other project or sector-specific technicalities. The assessment should be conducted bottom up starting from the review of the portfolio including design and monitoring documents. It should be based on criteria 1a of the project results assessment (PRA), relevance of project objectives. The evidence collected can be organized by sector depending on the breakdown of the work within the CSPE team.

Rating scale 6 – Highly Satisfactory: More than 90% of the projects examined in the country had a rating of highly satisfactory for criteria 1a. None had a highly unsatisfactory rating.

5 – Satisfactory: More than 80% of the projects examined in the country had a rating of satisfactory or higher for criteria 1a.

4 – Moderately Satisfactory: More than 60% of the projects examined in the country had a rating of satisfactory or higher for criteria 1a. Less than 30% had a rating of unsatisfactory or less.

3 – Moderately Unsatisfactory: Less than 60% of the projects examined in the country had a rating of satisfactory or higher for criteria 1a. More than 50% had a rating of moderately satisfactory or higher.

2 – Unsatisfactory: Less than 50% of the projects examined in the country had a rating of moderately satisfactory or higher for criteria 1a.

1 – Highly Unsatisfactory: Less than 30% of the projects examined in the country had a rating of moderately satisfactory or higher for criteria 1a.

M7 – Project design: the design of Bank’s projects address constraints to contributing to CSP outcomes


Hypothesis and judgement criteria The assumption is that the project design should include a thorough analysis of the context and factor in potential capacity issues and all other constraints to achieving objectives. These would then be reflected in

120

particular in the choice of the implementation arrangements and the choice of using national procurement and financial management systems for the implementation of the project. A typical case of failure of this mechanism is when across the portfolio, the lack of capacity of implementation partners is identified as the major reason for poor project performance in terms of the timely and cost-effective delivery of outputs. The assessment should be conducted bottom up starting from the review of the portfolio including design and monitoring documents. It should be based on criteria 1b of the project results assessment, relevance of project design. The evidence collected can be organized by sector depending on the breakdown of the work within the CSPE team.

Rating scale 6 – Highly Satisfactory: More than 90% of the projects examined in the country had a rating of highly satisfactory for criteria 1b. None had a highly unsatisfactory rating.

5 – Satisfactory: More than 80% of the projects examined in the country had a rating of satisfactory or higher for criteria 1b.

4 – Moderately Satisfactory: More than 60% of the projects examined in the country had a rating of satisfactory or higher for criteria 1b. Less than 30% had a rating of unsatisfactory or less.

3 – Moderately Unsatisfactory: Less than 60% of the projects examined in the country had a rating of satisfactory or higher for criteria 1b. More than 50% had a rating of moderately satisfactory or higher.

2 – Unsatisfactory: Less than 50% of the projects examined in the country had a rating of moderately satisfactory or higher for criteria 1b.

1 – Highly Unsatisfactory: Less than 30% of the projects examined in the country had a rating of moderately satisfactory or higher for criteria 1b.

M8 – Managing for results & Learning: the Bank's interventions are designed and managed for development results at country level and informed by evidence of what works and doesn't work.

Related CSPE questions EQ20 - To what extent has the Bank successfully implemented management systems that focus on results and allow learning from past experience?

Hypothesis and judgement criteria The assumption is that the Bank has systems in place that allow managing for development results. Together with the constraints and opportunities arising from country-specific or regional development contexts, lessons learned from past experience provide the main rationale for the CSP process. The purpose of the identification of lessons learned is to capture what the Bank has done well under previous strategies, what it has not done well and for what reasons, and to explain how these lessons will be integrated in the design of the new strategy. Staff has a good understanding of basic RBM concepts and is able to apply these concepts appropriately in the context of project / program development and implementation Bank management has a good understanding of basic RBM concepts and are able to use results information appropriately to inform decision-making and program / project management. Country offices and country partners have sufficient understanding, capacity and ownership to implement rigorous RBM suitable for project management. The Bank’s standardized Results-Based Logical Framework (RLF) is designed to capture the results chain and provide a clear description of the intervention logic. The soundness of the project/program design should be demonstrated by a clear articulation of the cause-effect relationship (results-chain) between activities, outputs, outcomes and impacts of the operation. For each component, the operation should present the key activities and their resulting outputs. For the RLF to be useful for RBM, it is assumed that RLFs are credible and realistic in terms of what the expected results might be, and what else is assumed to happen, if the intended outcomes are to be delivered. Available data is of sufficient quality to inform project / program management and assess progress

121

toward results. Examples of judgment criteria / indicators that can be included in the CSPE evaluation matrix rate this mechanism include:

Explicit reference to the requirement for managing for results in the main strategy documents of the Bank

Evidence of use of evaluative information for improving the Bank’s performance

Evidence of quality enhancement measures and function to improve the quality of intervention design

Training of staff (hours) on how to operationalize RBM, to assist them in preparing results-based plans and report on results achieved

Evidence of the application of results-based management (RBM) in the work of the country team

Satisfaction with internal systems and procedures for monitoring the achievement of strategic objectives, knowledge management and systematic learning, across the organizational matrix

Share of Bank’s interventions with clearly spelled out, valid and complete indicators of achievements

Share of Bank’s interventions with baseline data available/collected at the design stage

Share of Bank’s interventions with detailed monitoring data and plan agreed upon with Executing Agency before launch (and indicated in the PAR)

Degree of harmonization and coherence in the set of indicators used overtime to measure outcome and impact achievements (sectoral or overall level)

Share of projects with indicators not measurable due to lack of data

Rating scale 6 – Highly Satisfactory: The Bank’s interventions and systems demonstrate all characteristics of robust management for results: awareness and capacity of staff, strategic emphasis on results, robust monitoring systems guiding the portfolio, intervention logic and results-based framework are fully clear and realistic at both strategy and project levels. Bank’s strategy and projects systematically identify and integrate lessons from previous CSPs, include sound analysis of sustainability of past interventions, and comprehensively address past implementation issues implementing corrective measures.

5 –Satisfactory: The Bank’s interventions and systems demonstrate most characteristics of robust management for results: awareness and capacity of staff, strategic emphasis on results, robust monitoring systems guiding the portfolio, intervention logic and results-based framework are fully clear and realistic at both strategy and project levels. The Bank’s strategy and most projects identify and integrate lessons from previous CSPs, include sound analysis of sustainability of past interventions, and globally address past implementation issues implementing corrective measures.

4 – Moderately Satisfactory: The Bank’s interventions and systems demonstrate some of the characteristics of robust management for results: awareness and capacity of staff, strategic emphasis on results, robust monitoring systems guiding the portfolio, intervention logic and results-based framework are fully clear and realistic at both strategy and project levels. Bank’s strategy and most projects identify lessons from previous CSPs, address past implementation issues, and include an analysis of sustainability of past interventions. However the integration of lessons in the design and the implementation of corrective measures are not systematic.

3 – Moderately Unsatisfactory: The Bank’s interventions and systems demonstrate the existence of monitoring mechanisms but without clear focus on results in design and implementation of strategy and projects. Lessons are routinely mentioned but there is no evidence they are used to improve the design of strategy and projects.

2 – Unsatisfactory: The Bank’s interventions are not guided by results with generally unclear intervention logics and monitoring generally focusing on financial aspects. . The lack of learning is visible in the same factors explaining poor performance mentioned over time.

1 – Highly Unsatisfactory: The Bank has no monitoring system in place. The lack of learning is visible in the same factors explaining poor performance mentioned over time.

M9 – Partnership and coordination: Bank’s interventions are designed and implemented in partnership with

122

the government and other development partners at country level

Related CSPE questions EQ18 - To what extent are the Bank’s interventions harmonized with those of other donors (avoiding duplication, simplifying procedures etc.)?

Hypothesis and judgement criteria As a signatory of the Paris Declaration and then of the Busan partnership, the Bank has committed to harmonizing its interventions with other development partners at country level and to operating in partnership at country level. The Bank operates in a complex environment, and the choice of portfolio should be conditioned by the strategic focus of a government client together with the Bank’s comparative advantage as part of a mutually beneficial partnership. The government commitment to delivery in the areas to which the Bank intends to contribute is a key factor determining the degree to which successful implementation and performance reflects what was intended in the country strategy formulated. When a country is not fully committed to the outcomes to which the Bank is contributing, the Bank often faces a tough choice between disengaging from significant lending or engaging in areas that may be only tangentially related to the most important areas identified in its analysis. High-quality and timely diagnostics, policy dialogue, and technical assistance should help identify entry points and lay the groundwork for greater engagement, but the Bank's current approach is based on the assumption that greater genuine consultation will lead to greater ownership, as the Bank adjusts what it does to align with the actual priorities of key stakeholders. With regards coordination with other development partners, it is expected from the Bank that opportunities for partnership arrangements are systematically explored (Joint Assistance Strategies, co-financing arrangements, joint monitoring and evaluation arrangements, etc.), normally as part of the country strategy development process. The implementation of the program would also on-going coordination through the various mechanisms in place at country level. Examples of judgment criteria / indicators that can be included in the CSPE evaluation matrix rate this mechanism include:

Level of ownership of the Bank strategies by the government and other development stakeholders through participation to design and implementation

Level of analysis of country systems (PFM, procurement, MfDR) and inclusion of capacity development measures

Use of country system (PFM, Procurement, MfDR)

Appropriateness and effective functioning of DP coordination mechanism and role played by the Bank

Number of donor coordination working groups actively attended by Bank’s staff (overtime)

Perceived contribution of the Bank’s to donor coordination mechanisms

Bank’s participation in joint programs (i.e., multi-donor trust funds)

Extent of actual partner’s disbursement over allocations

Extent of involvement and consultation of other development partners in the formulation of CSPs and program proposals

Degree of collaboration with other development partners in analytical work, reviews and evaluation

Number of projects whose outcome achievements depend significantly upon other donors’ interventions

Number and importance of cases of duplication of interventions or major gaps (sectoral level)

Coherence of the approaches to budget support used by the Bank and other development partners

Rating scale 6 – Highly Satisfactory: The design of the strategy and projects proposes a clear analysis of donor’s positioning and coordination requirements, and involves a fully documented consultation process with all stakeholders fully demonstrating the effect of such consultation on the design. The Bank is an active contributor to donor coordination in all sectors covered and leads in at least half of the sectors. Consultations with all stakeholders are active and fully documented throughout the implementation of the program.

5 –Satisfactory: The design of the strategy and projects proposes a clear analysis of donor’s positioning and coordination requirements, and involves a fully documented consultation process with all stakeholders with the

123

effect of such consultation on the design partially demonstrated. The Bank is an active contributor to donor coordination in all sectors covered and leads in least one sector. Consultations with all stakeholders are active and mostly documented throughout the implementation of the program.

4 – Moderately Satisfactory: The design of the strategy and projects proposes an analysis of donor’s positioning and coordination requirements, and documents the consultation process with all stakeholders. The effect of such consultation on the design is visible although not clearly demonstrated. The Bank is an active contributor to donor coordination in all sectors covered. Consultations with all stakeholders are active during implementation although not always documented.

3 – Moderately Unsatisfactory: The design of the strategy and projects proposes a mapping of donor’s positioning with no clear analysis. The design documents the consultation process with all stakeholders but the effect of such consultation on the design is not documented. The Bank is a contributor to donor coordination in all sectors covered but not fully active in all. Consultations with all stakeholders are active during implementation but rarely documented.

2 – Unsatisfactory: The design of the strategy and projects proposes a mapping of donor’s positioning with no clear analysis. The design documents mentions a consultation process with all stakeholders but does not provide any detail on this process. The Bank is a passive contributor to donor coordination in limited sectors. There is no evidence of consultations with all stakeholders during implementation.

1 – Highly Unsatisfactory: The design of the Bank’s strategy and projects show no analysis of coordination constraints. No evidence of a consultation process with stakeholders can be found. The Bank Is inactive in the donor coordination mechanisms at country level.

C1 - Country ownership: the country has included the Bank’s CSP objectives as contributing results in national plans and monitors the achievement of CSP objectives


Hypothesis and judgement criteria The under-lying assumption is that development assistance is more effective when it is supporting what the government and those responsible for its delivery see that it is clearly contributing to objectives that are a priority to the key national stakeholders and where sufficient resources are devoted to make a difference. This is a similar argument for selectivity in what the Bank itself does in a country. The challenge is that in most contexts, what is truly important and a priority is not reflected in official government plans and strategies. In aid dependent countries, many strategy documents have been drafted for external consumption by technical experts. As such, these official documents do not reflect the political reality. In all governments, priorities will shift to varying degrees, in response to changing context. The presence of ownership would materialize in the fact that between 2004 and 2013, there is evidence of national operational strategies in the sectors/sub-sectors in which the Bank's portfolio is focused that drive decision making within government. There is also significant attention from ministerial/presidential level and by senior civil servants in achievement of outcomes under these strategies to which AfDB support will contribute. On the contrary, absence of the condition would be translated in the fact that national operational strategies in the sectors/sub-sectors in which the Bank's portfolio is focused are in place but have little influence on either what government does/prioritizes or the allocation of resources. In practice, there is little sense that delivery by the Bank investments/support is a concern as it is important to national stakeholders in terms of delivering the actual policy outcomes of the government.

Rating scale

124

Absence / Presence



Hypothesis and judgement criteria Contextual analyses type PEST (Political, Economic, Social, Technological) aspects are conducted in both CSP and project design documents and identify the key policy and legal constraints to the Bank making a significant contribution. It can be assumed that helping RMCs address such constraints is a major focus of the Bank's policy engagement. The under-lying assumption is that the Bank has been realistic about the scope, and commitment, of the Government to enact change in the policy and legal framework that identified as essential. The presence of this readiness to reform would materialize in the fact that between 2004 and 2013, the majority of key policy and legal constraints to the Bank making a significant contribution that are identified in the CSP/project documents' PEST type analyses were effectively addressed by Government On the contrary, the absence would be evidenced by the fact that between 2004 and 2013, the majority of key policy and legal constraints to the Bank making a significant contribution that are identified in the CSP/project documents' PEST type analyses were not effectively addressed by Government

Rating scale Absence / Presence

125

CEDR Synthesis

Country analysis template

21 September 2015

126

M1 – Knowledge and strategic advice: the Bank provides the country with relevant knowledge and appropriate advice, and participates effectively to policy dialogue

Analysis:

Rating: 6 – Highly Satisfactory: The Bank has delivered significant knowledge work in all sectors selected for its interventions that has been recognized by all other development partners including the government, allowed the Bank for positioning itself as leader in policy dialogue in these areas, and has triggered change in the policy and/regulatory framework of the country.

5 –Satisfactory: The Bank has delivered significant knowledge work in most sectors selected for its interventions that has been recognized by all other development partners including the government, allowed the Bank for positioning itself as leader in policy dialogue in these areas, and has triggered change in the policy and/regulatory framework of the country.

4 – Moderately Satisfactory: The Bank has delivered knowledge work in few sectors selected for its interventions that has been recognized by all other development partners including the government, and allowed the Bank to contribute to policy dialogue in these areas.

3 – Moderately Unsatisfactory: The Bank has delivered limited knowledge work in a few sector selected for its interventions that has been recognized by all other development partners including the government, and is contributing to policy dialogue, but with no linkage between the two aspects.

2 – Unsatisfactory: The Bank has delivered minimal knowledge work in a few sectors selected for its interventions with no recognition by / influence on other development partners and government. The Bank is routinely involved in policy dialogue.

1 – Highly Unsatisfactory: The Bank has not delivered any significant piece of knowledge work. It is seen as one actor among many in the development landscape of the country by most stakeholders.

127

M2 - Adapted solutions: the Bank’s strategy at country level is well designed, and proposes the right solutions to challenges

Analysis:

Rating: 6 – Highly Satisfactory: The Bank’s strategy is based on a compelling intervention logic demonstrating a thorough understanding of the country’s evolving context, proposing solutions fully adapted to this context in all areas of intervention, and showing substantive innovation (analysis and product mix) over time in responding to challenges to achieving results.

5 –Satisfactory: The Bank’s strategy is based on a sound intervention logic demonstrating a thorough understanding of the country’s evolving context, proposing solutions variably adapted to this context depending on the area of intervention, and showing some innovation (analysis and product mix) over time in responding to challenges to achieving results.

4 – Moderately Satisfactory: The Bank’s strategy is based on a sound intervention logic demonstrating a good understanding of the country’s evolving context, proposing solutions variably adapted to this context depending on the area of intervention, and not showing much innovation (analysis and product mix) over time in responding to challenges to achieving results.

3 – Moderately Unsatisfactory: The Bank’s strategy is based on an intervention logic showing partial understanding of the country’s evolving context, proposing solutions variably adapted to this context depending on the area of intervention, and not showing much innovation (analysis and product mix) over time in responding to challenges to achieving results.

2 – Unsatisfactory: The Bank’s strategy is based on an unclear intervention logic showing low understanding of the country’s evolving context, proposing “business as usual” solutions variably adapted to this context depending on the area of intervention.

1 – Highly Unsatisfactory: The Bank’s strategy is disconnected from the evolving context of the country and used as justification for an ongoing portfolio of operations designed and implemented as a compilation of projects agreed with direct national counterparts in the area covered.

128

M3 - Strategic focus: the Bank’s strategy is focused on areas of comparative advantage, and well integrated.

Analysis:

Rating: 6 – Highly Satisfactory: The Bank’s strategy presents a compelling analysis of the respective positioning of development partners, areas of comparative advantage and matching of this analysis with the evolving context and challenges of the country to define priority areas of assistance for the Bank and the interconnections between those from a programmatic perspective.

5 –Satisfactory: The Bank’s strategy presents a clear analysis of the respective positioning of development partners, areas of comparative advantage and matching of this analysis with the evolving context and challenges of the country to define priority areas of assistance for the Bank but does not fully articulate interconnections and integration constraints in the program.

4 – Moderately Satisfactory: The Bank’s strategy presents an analysis of the respective positioning of development partners and areas of comparative advantage but the analysis does not fully show how this translates into priority areas of assistance for the Bank matching the evolving context and challenges of the country.

3 – Moderately Unsatisfactory: The Bank’s strategy proposes an analysis of positioning and comparative advantage. Priority areas of assistance aligned with needs are proposed without relating them to this analysis.

2 – Unsatisfactory: The Bank’s strategy proposes a basic analysis of positioning and comparative advantage but fails to articulate clearly the result in terms of areas of intervention which in practice overlap with areas of intervention of the ongoing portfolio.

1 – Highly Unsatisfactory: The Bank’s strategy replicates areas of intervention of the ongoing portfolio without any convincing analysis of positioning.

129

M4 - Leverage: the Bank’s strategy and program are designed so as to crowd in additional resources and with attention to scaling up

Analysis:

Rating: 6 – Highly Satisfactory: The Bank’s has demonstrated an instrumental role in bringing in additional financing in the country through dialogue and usage of relevant instruments, with a leveraging effect of at least 1 to 5 overall. The program design shows explicit and consistent attention to scaling up both at strategic and project level.

5 –Satisfactory: The Bank’s has demonstrated a significant role in bringing in additional financing in the country through dialogue and usage of relevant instruments with a leveraging effect of at least 1 to 3 overall. Most key projects show explicit attention to scaling up.

4 – Moderately Satisfactory: The Bank’s has had an opportunistic but active role in bringing in additional financing in the country through dialogue and usage of relevant instruments. A few projects show a leverage effect and explicit attention to scaling up.

3 – Moderately Unsatisfactory: Leverage is taken as an ad-hoc consideration at project level without specific attention given at the strategic level. No specific consideration is given to scaling up.

2 – Unsatisfactory: The Bank mentions leverage as an objectives and articulates some action but with no visible effect.

1 – Highly Unsatisfactory: The Bank had no leverage effect on financing in the country and does not articulate any action for that purpose.

130

M5 – Supervision: the Bank’s program is monitored effectively with a focus on ensuring the achievement of expected results

Analysis:

Rating: 6 – Highly Satisfactory: The Bank’s has established a robust monitoring system including timely and quality supervisions, systematic follow up on actions, and review processes at all levels (operation to sector to country) including issues and risks and involving all stakeholders. The effect on the quality of the portfolio can be demonstrated.

5 –Satisfactory: The Bank’s has established a monitoring system including timely and quality supervisions, and systematic follow up on actions. Review processes include issues and risks and involve all stakeholders, but are not systematic at all levels (operations to sector to country). The effect on the quality of the portfolio can be demonstrated.

4 – Moderately Satisfactory: The Bank’s has established a monitoring system including timely and quality supervisions, and follow up on actions at the project level, with no overall review processes. It is difficult to clearly demonstrate the effect on the quality of the portfolio.

3 – Moderately Unsatisfactory: The Bank’s has established a monitoring system including timely supervisions, but the quality of supervision reports is variable as well as follow up on actions. It is difficult to clearly demonstrate the effect on the quality of the portfolio.

2 – Unsatisfactory: The Bank’s has established a monitoring system including timely supervisions, but the quality of supervision reports is poor and follow up on actions is not clear. There is no evidence that it is having any sort of effect on the quality of the portfolio.

1 – Highly Unsatisfactory: The supervision process is managed on an ad-hoc basis to comply with institutional requirements and there is no evidence that it is having any sort of effect on the quality of the portfolio.

131

M6 – Project focus: the design and implementation of Bank’s projects are focused on contributing to CSP outcomes

Analysis:

Rating: 6 – Highly Satisfactory: More than 90% of the projects examined in the country had a rating of highly satisfactory for criteria 1a. None had a highly unsatisfactory rating.

5 – Satisfactory: More than 80% of the projects examined in the country had a rating of satisfactory or higher for criteria 1a.

4 – Moderately Satisfactory: More than 60% of the projects examined in the country had a rating of satisfactory or higher for criteria 1a. Less than 30% had a rating of unsatisfactory or less.

3 – Moderately Unsatisfactory: Less than 60% of the projects examined in the country had a rating of satisfactory or higher for criteria 1a. More than 50% had a rating of moderately satisfactory or higher.

2 – Unsatisfactory: Less than 50% of the projects examined in the country had a rating of moderately satisfactory or higher for criteria 1a.

1 – Highly Unsatisfactory: Less than 30% of the projects examined in the country had a rating of moderately satisfactory or higher for criteria 1a.

132

M7 – Project design: the design of Bank’s projects address constraints to contributing to CSP outcomes

Analysis:

Rating: 6 – Highly Satisfactory: More than 90% of the projects examined in the country had a rating of highly satisfactory for criteria 1b. None had a highly unsatisfactory rating.

5 – Satisfactory: More than 80% of the projects examined in the country had a rating of satisfactory or higher for criteria 1b.

4 – Moderately Satisfactory: More than 60% of the projects examined in the country had a rating of satisfactory or higher for criteria 1b. Less than 30% had a rating of unsatisfactory or less.

3 – Moderately Unsatisfactory: Less than 60% of the projects examined in the country had a rating of satisfactory or higher for criteria 1b. More than 50% had a rating of moderately satisfactory or higher.

2 – Unsatisfactory: Less than 50% of the projects examined in the country had a rating of moderately satisfactory or higher for criteria 1b.

1 – Highly Unsatisfactory: Less than 30% of the projects examined in the country had a rating of moderately satisfactory or higher for criteria 1b.

133

M8 – Managing for results & Learning: the Bank's interventions are designed and managed for development results at country level and informed by evidence of what works and doesn't work.

Analysis:

Rating: 6 – Highly Satisfactory: The Bank’s interventions and systems demonstrate all characteristics of robust management for results: awareness and capacity of staff, strategic emphasis on results, robust monitoring systems guiding the portfolio, intervention logic and results-based framework are fully clear and realistic at both strategy and project levels. Bank’s strategy and projects systematically identify and integrate lessons from previous CSPs, include sound analysis of sustainability of past interventions, and comprehensively address past implementation issues implementing corrective measures.

5 –Satisfactory: The Bank’s interventions and systems demonstrate most characteristics of robust management for results: awareness and capacity of staff, strategic emphasis on results, robust monitoring systems guiding the portfolio, intervention logic and results-based framework are fully clear and realistic at both strategy and project levels. The Bank’s strategy and most projects identify and integrate lessons from previous CSPs, include sound analysis of sustainability of past interventions, and comprehensively address past implementation issues implementing corrective measures.

4 – Moderately Satisfactory: The Bank’s interventions and systems demonstrate some of the characteristics of robust management for results: awareness and capacity of staff, strategic emphasis on results, robust monitoring systems guiding the portfolio, intervention logic and results-based framework are fully clear and realistic at both strategy and project levels. Bank’s strategy and most projects identify lessons from previous CSPs, address past implementation issues, and include an analysis of sustainability of past interventions. However the integration of lessons in the design and the implementation of corrective measures are not systematic.

3 – Moderately Unsatisfactory: The Bank’s interventions and systems demonstrate the existence of monitoring mechanisms but without clear focus on results in design and implementation of strategy and projects. Lessons are routinely mentioned but there is no evidence they are used to improve the design of strategy and projects.

2 – Unsatisfactory: The Bank’s interventions are not guided by results with generally unclear intervention logics and monitoring generally focusing on financial aspects. . The lack of learning is visible in the same factors explaining poor performance mentioned over time.

1 – Highly Unsatisfactory: The Bank has no monitoring system in place. The lack of learning is visible in the same factors explaining poor performance mentioned over time.

134

M9 – Partnership and coordination: Bank’s interventions are designed and implemented in partnership with the government and other development partners at country level

Analysis:

Rating: 6 – Highly Satisfactory: The design of the strategy and projects proposes a clear analysis of donor’s positioning and coordination requirements, and involves a fully documented consultation process with all stakeholders fully demonstrating the effect of such consultation on the design. The Bank is an active contributor to donor coordination in all sectors covered and leads in at least half of the sectors. Consultations with all stakeholders are active and fully documented throughout the implementation of the program.

5 –Satisfactory: The design of the strategy and projects proposes a clear analysis of donor’s positioning and coordination requirements, and involves a fully documented consultation process with all stakeholders with the effect of such consultation on the design partially demonstrated. The Bank is an active contributor to donor coordination in all sectors covered and leads in least one sector. Consultations with all stakeholders are active and mostly documented throughout the implementation of the program.

4 – Moderately Satisfactory: The design of the strategy and projects proposes an analysis of donor’s positioning and coordination requirements, and documents the consultation process with all stakeholders. The effect of such consultation on the design is visible although not clearly demonstrated. The Bank is an active contributor to donor coordination in all sectors covered. Consultations with all stakeholders are active during implementation although not always documented.

3 – Moderately Unsatisfactory: The design of the strategy and projects proposes a mapping of donor’s positioning with no clear analysis. The design documents the consultation process with all stakeholders but the effect of such consultation on the design is not documented. The Bank is a contributor to donor coordination in all sectors covered but not fully active in all. Consultations with all stakeholders are active during implementation but rarely documented.

2 – Unsatisfactory: The design of the strategy and projects proposes a mapping of donor’s positioning with no clear analysis. The design documents mentions a consultation process with all stakeholders but does not provide any detail on this process. The Bank is a passive contributor to donor coordination in limited sectors. There is no evidence of consultations with all stakeholders during implementation.

1 – Highly Unsatisfactory: The design of the Bank’s strategy and projects show no analysis of coordination constraints. No evidence of a consultation process with stakeholders can be found. The Bank Is inactive in the donor coordination mechanisms at country level.

135

C1 - Country ownership: the country has included the Bank’s CSP objectives as contributing results in national plans and monitors the achievement of CSP objectives

Analysis:

Rating: The presence of ownership would materialize in the fact that between 2004 and 2013, there is evidence of national operational strategies in the sectors/sub-sectors in which the Bank's portfolio is focused that drive decision making within government. There is also significant attention from ministerial/presidential level and by senior civil servants in achievement of outcomes under these strategies to which AfDB support will contribute. On the contrary, absence of the condition would be translated in the fact that national operational strategies in the sectors/sub-sectors in which the Bank's portfolio is focused are in place but have little influence on either what government does/prioritizes or the allocation of resources. In practice, there is little sense that delivery by the Bank investments/support is a concern as it is important to national stakeholders in terms of delivering the actual policy outcomes of the government.

136


Analysis:

Rating: The presence of this readiness to reform would materialize in the fact that between 2004 and 2013, the majority of key policy and legal constraints to the Bank making a significant contribution that are identified in the CSP/project documents' PEST type analyses were effectively addressed by Government On the contrary, the absence would be evidenced by the fact that between 2004 and 2013, the majority of key policy and legal constraints to the Bank making a significant contribution that are identified in the CSP/project documents' PEST type analyses were not effectively addressed by Government