Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
ZAMBIA - Nkana Water Supply and Sanitation Project 1. BASIC INFORMATION
a. Basic project data
Project title: Nkana Water Supply and Sanitation Project
Project code: P-ZM-E00-008 Instrument number(s): 2100150018345
Project type: Public Operation Sector: Water and Sanitation
Country: Zambia Environmental categorization (1-3) : 2
Processing Milestones Key Events Disbursement and Closing date
Date approved: 27/11/2008 Cancelled amount: Original disbursement deadline:
31/12/2013
Date signed: 22/12/2008 Supplementary financing: NA Original closing date: 31/12/2013
Date of entry into force :
29/01/2009
Restructuring: NA Revised disbursement deadline:
31/03/2015
Date effective for 1st
disbursement: 01/09/2009
Extensions (specify dates): 31/12/2013 to
30/09/2014, 30/09/2014 to 31/03/2015
Revised closing date: 31/03/2015
Date of actual 1st : 01/09/2009
b. Financing sources
Financing source/ instrument
(MUA)
Approved amount
(MUA) :
Disbursed amount
(MUA) :
Percentage disbursed
(%):
Loan: ADF 34,998,407.20 34,998,407.20 99.99%
Government: 3,820,000 3,820,000 89.10%
Other (ex. Co-financiers):
TOTAL : 38,818,407.20 38,818,407.20 98.92%
Co-financiers and other external partners:
Execution and implementation agencies: Nkana Water and Sewerage Company
c. Responsible Bank staff
Position At approval At completion
Regional Director - Kennedy Mbekeani (OIC)
Sector Director Andrianarison Rakotobe Mohamed El Azizi
Sector Manager Sering Jallow Osward Chanda
Task Manager Mecuria Assefaw Herbert Chinokoro
Alternate Task Manager -
PCR Team Leader - Alex Gomani
PCR Team Members - Herbert Chinokoro
Monia Moumni (peer reviewer)
Aicha Mohamed Moussa (peer
reviewer)
Kelvin Banda (peer reviewer)
d. Report data
PCR Date : 30/11/2015
PCR Mission Date: From: 19/10/2015 To: 23/10/2015
PCR-EN Date: 29/06/2018
Evaluator/consultant : Daniel VAN ROOIJEN Peer Reviewer/Task Manager: Ram Janakiram
PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS
2. PROJECT DESCRIPTION Summary from Appraisal Report including addendum/corrigendum or loan agreement, and taking into account any modification that
occurred during the implementation phase.
The project aimed to increase improved access to water supply and sanitation to residents in three urban centres of
Kitwe, Kalulushi and Chambishii in the Copperbelt Region of Zambia. The Government of Zambia was the
borrower of the ADF loan, with the Nkana Water and Sewerage Company (NWSC), under the MLGH, being the
Executing Agency of the project. Most of the funding was provided by the African Development Fund through a
loan with a small contribution provided by the Government of Zambia. The project agreement was signed in
December 2008 and closed in March 2015 after 2 extensions.
a. Rationale and expected impacts:
The project rationale was born from the current physical conditions of old water and sewerage infrastructure in the
three urban centres in whit substantial water losses occurring in the water supply systems. In order to keep up with
the growing population and subsequent demand for water and sewerage infrastructure, the need for rehabilitation
and expansion of the current systems was considered essential. Another reason to develop this project was the
understanding of the important factor of water and sanitation services for the status of human health. The expected
impact envisaged by the project was to increase the proportion of the population with access to adequate water
supply and sanitation services as well as improvement of the health conditions of the entire population.
b. Objectives and Expected Outcomes: The overall project objective was to improve the access to adequate water supply and sanitation services in Kitwe,
Kalulushi and Chambishi Towns.
Discrepancies exist between the outcomes listed in the appraisal report and outcomes listed in the PCR. The
appraisal report has listed 4 outcomes while the PCR lists 5 more outcomes included which are in the area of the
performance water service delivery (outcome 4-8) and the functioning of so called resident development
committees (outcome 9). The 4 outcomes as listed in the project appraisal report were:
1) Increased proportion of the residents of the three urban centres benefiting from improved water supply services
2) Increased number of the people in the three urban centres benefiting from improved sanitation facilities
3) Reduced incidence of waterborne diseases
4) Improved management of water and sanitation services
Beneficiaries of these outcomes, as listed in the log-frame of the appraisal report, are:
- Urban and peri-urban populations of Kitwe, Kalulushi and Chambishi
- The two local authorities
- Nkana Water and Sewerage Company (NWSC)
c. Outputs and intended beneficiaries: Expected outputs were:
o Detailed designs prepared
o Water supply system rehabilitated, augmented and extended
o Sewerage system rehabilitated, augmented and extended
o Increased awareness on the need for improved public health and personal hygiene
o Gender mainstreamed and integrated in the training and communication materials
o Community organizations in semi urban trained and able to manage public water supply points
o Water kiosks constructed
o On-site sanitation facilities constructed
o Water supply facilities provided and rehabilitated in selected public institutions
o Sanitation facilities provided and rehabilitated in selected public institutions
o Support to improved solid waste management
o M&E system including gender specific indicators developed
o Performance Contract signed between NWSC and MLGH
o Enhanced institutional capacity of NWSC to manage water and sewerage services
o Enhanced institutional capacity of municipal authorities to manage solid waste services
Beneficiaries of these outputs, as listed in the log-frame, were:
- Nkana Water and Sewerage Company (NWSC)
- Kitwe City Council (KCC)
- Local authorities
- Contractors
- Residents of the three towns
- Community organizations
- NGOs
- Ministry of Local Government and Housing (MLGH)
- Ministry of Energy and Water Development (MEWD)
- Health Centres
- Schools
d. Principal Components and/or activities:
The project has the following 5 components:
1. Water supply works
2. Sewerage works
3. Sanitation and Hygiene Education
4. Institutional Support to NWSC
5. Project Management
Principal activities, as listed in the log-frame of the appraisal report, are:
1. Engineering services
2. Construction works for water supply system
3. Construction works for the sewerage system
4. Sensitization and promotion of sanitation and personal hygiene for the urban poor in the project areas
5. Promotion of use of public water points and on-site sanitation in semi/peri urban areas
6. Water supply and sanitation facilities in public institutions
7. Institutional Support
The main activities by project components are provided below:
Component One : Water Supply Works: Kitwe, Kalulushi and Chambishi
1.Rehabilitation of Bulangililo Water Treatment Plant (WTP),
2. Rehab & expansion of East Nkana WTP for an additional 32,000 m3/d;
3.Rehabilitation of the existing water transmission and distribution systems in Kitwe;
4.New intake works on Mwambashi river and a 10 km pumping main to existing 7 Shaft WTP (15,000 m3/d),
refurbishment of the plant, and rehabilitation and extension of distribution system;
5.Transmission from the mine raw water mains to a new treatment plant with a 10,000 m3/d capacity, and
Rehabilitation of distribution system.
Component Two : Sewerage Works in Kitwe, Kalulushi and Chambishi
1.Rehabilitation of waste water collection and, transportation system and treatment plants (82,560 m3/d) in Kitwe
and expansion of treatment capacity by 5,000 m3/d;
2.Rehabilitation of existing sanitation facilities in Kalulushi including 10,000 m3/d waste stabilization ponds;
3.Rehabilitation of existing sanitation facilities in Chambishi including 5,000 m3/d waste stabilization ponds
Component Three: Sanitation and Hygiene Education
1. Carry out sanitation promotion, hygiene education and support for onsite household sanitation.
2.Provision of water and sanitation facilities for public institutions (schools, health centers) and Community
capacity building, and support related to solid waste management in the three urban and peri-urban areas covered
by the project.
Four: Institutional Support to NWSC
1.Improvement of NWSC’s MIS, tariff/affordability study,
2.Support for Consumer Watch Groups, workshop and
office rehabilitation, leakage control, and provision of
logistics, including operational and specialized utility vehicles
Five: Project Management
1.Provision of engineering services for detailed designs, procurement and supervision, including assessment of
alternative sustainable sources of water in the project area,
2.Support to PIT for project management and Project Audit
3. PROJECT PERFORMANCE ASSESSMENT
RELEVANCE
a. Relevance of the project development objective:
The project’s development objective, to improve the access to adequate water supply and sanitation services in
Kitwe, Kalulushi and Chambishi Towns, is considered very relevant with regards to the government’s goals with
regards to reducing poverty and improving health outcomes of the population of Zambia. The project scope fits
well with strategies and priorities by the Bank (for example its policy on integrated water resources management).
It is also well embedded into the water vision and sector strategies, principles and policies by the Government of
Zambia. The vision in Zambia’s Water policy states: “To optimally harness water resources for the efficient and
sustainable utilisation of this natural resource to enhance economic productivity and reduce poverty.” In addition,
the programme is aligned with the water act and other sector strategies. An important strategy, published by
Zambia’s Ministry of Local Government and Housing in the final year of the project implementation period, is
the national urban and peri-urban sanitation strategy. This strategy aims to “provide adequate, safe and cost-
effective sanitation services to 90 percent of the urban population by 2030”. The project’s scope will allow it to
greatly contribute to this goal. The evaluator concurs with the PCR rating for this criterion and considers it
as highly satisfactory (4). The reviewer concurs with the assessment and rating for this criterion - 4 –
Highly satisfactory.
b. Relevance of project design (from approval to completion):
The project design is considered comprehensive, logic and relevant in light of the development objectives and
sector strategies set by the government of Zambia. The appraisal report demonstrates that risks have been
assessed in an adequate manner and environmental and social protection measures have been well incorporated.
The design has taken into account the lessons learned from a precious urban water and sanitation project financed
by the Bank for Kitwe Town. The inclusion of knowledge management and water supply performance indicators
is much welcomed. Participatory approaches in project design (and implementation), as well as a strong focus on
water demand management with community participation is recognized as very relevant to ensure that benefits
from the project are sustained after project completion. The evaluator concurs with the PCR rating for this
criterion and considers it as satisfactory (3).
Additional design elements incorporated in the project are:
The NWSSP was designed as a stand-alone operation which was expected to contribute to the
development of a program approach for development of water supply and sanitation services in all urban
and peri-urban areas of the country
Based on priorities identified through a pre-feasibility study carried out by the Borrower encompassing
technical, economic and financial analyses.
Included conditions on reporting on the key financial indicators of the company to ensure sustainability
of the project outputs and outcomes.
Whenever possible and without sacrificing the expected outputs, standard and less expensive solutions
were selected. For example, uniformity of types of equipment and fittings like pumps and valves, meters
and other fittings, etc.
EFFECTIVENESS
c. Effectiveness in delivering outputs:
Discrepancies exist between the outputs listed in the appraisal report and outputs listed in the PCR. The appraisal
report has listed 15 outputs while the PCR lists 18 outputs. This evaluation will focus on the 18 outputs listed in
the PCR. Some of the outputs listed in the project appraisal report such as output 4.1 on “increased awareness” or
4.3 on “ability to manage water points” focus on behavioural change and capacity strengthening and hence should
be considered as project outcomes instead of outputs.
The PCR narrative is limited to the civil works carried out by contractors being compliant to the contracts. It does
not cover other outputs listed.
Regarding output 2 and 3 on water production and wastewater treatment, the electricity disruption affecting water
production is considered as a valid reason which lies outside the power of the project. Output 4 (Number of
operable public water points) does not capture whether the water points are in operation or not. Instead of
“operable water points”, this should have been “functioning water points”, in order to monitor actual water
access. Regarding output 5 (% public water points operated by women), the reported figure on progress was not
well calculated. Since target was not reached, the value should be below 100%. Regarding output 7 on the
percentage of female headed households benefiting from onsite sanitation, the narrative uses an absolute number
while the indicator is used as %. The reported figure on progress was not well calculated. With current value 30%
against target of 43.3% the target was not reached. The value on progress should be below 100%. Regarding
output 8 on water supply to schools and health centres, there is unfortunately no details provided on the no. of
schools vs the no. of health centres. Regarding sold waste management (output 10), it would have been good to
get more details on the type of solid waste management (what, how, where, etc.). Regarding output 14, it is
unclear how progress of 84% was calculated. Based on the no. of teachers and health inspectors combined, the
progress has exceeded the target (>100%) which is not reflected in the PCR.
Based on the below reflection on output-level project performance, the evaluator would rate this criterion
as satisfactory (3) which is in line with the PCR score.
d. Effectiveness in delivering outcomes:
Discrepancies exist between the outcomes listed in the appraisal report and outcomes listed in the PCR. The
appraisal report has listed 4 outcomes while the PCR lists 5 more outcomes included which are in the area of the
performance water service delivery (outcome 4-8) and the functioning of so called resident development
committees (outcome 9). This evaluation will focus on the 9 outcomes listed in the PCR.
While the PCR narrative rightly points at the important livelihoods opportunities which the project brought in
terms of building skills in construction and casting sanitation platforms, these software gains have not been
included in the M&E framework or captured in the list of outcomes.
The increase in access coverage of water supply services is commendable. It would have been good to know if
water supply was also expanded to unplanned settlements, as was done for sanitation. For sanitation, it is not
entirely clear what proportion of people benefiting from improved sanitation is at the household level. Regarding
disease incidence, the reduction in disease incidence can likely be attributed to the project interventions. It would
have been good to see the numbers for all years in order to see whether an actual positive trend can be observed.
The fact that there has not been any major disease outbreak since 2012 is definitely a good sign, however in order
to conclude something on attribution this should always be compared with other urban centres that did not have
an intervention. The progress made on increasing metered connections deserves praise. Regarding unaccounted
for water, the reported progress made of 176% should be negative since recent value is worse than the baseline
value. It is based on a wrong calculation. Baseline is 42%, target is 25% and current value is 44%. This would
lead to the following calculation: Progress against target = (recent value – baseline value) / (target value – baseline) = (44% - 42%) / (25% - 42%) = -12%
Regarding outcome 6 on increasing the collection ratio, the stated reasons for not meeting the target are well
justified. Regarding average hours of water supply (outcome 7); the reported progress made of 67% should be
negative (-33%) since the most recent value is worse than the baseline value. The target of 24hours water supply
on average is considered not realistic and should have been lowered at project design phase. Regarding outcome 8
on water quality; it would have been good to have included a separate indicator for water treated for consumption
and treated wastewater because each water category have different water quality requirements and subsequent
standards.
Based on the below reflection on outcome-level project performance, and given the shortcomings, lack of
robust evidence and data to determine the number of beneficiaries, (due in large part to very weak and
non-functional M&E system), absence of assessment of the outcomes of the capacity building program, the
evaluator would rate this criterion as unsatisfactory (2) which is lower than the PCR score.
e. Project development outcome:
Despite some incomplete or inaccurate reporting at the outcome and output level, the overall progress made with
regards to the development objective is considerable. Substantial gains have been made with both the expansion
of water service provision as well as sanitation infrastructure in the 3 towns. It would have been good to include
indicators for behavioural change with regards to hygiene and capacity strengthening in the log-frame, in order to
capture progress in these important soft interventions areas. This would have been relevant with regards to
sustainability as well.
However, given the 2 ranking for outcomes, this criterion is ranked unsatisfactory (2).
f. Beneficiaries:
The PCR has listed 7 categories of beneficiaries. Some overlap seem to exist in the report, for example when
comparing row 2 and row 4 on beneficiaries of water supply and sanitation in public places. There is also a lack
of consistency in the use of peri-urban or urban population, which makes it difficult to ascertain whether peri-
urban or urban population is referred to. The gender proportion has been included for all beneficiary groups,
however they have not been disaggregated for each group.
g. Unanticipated additional outcomes (positive or negative, not taken into consideration in the project
logical framework):
The PCR reports 4 unanticipated additional positive outcomes with regards to health, private sector development,
gender equality and social equality for disabled people which were not included in the original log-frame. These
outcomes are considered valid and relevant to the development objective of the project.
EFFICIENCY
h.Timeliness: The project period lasted 6.22 years instead of the planned 5 years which is just over a year longer than planned.
The resulting ratio of planned implementation time (PAR) and actual project implementation time from the date
of effectiveness is 0.80 which is within the category (<1 and ≥0.75) and therefore considered satisfactory. The
PCR provides comprehensive details as to why the project implementation took longer than anticipated and the
reasons as to why the delays have occurred.
However, the reasons provided in the PCR are inadequate to justify the delays in implementation as mentioned
above. More proactive steps could have been taken to reduce the delays. The PCR has rated this indicator as 2
which is appropriate. The reviewer recommends to retain this rating.
i. Resource use efficiency:
The project shows overall good evidence of physical implementation versus resources used. The median
percentage physical implementation of 100%, compared to a total commitment rate of 98.92%. The calculated
proportion between the median physical implementation rate and commitment rate is 1.01 which falls into the
category of ≥1 and corresponds to a rating of highly satisfactory (4), which is higher than the PCR rating of
satisfactory (3) for this criterion.
j. Cost-benefit analysis:
The appraisal report mentions that the Economic Internal Rate of Return of 16.9%. The PCR states that the EIRR
has risen from 16.9% to 17.8% upon project completion. The methodology and calculations carried out as part of
the economic analysis is not detailed in the PCR itself. Reference is made to an annex which is not available to
the evaluator. It is therefore difficult to assess the validity and assumptions made for analysing the economic costs
vs benefits of this project based on the information provided. Therefore, it is difficult for the evaluator to compare
and evaluate the ratio of anticipated EIRR at appraisal versus the EIRR at project completion.
However, the economic analysis carried out at appraisal makes the following assumptions:
(i) UFW is projected to be reduced from 43% to 25% as a result of the project,
(ii) Incremental sanitation revenue estimated at 25% of water revenue,
(iii) Average tariff is projected to increase from ZMK 1350/ m3 to reach ZMK 2500/ m3 by 2021
(iv) Incremental operations and maintenance costs are estimated at 15% of the incremental revenue.
(v) Health benefits from improved water supply and sanitation services, as well as economic value of
sludge produced from the improved facilities are considered
(vi) Economic life of the investment is estimated over 25 years.
Since the above assumptions were made at appraisal in 2008, it would be useful to demonstrate how realistic
these assumptions were, deviations from these estimates and provide the relevant evidence and data to support
this.
k. Implementation progress:
As noted earlier substantial progress was made during the project implementation. Sufficient evidence has been
given of progress at the output, outcome and impact level.
The PCR narrative reports that Procurement and financial management remained consistent with Bank systems
and procedures. However, a number of audit reports reported on issues found with compliance to financial
protocols and procedure, poor financial management including incomplete financial reporting, poor record
keeping and improper use of accounting software.
The monitoring and evaluation framework has not sufficiently captured progress made in the areas of capacity
building, institutional strengthening and behavioural change. Social and environmental safeguards were
sufficiently accounted for in the appraisal report. The PCR reports that an environmental and social management
plan and resettlement policy framework was put in place and monitored during project implementation. It would
have been useful if key data from this framework would have been included in the results reporting, for
evaluation. The evaluator rates this criterion with unsatisfactory (2), which is lower than the PCR score of
satisfactory (3).
SUSTAINABILITY
(l).Financial sustainability:
The PCR narrative for this section is comprehensive and demonstrates a realistic view on issues with operational
management of the (peri-) urban water supply schemes. It is not clear whether the Nkana water and sewerage
company has received technical support in terms of water demand modeling and scenario analysis of non-
revenue water in light of overall financial sustainability of the water supply systems. As the narrative rightly
points out, it will be critical to periodically carry out regular tariff adjustments if necessary. Community
sensitisation is a critical aspect for making sure consumers are willing to pay for improved water services. There
is no mention of payment modalities of public sanitation facilities, despite public sanitation being one of the core
intervention areas of the project. From literature it is known that users of sanitation facilities have a capacity and
willingness to pay for using these services. Therefore it should have been included in the overall assessment of
financial sustainability of the project outcomes. The evaluator would concur with the PCR rating for this
criterion and rates it as unsatisfactory (2).
The appraisal report mentions that at full capacity, the incremental revenue (ZMK 47 billion) generated from the
project will more than offset the incremental recurrent costs (ZMK 6.4 billion). The incremental revenues as well
as the cost savings generated from the project will more than offset the incremental recurrent costs. Consequently,
there will not be any need for budgetary support or subsidies from GRZ to the operations and maintenance of the
water supply and sanitation system in the project area. (Source; Appraisal report, August 2008, Section 4.4.
Sustainability section). Was this incremental revenue achieved? Evidence and supporting data is needed to
validate this achievement.
m. Institutional sustainability and strengthening of capacities:
The PCR report has discussed the various activities carried out that have led to strengthening technical capacities
of the water and sewerage company and government agencies as well as community groups. The project also
supported the enactment of the Zambia Environmental Management Agency Act of 2011, which is commendable
and will help manage the environment. While all these elements are positive and would help ensure institutional
sustainability, these outcomes should have been measured and included in the M&E framework. Simple surveys
could have captured changed capacities at the organisation level, community level, as well as at the individual
level. This is considered essential and without this kind of evidence the evaluator is unable to conclude the rate to
which institutional sustainability is ensured. Therefore the evaluator is unable to rate this criterion.
n. Ownership and sustainability of partnerships:
Despite a comprehensive narrative provided, the information available for true verification of this criterion
remains somewhat limited. The evaluator agrees with the justification provided with regards to efforts made to
ensure ownership and sustainability of partnerships, especially by involving the communities at the various
project stages. This criterion is rated as satisfactory (3), in line with the PCR score.
o. Environmental and social sustainability:
The PCR reports that Environmental and Social Management Plan and Resettlement Policy Framework was put
in place and monitored during project implementation. As noted earlier, it would have been useful if key data
recorded as part of this plan and framework would have been included in the results reporting, for evaluation. The
project has reported strong involvement of communities at various stages of the project which is considered key
to ensure social sustainability. To ensure social acceptance it will important to ensure that water tariffs are
appropriate and if possible pro-poor. The gains made in terms of increased wastewater treatment capacities and
water saved through reduction of physical losses are very likely to have a positive effect on the environment. The
evaluator concurs with the PCR score and rates this criterion as satisfactory (3).
4. PERFORMANCE OF STAKEHOLDERS
a. Bank Performance:
The bank has overall performed well during the design, implementation and completion phase of this project.
Regular field visits have been carried out and performance including elements of adherence to Banks protocols
have been evaluated during the project by external evaluators. Perhaps more attention could have been given to
timeliness of implementation of activities and a stronger supervision on financial administration after issues
reported in the audits. There is no evidence that a mid-term review has taken place in this project, which is
considered essential in project supervision. The evaluator would rate the Banks performance as satisfactory
(3) which is in line with the PCR rating.
h. Borrower Performance
The project Nkana Water and Sewerage Company Government was the executing agency and the Republic of
Zambia was the borrower. Performance of the Borrower is considered satisfactory (3), despite various issues
raised on financial management and compliance to procedures and protocols and despite high staff turnover as
reported by the water and sewerage company. The borrower could have done a better job at measuring progress
made with regards to behavioural change and hygiene practices among communities as well as strengthening of
staff capacities at the water utility.
c. Performance of other stakeholders:
Other stakeholders are mainly the contractors that were signed to carry out the major civil works for construction
of water supply and sewerage infrastructure as well as the water points and sanitation facilities. No major issues
were reported with regards to their performance apart from an incidental delay on the Kitwe Water Works. The
evaluator concurs with the PCR rating and rates the performance of the contractors as satisfactory (3).
6. SUMMARY OF OVERALL PROJECT PERFORMANCE
a. Overall assessment:
The project’s development objective, to improve the access to adequate water supply and sanitation services in
Kitwe, Kalulushi and Chambishi Towns, is considered very relevant with regards to the government’s goals with
regards to reducing poverty and improving health outcomes of the population of Zambia. The project design is
considered comprehensive, logic and relevant in light of the development objectives and sector strategies set by
the government of Zambia. A number of discrepancies exist between the outputs and outcomes listed in the
appraisal report and outputs and outcomes listed in the PCR. Despite some incomplete or inaccurate reporting at
the outcome and output level, the overall progress made with regards to the development objective is
considerable. Some overlap seem to exist in the PCR with regards to reporting on beneficiaries reached. The
project has performed well in terms of timeliness, resource use efficiency and creating ownership. The project
could have done better with regards to progress of implementation and ensuring financial sustainability. Overall,
the bank, borrower and contractors have performed well in this project.
The program is satisfactory with regards to the dimensions of relevance of program development objective and
program design. The project has been successful in delivering the water supply and sanitation infrastructure. With
regards to the program objectives in the remaining three of the four dimensions – viz. effectiveness, efficiency
and sustainability, the overall program performance is partly satisfactory (2.7) – due in large part to a lack of
sufficient and relevant evidence to support the assessment and achievements of the project development
objectives.
b. Design, implementation and utilization of the M&E (appreciation of the evaluator):
The monitoring and evaluation framework is considered somewhat limited in design since it did not capture
progress made in the areas of capacity building, institutional strengthening and behavioural change. For example,
while the PCR narrative rightly points at the important livelihoods opportunities which the project brought in
terms of building skills in construction and casting sanitation platforms, these software gains have not been
included in the M&E framework or captured in the list of outcomes.
6. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS
a. Lessons learned:
The evaluator fully concurs with the 2 lessons learned as listed in the PCR. These are on 1) better planning at
design phase to reduce risks during implementation and 2) stronger community involvement. On the second
lesson learned, the evaluator would note that this involvement would apart from savings in hardware costs also
benefit community acceptance of (changes in) tariffs, and could enable the adoption of a pro-poor tariff structure.
In addition to the above, the inclusion of a more comprehensive monitoring and evaluation framework, covering
software elements, should be considered as a lesson learned.
One of the activities in this project was to document the knowledge and experiences gained during the
implementation of this project which would add value and build up lessons for the design and management of
similar projects within the communities, replicated throughout the country. It would be helpful to elaborate in
more detail the specific lessons learnt from this project which were utilized in the design and implementation of
similar projects in Zambia.
The Evaluation’s opinion of each lesson learned is given in summary below:
1. VALIDATED
2. VALIDATED
The evaluator has no additional lessons learned to be suggested for inclusion.
b. Recommendations:
The evaluator fully agrees with the 4 recommendations listed in the PCR. It is a good sign that all agencies
targeted to consider these evaluations are the national agencies and government bodies. In addition to these
recommendation the evaluator would add that it is important for the water and sewerage company to keep track of
consumer satisfaction for example through the use of continued surveys or a hotline or other tools. Water demand
plans recommended should include scenario modeling of water demand based on realistic population growth
scenarios.
The Evaluation’s opinion of each recommendation is given in summary below:
1. VALIDATED
2. VALIDATED
3. VALIDATED
4. VALIDATED
The evaluator would like to add the following 2 recommendations:
1) Implementation risks in future projects with a similar design could be reduced through: (i) a thorough
institutional analysis, including capacity gap analysis with a training plan to enhance and strengthen sector
institutions and agencies both in public and private sector,
2) In any development project, a simple monitoring system needs to be included, that covers both
implementation progress as well as outcome monitoring, which establishes baseline values no later than at
appraisal stage.
7. COMMENTS ON PCR QUALITY AND TIMELINESS:
The PCR quality is rated 2.90 based on the scoring of the 9 criteria listed in Annex 1.
The extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various
sections was considered satisfactory. The extent of objectivity of PCR assessment score was considered
satisfactory. The extent of internal consistency of PCR assessment ratings; inaccuracies; inconsistencies was
considered as unsatisfactory. The outputs and outcomes listed in the PCR are not consistent with those listed in
the log-frame of the project appraisal report. The extent of identification and assessment of key factors (internal
and exogenous) and unintended effects (positive or negative) affecting design and was considered satisfactory.
The adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization was considered very
satisfactory. The extent of soundness of data generating and analysis process (including rates of returns) in
support of PCR assessment was considered satisfactory. The overall adequacy of the accessible evidence from
PCR including annexure and other data provided was considered unsatisfactory. However, not all annexes were
accessible. The extent to which lessons learned (and recommendations) are clear and based on the PCR
assessment (evidence & analysis) was considered highly satisfactory. The extent of overall clarity and
completeness of the PCR was considered satisfactory.
PCR timeliness was considered very late (1) since the PCR report was completed on 29 June 2018 and the revised
project completion date was 31 March 2015 – which was much more than 6 months after project completion.
8. SUMMARY OF THE EVALUATION
Criteria PCR PCREN Reason for disagreement/
Comments
RELEVANCE 3.5 3.5 Relevance of project development objective 4 4 Highly satisfactory.
Relevance of project design 3 3 Satisfactory.
EFFECTIVENESS 3 2
Development objective (DO) 3 2 Unsatisfactory. Although, the overall
progress made with regards to the
development objective is considerable,
the PCR only reports on 4 of the 9
outcomes anticipated in the Project
Appraisal Report. Taking the wider set
of outcomes into consideration reduces
the performance. For instance, it would
have been good to include indicators for
behavioral change with regards to
hygiene and capacity strengthening in
the log-frame, in order to capture
progress in these important soft
interventions areas. This would have
been relevant with regards to
sustainability as well.
EFFICIENCY 2.75 2.67 Timeliness 2 2 Unsatisfactory. No disagreement
Resource use efficiency 3 4 Highly satisfactory. The project shows
overall good evidence of physical
implementation versus resources used.
The median percentage physical
implementation of 100%, compared to a
total commitment rate of 98.92%. The
calculated proportion between the
median physical implementation rate
and commitment rate is 1.01 which falls
into the category of ≥1 and corresponds to a rating of highly satisfactory (4),
which is higher than the PCR rating of
satisfactory (3) for this criterion.
Cost-benefit analysis 3 UTR
Implementation progress (IP) 3 2 Unsatisfactory. The PCR narrative
reports that procurement and financial
management remained consistent with
Bank systems and procedures. However,
a number of audit reports reported on
issues found with compliance to
financial protocols and procedure, poor
financial management including
incomplete financial reporting, poor
record keeping and improper use of
accounting software. The monitoring
and evaluation framework did not
sufficiently capture progress made in the
areas of capacity building, institutional
strengthening and behavioural change.
Social and environmental safeguards
were sufficiently accounted for in the
appraisal report. The PCR reports that
Environmental and Social Management
Plan and Resettlement Policy
Framework was put in place and
monitored during project
implementation. It would have been
useful if key data from this framework
would have been included in the results
reporting, for evaluation.
SUSTAINABILITY 2.75 2.67 Financial sustainability 2 2 Unsatisfactory.
Institutional sustainability and strengthening
of capacities 3 UTR The PCR report has discussed the
various activities carried out that have
led to strengthening technical capacities
of the water and sewerage company and
government agencies as well as
community groups. While all these
elements are positive and would help
ensure institutional sustainability, these
outcomes should have been measured
and included in the M&E framework.
Simple surveys could have captured
changed capacities at the organisational
level, community level, as well as at the
individual level. This is considered
essential and without this kind of
evidence the evaluator is unable to
conclude to what extend institutional
sustainability is ensured.
Ownership and sustainability of partnerships 3 3 Satisfactory.
Environmental and social sustainability 3 3 Satisfactory.
OVERALL PROJECT COMPLETION
RATING 3 2.71
Bank performance: 3 3 Satisfactory.
Borrower performance: 3 3 Satisfactory.
Performance of other shareholders: 3 3 Satisfactory.
Overall PCR quality: - 2.90 Satisfactory.
9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR PERFORMANCE
EVALUTION REPORT, IMPACT EVALUTION, COUNTRY/SECTOR REVIEWS OR
THEMATIC EVALUATION STUDIES:
- Project is part of a series and suitable for cluster evaluation
- Project is a success story
- High priority for impact evaluation
- Performance evaluation is required to sector/country review
- High priority for thematic or special evaluation studies (Country)
- PPER is required because of incomplete validation rating
Major areas of focus for future evaluation work: (i) Contribution of this project to the design of the National Urban Water Supply and
Sanitation Program (NUWSSP) which was expected to be ready in early 2009 and current status of this program
(ii) Assessment of financial, institutional, and environmental sustainability especially of Nkana Water and
Sewerage Company (NWSC)
(iii) Revised economic and financial analysis based on relevant and reliable data
(iv) Reduction in water borne diseases, mortality rates. Health related costs of households and the labor burden for
women as a result of this project.
a) Performance evaluation is required for sector/ country review
b) Cluster evaluation (institutional support)
c) Sector evaluation (budgetary support or public finance management reforms)
Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to
facilitate a), b) and/or c).
Division Manager clearance Director signing off
Data source for validation:
Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or
email)
Documents/ Database reports
Attachment:
PCR evaluation note validation sheet of performance ratings
List of references
Appendice 1
PROJECT COMPLETION REPORT EVALUATION NOTE Validation of PCR performance ratings
PCR rating scale:
Score Description 4 Very Good – Fully achieved with no shortcomings 3 Good – Mostly achieved despite a few shortcomings 2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced 1 Poor – very limited achievement with extensive shortcomings
UTS Unable to score/rate NA Non Applicable
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
RELEVANCE Relevance of the project
development objective
(DO) during
implementation
4 4 Highly satisfactory.
Relevance of project
design (from approval to
completion)
3 3 Satisfactory.
OVERALL RELEVANCE SCORE 3.50 3.50
EFFECTIVENE
SS*
Effectiveness in delivering outcomes
Outcome 1: Percentage
of residents benefiting
from adequate water
supply services
No
score 4
Highly satisfactory. The increase in access coverage
of water supply services is commendable. It would
have been good to know if water supply was also
expanded to unplanned settlements, as was done for
sanitation.
Outcome 2: Percentage
of residents benefiting
from improved
sanitation facilities
No
score 3 Satisfactory.
Outcome 3: Incidence of
diarrhoea and dysentery
No
score 3
Satisfactory. The reduction in disease incidence can
likely be attributed to the project interventions. It
would have been good to see the numbers for all
years in order to see whether an actual positive trend
can be observed. The fact that there has not been any
major disease outbreak since 2012 is definitely a
good sign, however in order to conclude something
on attribution this should always be compared with
other urban centres that did not have an intervention.
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
Outcome 4: Increased
Metered connections
No
score 3 Satisfactory and well justified.
Outcome 5:
Unaccounted for water
(UFW)
No
score 2
Unsatisfactory. The reported progress made of 176%
should be negative (-12%) since the most recent
value is worse than the baseline value. (see
discussion in section 3d)
Outcome 6: Increase
Collection ratio
No
score 3
Satisfactory. The stated reasons for not meeting the
target are well justified.
Outcome 7: Average
daily hours of supply
No
score 2
Unsatisfactory. The reported progress made of 67%
should be negative (-33%) since the most recent
value is worse than the baseline value. The target of
24-hours water supply on average is considered not
realistic and should have been lowered at project
design phase.
Outcome 8: Quality of
both water supply and
treated waste water
No
score
2
Unsatisfactory. It would have been good to have
included a separate indicator for water treated for
consumption and treated wastewater because each
water category have different water quality
requirements and subsequent standards.
Outcome 9: Functional
Resident Development
Committees (RDCs) and
Increased women
participation from 30%
to 50%.
No
score 3 Satisfactory.
Effectiveness in delivering output
Output 1: Audit report
submission
No
score 3 Satisfactory.
Output 2: Quantity of
water produced and
distributed in Kitwe,
Kalulushi & Chambishi
(m3)
No
score 3
Satisfactory. The electricity disruption affecting
water production is considered as a valid reason
which lies outside the power of the project.
Output 3: Quantity of
waste water treated
(estimate) (m3)
No
score 3
Satisfactory. The electricity disruption affecting
wastewater treatment is considered as a valid reason
which lies outside the power of the project.
Output 4: Number of
operable public water
points
No
score 2
This output does not capture whether the water
points are in operation. Instead of “operable water
points”, this should have been “functioning water
points”, in order to monitor actual water access.
Output 5: % public
water points operated by
women
No
score 3
The reported figure on progress was not well
calculated. Since target was not reached, the value
should be below 100%.
Output 6: Number of
improved onsite
sanitation facilities
reaching vulnerable
groups
No
score 3 Very good. Satisfactory.
Output 7: % Female
headed households
No
score 2
Unsatisfactory. Narrative uses an absolute number
while the indicator is used as %. The reported figure
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
benefiting from onsite
sanitation
on progress was not well calculated. With current
value 30% against target of 43.3% the target was not
reached. The value on progress should be below
100%.
Output 8: Number of
new water supply
facilities provided and
rehabilitated in schools
and health centres
No
score 3
Satisfactory. There is unfortunately no details
provided on the no. of health centres.
Output 9: Number of
new sanitation facilities
provided and
rehabilitated in schools
and health centres
No
score 3
Satisfactory. No details are provided on the no. of
health centres.
Output 10: Provision of
solid waste management
equipment
No
score 3
In order to evaluate this, it would have been good to
get more details on the type of solid waste
management (what, how, where, etc.)
Output 11:
Rehabilitation of offices
and provision of
equipment
No
score 3
Satisfactory.
Output 12: Provision of
specialized utility
vehicle
No
score 3
Satisfactory.
Output 13: Provision of
logistic and utility
vehicles
No
score 3
Satisfactory.
Output 14: No. teachers
and health inspectors/
CDOs trained in
sanitation and hygiene
promotion and
HIV/AIDs as per PAR
No
score 3
Not clear how progress of 84% was calculated.
Based on the no. of teachers and health inspectors
combined, the progress has exceeded the target
(>100%) which is not reflected in the PCR.
Output 15: 50% RDC
women trained S & H
and management
No
score 3 Satisfactory.
Output 16: 7,000
community members
trained in S&H
HIV/Aids
No
score 3 Satisfactory.
Output 17: Reduce girls
absence from schools by
60% per month
No
score 2 Target not reached. Unsatisfactory.
Output 18: Clearing of
Dambo’s
No
score 3 Satisfactory.
Development objective (DO)
Development objective
rating
3 2 Unsatisfactory, there is incomplete or inaccurate
reporting especially at the outcome level, where only
4 of the 9 target outcomes are considered by the
PCR.
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
Beneficiaries
Beneficiary 1: 704,143
people with improved
access to water supply
in Kitwe, Kalulushi and
Chambishi towns
105% 3 Satisfactory.
Beneficiary 2: 457,693
people accessing
improved water supply
and sanitation at public
places, schools and
health centres
105% 3 Satisfactory. Overlap exists with the fourth row of
beneficiaries.
Beneficiary 3: 519,302
people with improved
access to sanitation in
Kitwe, Kalulushi and
Chambishi towns
84% 3 Satisfactory.
Beneficiary 4: 14,000
people accessing
improved water supply
and sanitation at public
places such as markets
and bus stops, schools,
and health centres
100% 3 Satisfactory. This is overlapping with the second row
of beneficiaries.
Beneficiary 5: 9,924
people benefitting
from 1,654 sanitation
facilities in Twaiteka,
Ipusukilo and Kamatipa.
110% 3 Satisfactory.
Beneficiary 6: 9,621
Community members
sensitized in WASH,
HIV and malaria
137% 3 Satisfactory.
Beneficiary 7: 625
teachers and 4 Health
inspectors
84% 3 Satisfactory.
Unanticipated outcomes (positive or negative not considered in the project logical
framework) and their level of impact on the project (high, moderate, low) Institutional
development
No
score
modera
te
Gender No
score high Project has well defined gender objectives.
Environment & climate
change
No
score high
Reduction of environmental pollution through higher
wastewater treatment capacity.
Poverty reduction No
score
modera
te
Private sector
development
No
score high
Regional integration No
score -
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
Other (specify)
EFFECTIVENESS OVERALL
SCORE
3 2
EFFICIENCY Timeliness (based on the
initial closing date) 2 2 No diasgreement
Resource used
efficiency 3 4
Highly satisfactory. The project shows overall good
evidence of physical implementation versus
resources used. The median percentage physical
implementation of 100%, compared to a total
commitment rate of 98.92%. The calculated
proportion between the median physical
implementation rate and commitment rate is 1.01
which falls into the category of ≥1 and corresponds
to a rating of highly satisfactory (4), which is higher
than the PCR rating of satisfactory (3) for this
criterion.
Cost-benefit analysis 3 UTR
Regarding calculations and assumptions made,
reference is made to an annex which is not available
to the evaluator. It is therefore difficult to assess the
validity and assumptions made for analysing the
economic costs vs benefits of this project based on
the information provided. Therefore, the evaluator is
unable to compare and evaluate the ratio of
anticipated EIRR at appraisal versus the EIRR at
project completion.
Implementation progress
(from the IPR)
3 2 Unsatisfactory. The PCR narrative reports that
Procurement and financial management remained
consistent with Bank systems and procedures.
However, a number of audit reports reported on
issues found with compliance to financial protocols
and procedure, poor financial management including
incomplete financial reporting, poor record keeping
and improper use of accounting software.
Other (specify)
OVERALL EFFICIENCY SCORE 2.75 2.67 Satisfactory.
SUSTAINABILITY Financial sustainability 2 2
Unsatisfactory.
Institutional
sustainability and
strengthening of
capacities
3 UTR
The PCR report has discussed the various activities
carried out that have led to strengthening technical
capacities of the water utilities and government
agencies as well as community groups. While all
these elements are positive and would help ensure
institutional sustainability, these outcomes should
have been measured and included in the M&E
framework. Simple surveys could have captured
changed capacities at the organisation level,
community level, as well as at the individual level.
This is considered essential and without this kind of
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
evidence the evaluator is unable to conclude to what
extend institutional sustainability is ensured.
Ownership and
sustainability of
partnerships
3 3
Satisfactory.
Environmental and
social sustainability 3 3
Satisfactory.
*The rating of the effectiveness component is obtained from the development objective (DO) rating in the latest IPR of the
project (see Guidance Note on the IPR).
The ratings for outputs and outcomes are determined based on the project’s progress towards realizing its targets, and the
overall development objective of the project (DO) is obtained by combining the ratings obtained for outputs and outcomes
following the method defined in the IPR Guidance Note. The following method is applied: Highly satisfactory (4),
Satisfactory (3), Unsatisfactory (2) and Highly unsatisfactory (1).
Criteria Sub-criteria PCR
Work
score
IDEV
review Reasons for deviation/comments
BANK
PERFORMANCE Proactive identification and
resolution of problems at different
stage of the project cycle
No
score
2 Unsatisfactory. Perhaps more attention
could have been given to timeliness of
implementation of activities and a stronger
supervision on financial administration
after issues reported in the audits.
Use of previous lessons learned
from previous operations during
design and implementation
No
score
3 Satisfactory.
Promotion of stakeholder
participation to strengthen
ownership
No
score
3 Satisfactory.
Enforcement of safeguard and
fiduciary requirements
No
score
3 Satisfactory.
Design and implementation of
Monitoring & Evaluation system
No
score
2 Unsatisfactory. The Bank could have
played a stronger role in ensuring that the
monitoring and evaluation framework
sufficiently captured progress made in the
areas of capacity building, institutional
strengthening and behavioural change.
Quality of Bank supervision (mix
of skills in supervisory teams, etc)
No
score
3 Satisfactory.
Timeliness of responses to
requests
No
score
3 Satisfactory.
OVERALL BANK PERFORMANCE
SCORE
2.71
BORROWER
PERFORMANCE
Quality of preparation and
implementation
No
score
3 Satisfactory.
Compliance with covenants,
agreements and safeguards
No
score
2 Unsatisfactory. The audit reports raised a
number of issues with regards to financial
management and compliance to procedures
and protocols
Provision of timely counterpart
funding
No
score
3 Satisfactory.
Responsiveness to supervision
recommendations
No
score
3 Satisfactory.
Measures taken to establish basis
for project sustainability
No
score
3 Satisfactory. Some issues reported with
regards to financial sustainability.
Timeliness of preparing requests No
score
3 Satisfactory.
OVERALL BORROWER PERFORMANCE
SCORE
2.83
PERFORMANCE
OF OTHER
STAKEHOLDERS
Timeliness of disbursements by
co-financiers
No
score
3 Satisfactory. No major issues were
reported with regards to their performance.
Functioning of collaborative
agreements
No
score
3 Satisfactory.
Quality of policy dialogue with
co-financiers (for PBOs only)
No
score
3 Satisfactory.
Quality of work by service
providers
No
score
3 Satisfactory.
Responsiveness to client demands No
score
3 Satisfactory.
OVERALL PERFORMANCE OF OTHER
STAKEHOLDERS
3
The overall rating is given: Very Good, Good, Fair and Poor.
(i) Very Good (HS) : 4
(ii) Good (H) : 3
(iii) Fair (US) : 2
(iv) Poor (HUS): 1
DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING AND
EVALUATION (M&E)
Criteria Sub-criteria IDEV
Score Comments
M&E DESIGN
M&E system is in place, clear,
appropriate and realistic 2
Unsatisfactory. Discrepancies exist between
outcomes and outputs listed in appraisal
report vs PCR, which is not sufficiently
explained / clarified.
Monitoring indicators and
monitoring plan were duly
approved
3 Satisfactory.
Existence of disaggregated gender
indicator 3
Satisfactory. With regards to beneficiary
reporting, the gender proportion was included
for all groups, however they were not
disaggregated for each group.
Baseline data were available or
collected during the design 3 Satisfactory.
Other, specify
OVERALL M&E DESIGN SCORE 2.75
M&E
IMPLEMENTA-
TION
The M&E function is adequately
equipped and staffed 2 Unsatisfactory.
OVERALL M&E IMPLEMENTATION SCORE 2
Criteria Sub-criteria IDEV
Score Comments
M&E
UTILIZATION
The borrower used the tracking
information for decision UTR
Not clear from the information whether this
was done.
OVERALL M&E UTILIZATION SCORE UTR
OVERALL M&E PERFORMANCE SCORE 2.38 Unsatisfactory.
PCR QUALITY EVALUATION
Criteria
PCR-
EVN (1-
4)
Comments
QUALITY OF PCR
1. Extent of quality and completeness of the PCR
evidence and analysis to substantiate the ratings of
the various sections
3 Satisfactory.
2. Extent of objectivity of PCR assessment score 3 Satisfactory.
3. Extent of internal consistency of PCR assessment
ratings; inaccuracies; inconsistencies; (in various
sections; between text and ratings; consistency of
overall rating with individual component ratings)
2 Unsatisfactory. The outputs and outcomes listed in
the PCR are not consistent with those listed in the
log-frame of the project appraisal report
4. Extent of identification and assessment of key
factors (internal and exogenous) and unintended
effects (positive or negative) affecting design and
implementation
3 Satisfactory.
5. Adequacy of treatment of safeguards, fiduciary
issues, and alignment and harmonization
3 Satisfactory.
6. Extent of soundness of data generating and
analysis process (including rates of returns) in
support of PCR assessment
3 Satisfactory.
7. Overall adequacy of the accessible evidence (from
PCR including annexure and other data provided)
3 Satisfactory. However, not all annexes were
accessible.
8. Extent to which lessons learned (and
recommendations) are clear and based on the PCR
assessment (evidence & analysis)
3 Satisfactory.
9. Extent of overall clarity and completeness of the
PCR
3 Satisfactory.
Other (specify)
PCR QUALITY SCORE 2.90
PCR compliance with guidelines (PCR/OM ; IDEV)
1. PCR Timeliness (On time = 4; Late= 1) 1 PCR report was completed more than 6 months
after project completion.
2. Extent of participation of borrower, Co-financiers
& field offices in PCR preparation
3 Satisfactory.
3. Other aspect(s) (specify)
PCR COMPLIANCE SCORE 2
*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)
References
African Development Bank Group, Policy for integrated water resources management, April 2000
African Development Bank Group, Staff Guidance on Project Completion Reporting and Rating, Quality
Assurance and Results Department (ORQR), Quality Assurance Division (ORQR.2), August 2012
Project Completion Report for Nkana Water Supply and Sanitation Project. November 2015
Government of the Republic of Zambia, National Water Policy, Ministry of Energy and Water
Development, February 2010.
Government of the Republic of Zambia. National Urban and Peri-Urban Sanitation Strategy (2015 - 2030)
Ministry of Local Government and Housing (MLGH). November 2015
Government of the Republic of Zambia, Water Act Ministry of Legal Affairs, 2006
Various other internal reports, memos, trip reports, progress reports and audit reports