PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS · 2019. 3. 15. · critical service delivery in the water and sanitation sector. ... changes which transferred all the urban water

1. BASIC INFORMATION

a. Basic project data

Project title: Urgent Water Supply and Sanitation Rehabilitation Project

Project code: P-ZW-E00-002 (initial grant) P-ZW-E00-004 (supplementary grant)

Instrument number(s): Initial Grant No: 58500155000051

Supplementary Grant No: 58500155000201

Project type: Investment Sector: Water and Sanitation

Country: Zimbabwe Environmental categorization (1-3) : 2 Processing Milestones Key Events Disbursement and Closing date

Date approved: 07.04.2011 Cancelled amount: 0 Original disbursement deadline: 31.12.2013

Date signed: 10.06.2011 Supplementary financing: Original closing date: 31.12.2013

Date of entry into force : 10.06.2011

Restructuring: N/A Revised disbursement deadline: 30.06.2015

Date effective for 1st disbursement: 02.05.2012

Extensions (specify dates): Revised closing date: 30.06.2015

Date of actual 1st : 06.06.2012

b. Financing sources

Financing source/ instrument

(MUA)

Approved amount

(MUA) :

Disbursed amount

(MUA) :

Percentage disbursed

(%):

Loan:

Grant: 29,651,000.00 29,620,497.79 99.9

Supplementary grant: 13,956,000.00 13,923,719.69 99.8

Government:

Other (ex. Co-financiers):

TOTAL : 43,607,000.00 43,544,217.48 99.9

Co-financiers and other external partners: ZimFund Donors: Australia, Denmark, Germany, Norway, Switzerland, Sweden and United Kingdom.

Execution and implementation agencies: Procurement Agent: Crown Agents Limited, United Kingdom Implementing Agent: GKW Consult GmBH, Germany

c. Responsible Bank staff

Position At approval At completion

Regional Director Ebrima Faal Kennedy Mbekeani (OIC)

Sector Director Ali Kies Mohamed El Azizi

Sector Manager Sering Jallow Osward M. Chanda

Task Manager Eskendir Demissie Eskendir Demissie

Alternate Task Manager - -

PCR Team Leader - Benson Bumbe Nkhoma

PCR Team Members Herbert Nyakutsikwa, Eliot Zvarevashe and Alex Gomani

d. Report data

PCR Date : 6th November 2015

PCR Mission Date: From: 5th October 2015 To: 16th October 2015

PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS

PCR-EN Date: 21st May 2018

Evaluator/consultant : Daniel Van Rooijen Peer Reviewer: Ram Janakiram

2. PROJECT DESCRIPTION

The project entitled “Urgent Water Supply and Sanitation Rehabilitation Project” was developed to rehabilitate critical urban water supply and sewerage infrastructure in 6 towns in Zimbabwe including its capital Harare. The project consisted of 4 major components being rehabilitation of water supply and sanitation infrastructure, hygiene promotion, institutional capacity strengthening and project management. The project agreement became effective in June 2011. The initial completion date of September 2012 was extended to December 2014 with a second phase to further expand the outcomes and impacts achieved during the first phase. The second phase was completed in June 2015, 6 months later than planned.

The main goal of the project was “to improve the health and social wellbeing of the population through equitable provision of adequate water supply and sanitation services”. This project was one of projects contributing to provide the entire population with access to adequate water supply and improved sanitation by 2030, treat all wastewater by 2020, and eliminate outbreaks of water related diseases by 2015. The project goals and supporting activities are well aligned with Zimbabwe Agenda for Sustainable Socio-Economic Transformation (Zim-Asset, 2013 – 2018) which is the Government’s blue print for development in which infrastructure and utilities are considered as one of the four pillars of the plan. The project addressed the priority areas, i.e. rehabilitation of basic infrastructure as emphasized in the National Water Policy of Zimbabwe (2012). The project purpose, during the implementation period, remained fully aligned with the Bank’s Country Brief and the Government of Zimbabwe’s water sector and development strategies and the beneficiary needs focused on improved health outcomes through sustained access to clean potable water and sanitation facilities for the safe disposal of human waste. The project is also consistent with the updated Country Brief (2014-2016) through the emphasis for critical service delivery in the water and sanitation sector. The project was also aligned with the Bank’s Ten Years Strategy 2013-2022 through its contribution to the economic infrastructure development and the Bank Strategy for Addressing Fragility and Building Resilience, a key focus being strengthening state capacity, establishing effective institutions and promotion of inclusiveness to build resilient societies. (Source: Appraisal report October 2010 and PCR of the project – November 2015).

a. Rationale and expected impacts:

The main problems being addressed by the project are: 1. The WSS systems were operating well below their installed capacities – due to prolonged periods of

inadequate operation and maintenance, lack of provisions for standby and breakdown of a single piece of equipment would disrupt the city’s water supply.

2. Declining water quality: Due to lack of water treatment chemicals and equipment, the population was being supplied with water of substandard quality.

3. Deteriorated water sanitation facilities: Majority of the urban sewerage systems have almost ceased to function, wastewater spills especially in high density areas are still common, trunk sewers have collapsed or deliberately blocked to provide water for irrigation, many booster pump stations also do not work. This has resulted in little or no wastewater being treated, and untreated raw sewage being dumped into the environment causing serious health hazards.

4. Brain drain and institutional changes: Due to the uncertain macro-economic and political

environment, most of the qualified specialists have left the country resulting in deterioration of the

service quality and poor operation and maintenance of the infrastructure. Unexpected institutional

changes which transferred all the urban water supply and sanitation services from the local

authorities to Zimbabwe National Water Authority (ZINWA) by 2005 and then to local

government control after only 4 years was carried out without adequate preparatory work which

led to both loss of more production capacity and a further deterioration of the services.

The expected impacts from the project were improved access to water supply and sanitation services for 4.15 million people living in the Municipalities of Harare, Chitungwiza, Mutare, Chegutu, Masvingo and Kwekwe and reduced incidence of cholera and other water related diseases. The number of project beneficiaries somewhat vary between the reports and details on how these numbers where estimated seems to be lacking from the available information. The Phase 1 Appraisal report stated that the project will impact nearly 4.15 million people residing in the six urban centres. The Phase 2 appraisal report states that nearly 1.9 million are expected to benefit from the project. The IPR and PCR mention 2.4 million people that have benefited from the project through improved service delivery, directly through improved water service delivery and indirectly through improved natural environment as a result of a reduction in wastewater pollution.

b. Objectives/Expected Outcomes:

The main objective envisaged in the project was to provide support for restoration and stabilization of water supply and sanitation services and improve their sustainability in six urban centres in Zimbabwe (Harare, Chitungwiza, Mutare, Masvingo, Kwekwe and Chegutu), benefiting an estimated population of 4.15 million). The expected outcomes and respective indicators of the project are:

1. Increased reliability, quality and availability of water supply in the project areas; as measured by the total water production stabilized and increased to 806,000 m3/d from 775,000 m3/d by Sept. 2012

2. Wastewater treatment capacity restored : as measured by total wastewater treatment capacity of 184,325 m3/d restored for all the urban areas from 76,325 m3/d by Sept 2012;

3. Access to municipal water in the Municipalities of Harare, Chitungwiza, Mutare, Chegutu, Masvingo and Kwekwe.as measured by number of people provided with access – 4.15 million

4. Improved operational performance and efficiency: as measured by revenue collection increased by 20%, Non revenue water decreased by about 10% from estimated 50% presently; Cholera case fatality reduced to less than 1% by Sept. 2012

c. Outputs and intended beneficiaries:

The 19 projects outputs can be divided into a hardware (first 10 outputs; being civil works) and software component (last 9 outputs; consisting of trainings, gender-based results, strategies and plans prepared). The outputs that can be listed under civil works included rehabilitation of water treatment plants, replacement of water pumps, building of water reservoirs, rehabilitation of wastewater treatment plants and pumps, construction of sewer lines and operationalisation of sewer cleaning equipment. Software outputs included training of community WASH groups with increased proportion of women in these groups, training of school WASH groups with an increased proportion of girls in these groups, total people reached through training, people

reached in the project, revenue and cost recovery strategies and investment plans prepared at the municipal level.

d. Principal activities/Components:

The principal project components and activities, as per the appraisal report are: One : Rehabilitation of water supply and sewerage infrastructure: this consisted of the following activities:

(i) Repair and/or replacement of the most critical components to enable increased and improved treatment capacity of both water supply and that of wastewater.

(ii) Water Department in each Municipality to be provided with necessary plant, equipment, and means of transportation in order to improve the quality of O&M and thus service delivery.

Two: Promotion of Improved Sanitation and Hygiene Education: this consisted of the following activities:

(i) Sensitization of the communities in the project cities, especially those living in high density areas on the importance and practice of proper hygiene and sanitation practices both at household and neighborhood levels

Three: Institutional Support : this consisted of the following activities: (i) Institutional support to upgrade the skills of the local staff responsible for O&M, and finance and

accounting functions including improved customer service, billing and collection for the WSS services in each of the urban areas.

(ii) Hands-on training during project implementation that will be geared towards reducing non-revenue water.

Four: Project Management and Engineering Services: this consisted of the following activities:

(i) Project Implementation Entity (PIE) providing project management and coordination services; and (ii) Engineering services to undertake assessment of the rehabilitation works, preparation of scope of

works, detailed design, specifications and drawings

3. PROJECT PERFORMANCE ASSESSMENT

RELEVANCE

a. Relevance of the project development objective:

The project development objective of improving urban water supply and sanitation infrastructure is considered highly relevant and has remained highly before, throughout and beyond the implementation phase. From literature it is known that the state water sanitation and hygiene services are considered key factors for preventing of and responding the risk of outbreak of water borne diseases including cholera. The scope of the project is well aligned with strategies and priorities by both the Bank (water policy) and the Government of Zimbabwe (National Water Policy, Agenda for Sustainable Socio-Economic Transformation). The evaluator concurs with the PCR scoring (4) and rates this criterion with 4 (highly satisfactory).

b. Relevance of project design (from approval to completion):

The project was designed to carry out urgent rehabilitation and construction of water supply and wastewater treatment and disposal infrastructure. However, civil works faced delays and took more time than initially planned. While the objective of environmental protection clearly was met through a reduction in pollution of the environment with untreated sewerage, it is doubted to what extend social protection measures have effectively been taken. The question whether provision of water supply services is truly equitable could not be verified based on the available information. It is not clear how water pricing systems have been implemented and to what extend these are pro-poor. The evaluator concurs with the PCR scoring (4) and rates this criterion with 4 (highly satisfactory).

EFFECTIVENESS

c. Effectiveness in delivering outputs:

Based on the available information, it can be concluded that the project delivered well on its 19 intended outputs. Civil works have been completed however with delays of up to 15 months as noted in the mid-term review report. Regarding trainings, there have been some notable successes; the number of O&M staff trained at the municipal level has exceeded the target (163% achieved) and in schools the number of WASH groups trained was nearly double the target (193%). WASH sensitisation at the community level did not reach the intended target (67%), which is considered a critical omission towards achieving the intended outcomes with regards to WASH and health and to a lesser extend the overall project objective. With regards to gender, the project has generally been effective in ensuring that school girls and women are well represented in WASH trainings. The evaluator rates this criterion with satisfactory (3) which is in line with the rating done in the PCR report.

a. Project Cost According to the appraisal report, the initial total project cost was estimated at USD 29.651 million to be financed by the Multi-Donor Trust Fund (Zim-Fund) and the works to be implemented over a period of 18 months. There was an increase in the project cost by 47 percent, bringing the final total project cost to US$43.6 million. The increase was financed through a supplementary grant (documentation of this grant needs to be obtained and reviewed). An independent value for money analysis (VFM) was carried out in Feb 2013 to determine the reasons

for the significant increase in bid values received for Harare/Chitungwiza (UWSSRP002) and Chegutu/Kwekwe/Masvingo (UWSSRP003) and whether these tenders would provide value for money given the large variance – between 28 percent and 190 percent of the outcome of the bids compared to the appraisal estimates. The reasons found were: (i) inadequate bid specifications; (ii) time pressures placed on the implementing entity to obtain “call for tenders” as soon as possible; (iii) some instances of price gouging, (iv) significant errors in the engineering estimates; (v) scarcity of comparative and comparable data on bid and market prices; (vi) pricing in credit and political risk and all risks placed on the bidders. (vii) Existence of sanctions on an identified list of individuals in Zimbabwe placed constraints and added a level of complexity to the bidding process. (Source, Value for money analysis, final report, 20 February 2013). The project completion was delayed by 15 months, due to : (i) delays in project establishment, which resulted in time pressures to speed up implementation compromising the quality of work (specifically with respect to the specifications of the work to be carried out and the reliability of the estimates); (ii) complexity of governance and implementing structures – there were unusually large number of parties involved in both the governance and implementation (procurement and supervision) aspects of the project. (Source, Value for money analysis, final report, 20 February 2013).

b. Implementation arrangements, conditions and covenants and related technical assistance

Based on the information available, staff operating the rehabilitated water supply and sanitation systems has received proper technical trainings in system operation and maintenance as well as new equipment. It remains unclear whether institutions have been sufficiently set up to run the systems autonomously; without further financial support. It is not clear whether further improvements aiming at increasing cost-recovery, for example modernisation of the water metering system will effectively take place beyond the project lifeline.

c. Monitoring and Evaluation Design, Implementation & Utilization (Evaluator assessment)

The IPR mentions that an overall Project M&E Plan was in development, however it is not clear to what extent this was put to use as intended. The results framework presented in the appraisal report follows a clear logic, is overall clear but some of the indicators appeared to be difficult to measure (for example non-revenue water, as noted earlier in this PCR-EN). Not much emphasis was placed in the design of a robust M & E system in the appraisal report. It focused more on: (i) production of “monthly progress reports on status of the various project activities, financial and procurement status, asset management, community participation, safeguard and risks and mitigation measures” (appraisal report, section 4.6.1) and (ii) measuring the outputs rather than the outcomes. Most of the aide memoires from supervision missions and progress reports reviewed placed more importance on the achievement of outputs.Other issues (such as Safeguards, Fiduciary) It is unclear from the information whether end-users of the rehabilitated and constructed water supply systems have been fully consulted and sensitized with regards to ensuring their willingness to pay for improved water services. The narrative provided in the PCR for this criterion does not match very well with the required assessment of environmental and social sustainability. It is unclear to what extend the Environmental and Social Management Plan developed by each municipality as part of the project was actively implemented to ensure that social and environmental interests were taken into account. The evaluator wished to rate this criterion with satisfactory (3) which is lower than the score given in the PCR (4).

4.Evaluation of performance

a. Relevance of project objectives The project development objective of improving urban water supply and sanitation infrastructure is considered highly relevant and has remained highly before, throughout and beyond the implementation phase. From literature

it is known that the state water sanitation and hygiene services are considered key factors for preventing of and responding the risk of outbreak of water borne diseases including cholera. The scope of the project is well aligned with strategies and priorities by both the Bank (water policy) and the Government of Zimbabwe (National Water Policy, Agenda for Sustainable Socio-Economic Transformation).

The evaluator concurs with the PCR scoring (4) and rates this criterion with 4 (highly satisfactory).

b. Effectiveness in delivering outputs Based on the available information, it can be concluded that the project delivered well on its 19 intended outputs Civil works have been completed however with delays of up to 15 months as noted in the mid-term review report. Regarding trainings, there have been some notable successes; the number of O&M staff trained at the municipal level has exceeded the target (163% achieved) and in schools the number of WASH groups trained was nearly double the target (193%). WASH sensitisation at the community level did not reach the intended target (67%), which is considered a critical omission towards achieving the intended outcomes with regards to WASH and health and to a lesser extend the overall project objective. With regards to gender, the project has generally been effective in ensuring that school girls and women are well represented in WASH trainings.

The evaluator rates this criterion with satisfactory (3) which is in line with the rating done in the PCR report.

c. Effectiveness in achieving outcomes

The latest IPR published in August 2013 reported satisfactory progress, anticipating that activities would achieve the intended key-deliverables. The PCR shows quite some variance between outcomes with regards to final progress reporting, which is explained reasonably well in the narrative sections. In some areas, project delivery has not been fully comprehensive. Water production has significantly increased, however the reported physical water distribution issues have likely affected improvements in household level access to water. The appraisal report has only included access to water at the impact level. It is not clear from the information why household level access to water services was not included as an indicator in the project. Despite wastewater treatment capacity have been restored as intended, however actual wastewater treatment was lower due to apparent sewer challenges. Possibly the length of broken sewer lines was much higher than expected and the 5 km rehabilitated sewerage was not sufficient to ensure optimal wastewater flows into the treatment plants. This could affect human health risks and the quality of the natural environment. Additional issues worth noting are the low capacity and willingness to pay for improved water services, which has affected revenue collection. The PCR rightly acknowledges that this is caused by the socio-economic situation at the national level. The evaluator wonders to what extent sensitisation campaigns have effectively been carried out in order to attempt to increase household willingness to pay for improved service water delivery. The inclusion of NRW as an outcome level indicator is generally much welcomed, however the understandable lack of reliable measurements does not allow for progress reporting. The evaluator rates this criterion with satisfactory (3) which is in line with the rating done in the PCR report.

d. Efficiency in achieving outputs and outcomes

Timeliness: The total project, being phase 1 and phase 2 combined, faced significant delays in the area of constructing the water supply infrastructure. This resulted in the project being completed 6 month later than planned. The time needed to carry out the civil works may have been underestimated in the project design phase. The community level WASH promotions were reportedly constrained by time, which could possibly have been avoided with allowing a longer time period when the project was designed. The evaluator concurs with the PCR rating and would rate this criterion with unsatisfactory (2).

Resource use efficiency: All resources (99.9%) that were allocated as part of the grant and supplementary grant

have been used. The evaluator therefore rates this criterion with satisfactory (3). Cost-benefit analysis: The appraisal report provides an explanation of how the EIRR of 22% was arrived at. The PCR notes an assumed hurdle rate of 10%. It is not clear on the basis of what evidence this assumption was made. It is not clear from the available information how the EIRR at project completion was calculated. Despite the lack of clarity regarding evidence and method of calculation, the evaluator would regard this criterion as satisfactory (3).

e. Project Development outcome (PO):

The project appraisal report lists the indicators to be used to measure progress toward reaching the development outcomes and objectives. These indicators include coverage with safe drinking water and adequate sanitation and incidence of water related diseases. The available information has not included actual indicator-level data that would support a conclusion that the project has achieved the development objective. Based on global WASH literature it is however likely that the WASH interventions carried out in this project have contributed to some extend towards reaching the development objectives. The evaluator rates this criterion with satisfactory (3) which is in line with the rating done in the PCR report.

e. Risk to sustained achievement of project outcomes

The appraisal report identified the risk of failure to operate, and maintain the rehabilitated facilities and the mitigation measure was the provision of training of staff and essential O&M supplies; and making more resources available to continue with the rehabilitation of the water infrastructure. (Source – results matrix, appraisal report – there is no explicit analysis of risks in the appraisal report). The PCR has also identified the issues of non-revenue water affecting sustainability; revenues from water and sanitation not delinked from general revenues of the Municipal councils and the need for reserve funds for capital replacement in years 10 and 20 after implementation. Given the less than expected cost recovery from the beneficiaries for water supply and sanitation services, and revenues not yet ring fenced, it is unlikely that the risks identified during the appraisal and PCR will be reduced. The risk therefore, has been assessed as substantial (4).

f. Additional outcomes/Impacts (positive and negative, not captured in the log-frame):

Data supporting the claimed additional outcomes is limited or absent. Wastewater use for farming as stated in the PCR is indeed a positive additional outcome, however there are health risks associated to the unprotected use of untreated wastewater which could negatively affect development outcomes. The project found that there is a low willingness and capacity to pay for improved water services which could have been included as a negative social outcome possibly raising concerns on the extent to which equity and pro-poor water supply has been addressed Gender: The appraisal report and the PCR have highlighted the importance of gender equality and the important and vital role women play in ensuring the availability of water for daily household needs, taking care of children, etc. Only anecdotal evidence has been provided of the benefits from the project such as savings in time to obtain water, no sewage flowing into their living areas, etc. (Source, Appraisal report, section 3.4.4 – Gender; PCR, section 6 – Gender equality). It is unclear whether periodic surveys and randomized control trial methods were used to assess the impacts of the project.

g. Performance of Borrower: The borrower has performed reasonably well during the project. The leading ministry could perhaps have provided more engagement in managing the project. The municipalities could have been involved more during project implementation which may have affected sustainability of the project. From the available information, it is not clear to what extent recommendations made by the Bank have been effectively applied. Borrower’s performance is rated satisfactory (3).

h. Bank Performance: The Bank has performed well in supervising this project from start to end. A sufficient number of missions were carried out and the mid-term review carried out during March-April 2013 demonstrated a solid review with detailed recommendations for the borrower. Protocols and procedures in place at the Bank have been fully adhered to. The Bank demonstrated innovation with regards to design and implementation of the project despite issues to channel resources directly through the government system. Given the project’s success despite the challenging implementation, Bank’s performance is rated highly satisfactory (4). Comment:” Despite the regular supervision missions (unclear if specialists in M and E, change management were included) – little progress was made to ensure timely completion of the project, establishment of a robust M and E system, and achievement of institution building components – recommend that the rating be changed to Satisfactory (3).

i. Overall assessment:

The project has performed well despite a challenging implementation arrangement and despite a complex socio-economic situation. The core area of intervention of rehabilitation and construction of water supply and sanitation infrastructure is considered highly relevant with a high likeliness that the project has contributed to improving health outcomes for a significant fraction of the urban population of Zimbabwe. While the project is considered overall efficient and the issues with regards to delays in implementation are understandable and to a large extent justified. Some concerns have been noted in the PCR-EN with regards to the social and financial sustainability of the project. . Beneficiaries:

The evaluator poses some concerns with regards to the justification and calculation of the total number of beneficiaries. It is not clear how the 2.4 million people was calculated; whether this is the combined total population of all 6 towns, or a fraction. There is no disaggregation of number of people benefiting from improved water supply, improved sanitation and a cleaner natural environment. The calculation of the number of people trained seems to have been well calculated and includes sex-disaggregated data.

g. Unanticipated additional outcomes (positive or negative, not taken into consideration in the project logical framework):

Data supporting the claimed additional outcomes is limited or absent. Wastewater use for farming as stated in the PCR is indeed a positive additional outcome, however there are health risks associated to the unprotected use of untreated wastewater which could negatively affect development outcomes. The project found that there is a low willingness and capacity to pay for improved water services which could have been included as a negative social outcome possibly raising concerns on the extent to which equity and pro-poor water supply has been addressed.

EFFICIENCY

h. Timeliness:

The total project, being phase 1 and phase 2 combined, faced significant delays in the area of constructing the water supply infrastructure. This resulted in the project being completed 6 month later than planned. The time needed to carry out the civil works may have been underestimated in the project design phase. The community level WASH promotions were reportedly constrained by time, which could possibly have been avoided with allowing a longer time period when the project was designed. The evaluator concurs with the PCR rating and would rate this criterion with unsatisfactory (2).

i. Resource use efficiency:

All resources (99.9%) that were allocated as part of the grant and supplementary grant have been used. The evaluator therefore rates this criterion with satisfactory (3).

j. Cost-benefit analysis: The appraisal report provides an explanation of how the EIRR of 22% was arrived at. The PCR notes an assumed hurdle rate of 10%. It is not clear on the basis of what evidence this assumption was made. It is not clear from the available information how the EIRR at project completion was calculated. Despite the lack of clarity regarding evidence and method of calculation, the evaluator would regard this criterion as satisfactory (3). k. Implementation progress: The project faced a delay of about 12 months caused by the complex implementation arrangement, unforeseen changes in implementation requirements from the municipalities as well as an additional money-for value assessment that had to be carried out during the bidding process. At the national level, the new cabinet formation was also reported to have delayed formal processes of signing the contract with the government. Poor quality of some civil works could perhaps have been prevented through more intensive supervision of the contractor. The above reasons have been clearly outlined in the IPR report and have been further acknowledged in the PCR. The same reasons contributing to the overall delays form a reasonable justification. The evaluator wishes to rate this criterion with satisfactory (3) which is in line with PCR rating.

SUSTAINABILITY

l. Financial sustainability:

From the information made available to the evaluator, details were lacking on the type of tariff structures proposed or put in place for the new and rehabilitated water supply schemes. It is unclear to what extend different service levels have been tailored to the findings found from the household capacity and willingness to pay studies carried out as part of the design phase. Based on the information available, revenues derived from water consumption do not fully flow back to sustain budgets needed for further investments as well as operation and maintenance which questions whether the water supply systems are sustainable in the long term. The uncertainty of subsequent financial sustainability of the water supply systems was noted as one of the main project risks in the IPR and was rightly questioned in the PCR. It is not clear to what extend recommendations made have been followed in the final phase of the project. Based on the above, the evaluator rates this criterion with 2 (unsatisfactory) which is lower than the PCR rating of satisfactory (3). m. Institutional sustainability and strengthening of capacities:

Based on the information available, staff operating the rehabilitated water supply and sanitation systems have received proper technical trainings in system operation and maintenance as well as new equipment. It remains unclear whether institutions have been sufficiently set up to run the systems autonomously; without further financial support. It is not clear whether further improvements aiming at increasing cost-recovery, for example modernisation of the water metering system will effectively take place beyond the project lifeline. This criterion is rated satisfactory (3) in line with the respective PCR score. n. Ownership and sustainability of partnerships:

The available information suggests that the most important government counterpart in the project, the city councils, have demonstrated ownership in running the rehabilitated water supply and sanitation infrastructure. It

is unclear to what extend the interests of other parties, for example civil society groups have been considered during policy dialogue and overall consultation. The PCR claims that partnerships with farmers and other river water users will be enhanced however, it is not apparent whether the interests and opinions of these stakeholders have been recorded, let alone taken into account. This criterion is rated as satisfactory (2) which is in line with the respective PCR score. o. Environmental and social sustainability:

It is unclear from the information whether end-users of the rehabilitated and constructed water supply systems have been fully consulted and sensitized with regards to ensuring their willingness to pay for improved water services. The narrative provided in the PCR for this criterion does not match very well with the required assessment of environmental and social sustainability. It is unclear to what extend the Environmental and Social Management Plan developed by each municipality as part of the project was actively implemented to ensure that social and environmental interests were taken into account. The evaluator wished to rate this criterion with satisfactory (3) which is lower than the score given in the PCR (4).

4. PERFORMANCE OF STAKEHOLDERS

a. Bank performance: The Bank has performed well in supervising this project from start to end. A sufficient number of missions were carried out and the mid-term review carried out during March-April 2013 demonstrated a solid review with detailed recommendations for the borrower. Protocols and procedures in place at the Bank have been fully adhered to. The Bank demonstrated innovation with regards to design and implementation of the project despite issues to channel resources directly through the government system. Given the project’s success despite the challenging implementation, Bank’s performance is rated highly satisfactory (4).

b. Borrower performance:

The borrower has performed reasonably well during the project. The leading ministry could perhaps have provided more engagement in managing the project. The municipalities could have been involved more during project implementation which may have effected sustainability of the project. From the available information, it is not clear to what extend recommendations made by the Bank have been effectively applied. Borrower’s performance is rated satisfactory (3). c. Performance of other stakeholders:

The combined performance of other stakeholders was satisfactory despite a few issues. The physical distance of the procurement agent (UK-based) made it difficult to assess developments on the ground. The size of the civil works seem to have been stressed the capacity of the main contractor compromising the quality of the delivered works in some cases. Perhaps during the bidding process, the capacity of the contractor to carry out the work could have been better assessed. The overall performance of other stakeholders is rated as satisfactory (3).

5. SUMMARY OF OVERALL PROJECT PERFORMANCE

a. Overall assessment:

The project has performed well despite a challenging implementation arrangement and despite a complex socio-economic situation. The core area of intervention of rehabilitation and construction of water supply and sanitation

infrastructure is considered highly relevant with a high likeliness that the project has contributed to improving health outcomes for a significant fraction of the urban population of Zimbabwe. While the project is considered overall efficient and the issues with regards to delays in implementation are understandable and to a large extend justified. Some concerns have been noted in the PCR-EN with regards to the social and financial sustainability of the project.

b. Design, implementation and utilization of the M&E (appreciation of the evaluator):

The IPR mentions that an overall Project M&E Plan was in development, however it is not clear to what extend this was put to use as intended. The results framework presented in the appraisal report follows a clear logic, is overall clear but some of the indicators appeared to be difficult to measure (for example non-revenue water, as noted earlier in this PCR-EN).

6. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS

a. Lessons learned:

The evaluator generally agrees with the main lessons learned from the project, as listed in the PCR. These lessons mainly relate to the efficiency and sustainability components of the evaluation and some are combined as they point to the same challenges. The final lessons learned on capacity development of local contractors is regarded as outside the scope of the project. Implementation arrangements & ownership

The role and level of involvement of municipal councils needs to be critically assessed in the designed phase. Inclusion of mechanisms to ensure their commitment could possibly have ensure a stronger (sense of) ownership by the local councils. Roles of various institutions need to be very clearly defined and decision making simplified and accountability systems strengthened. Design of the project to address specific challenges in the respective cities

The project also could not fully address specific local challenges due to its one size fits all approach. The question is whether or not these local differences have not been observed during field/scoping studies carried during the design phase and what have been done with these findings. A more flexible design could have catered for addressing site specific challenges.

Strong supervision required from the Engineer (Consultant)

During the scoping studies a stronger focus could have been out on assessment of capacities of local contractors. This may have avoided to some extend the stated quality issues of works carried out by the main contractor and some sub-contracted companies,. The Evaluation’s opinion of each lesson is given in summary below:

1. VALIDATED

2. VALIDATED

3. VALIDATED

4. VALIDATED

5. NOT VALIDATED, NOT RELEVANT

b. Recommendations:

The evaluator generally agrees with the key recommendations from the project, as listed in the PCR. In particular the implementation arrangement and effectiveness of the role of the procurement agent should be carefully considered. In addition, the capacities of contractors should be more carefully assessed during project design phase. Regarding timeliness of the project implementation, perhaps the time needed to carry out a project with a large component of civil works was underestimated and could not meet the expectations following the urgency to respond to humanitarian needs to reduce to risk of the outbreak of water borne diseases. Robust monitoring and evaluation system to track measureable indicators of progress, outputs, outcomes and impacts of water supply and sanitation projects being prepared should be put in place and adequately funded for obtaining the full benefits and value for money. Appropriate mechanisms and programs should be put in place to transfer management, technical and operational skills to the national staff following the completion of services by the international specialists.

The Evaluation’s opinion of each recommendation is given in summary below:

1. NOT VALIDATED, TOO PROJECT SPECIFIC

2. VALIDATED

3. VALIDATED

4. NOT VALIDATED, REFORMULATED AS BELOW (second recommendation)

5. NOT VALIDATED, THIS IS NOT A RECOMMENDATION

The evaluator would like to add the following two recommendations:

- In future projects that include procurement agents, the implementation arrangement and effectiveness of

the role of the procurement agent should be carefully considered.

- In projects with a construction component, the capacities of available contractors should be assessed very

carefully during project design phase.

7. COMMENTS ON PCR QUALITY AND TIMELINESS

The overall PCR rating is based on all or part of the criteria presented in the annexe and other: The quality

of the PCR is rated as highly satisfactory (4), satisfactory (3), unsatisfactory (2), and highly unsatisfactory

(1). Le timeliness of the PCR is rated as on time (4) or late (1). The participation of the Borrower, co-

financier, and the bank’s external office(s) are rated as follows: Very Good (4), Good (3), Fair (2), Poor (1).

The quality of the PCR is rated as satisfactory (2.89). Generally the narrative and numeric scoring are consistent with each other and sufficient explanation has been provided.

The extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various sections was considered satisfactory. The extent of objectivity of PCR assessment score was considered satisfactory. The extent of internal consistency of PCR assessment ratings; inaccuracies; inconsistencies was considered satisfactory. The extent of identification and assessment of key factors (internal and exogenous) and unintended effects (positive or negative) affecting design and was considered satisfactory. The adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization was considered satisfactory. The extent of soundness of data generating and analysis process (including rates of returns) in support of PCR assessment was considered unsatisfactory. It is not clear how some beneficiary numbers were calculated or on the

basis of what assumptions these numbers were estimated. The overall adequacy of the accessible evidence (from PCR including annexure and other data provided was considered unsatisfactory. It is not clear how some numbers were calculated or on the basis of what assumptions these numbers were estimated. The extent to which lessons learned (and recommendations) are clear and based on the PCR assessment (evidence & analysis) was considered highly satisfactory. The extent of overall clarity and completeness of the PCR was considered satisfactory.

The timeliness of the PCR is rated as on time (4), since it was produced within 6 months after project

closure.

8. SUMMARY OF THE EVALUATION This is a summary of both the PCR and IDEV ratings with justification for deviations/comments. Appropriate section of the PCR

Evaluation should be indicated in the last column in order to avoid detailed comments. The evaluator must provide a reasonable

explanation for each criterion the PCR rating is not validated by IDEV. Consequently, the overall rating of the project could be

“equally satisfactory”.

Criteria PCR PCREN Reason for disagreement/ Comments

RELEVANCE

Relevance of project development objective 4 4

Relevance of project design 4 4

EFFECTIVENESS

Development objective (DO) 3 3 In the PCR a discrepancy exists between the specific rating for this criterion in the narrative (3) and in section V: overall rating (4). Despite the lack of solid indicator data presented in the PCR and other reports, it is likely that the WASH interventions carried out in this project have contributed to reaching the development objectives.

EFFICIENCY

Timeliness 2 2

Resource use efficiency 3 3 In the PCR a discrepancy exists between the specific rating for this criterion in the narrative (4) and in section V: overall rating (3).

Cost-benefit analysis 3 3 In the PCR a discrepancy exists between the specific rating for this criterion in the narrative (4) and in section V: overall rating (3).

Implementation progress (IP) 3 3

SUSTAINABILITY

Financial sustainability 3 2 The uncertainty of financial sustainability of the water supply systems was noted in the IPR as one of the main project risks and financial sustainability was rightly questioned in the PCR. It is not clear to what extend recommendations made in the IPR have been followed in the final phase of the project.

Institutional sustainability and strengthening of capacities

3 3

Ownership and sustainability of partnerships 2 2

Environmental and social sustainability 4 3 The narrative provided in the PCR for this criterion does not match very well with the required assessment of environmental and social sustainability.

OVERALL PROJECT COMPLETION

RATING

3 3

Bank performance: 4 3.56

Borrower performance: 2 3 The borrower has performed reasonably well during the project. From the available information, it is not clear to what extend recommendations made by the Bank have been effectively applied.

Performance of other shareholders: 3 3

Overall PCR quality: 3 Satisfactory.

9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR PERFORMANCE

EVALUTION REPORT, IMPACT EVALUTION, COUNTRY/SECTOR REVIEWS OR

THEMATIC EVALUATION STUDIES:

- Project is part of a series and suitable for cluster evaluation

- Project is a success story

- High priority for impact evaluation

- Performance evaluation is required to sector/country review

- High priority for thematic or special evaluation studies (Country)

- PPER is required because of incomplete validation rating

Major areas of focus for future evaluation work:

a) Performance evaluation is required for sector/ country review

b) Cluster evaluation (institutional support)

c) Sector evaluation (budgetary support or public finance management reforms)

Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to

facilitate a), b) and/or c).

Division Manager clearance Director signing off

Data source for validation:

Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or email)

Documents/ Database reports

Attachment:

PCR evaluation note validation sheet of performance ratings

List of references

Appendice 1

PROJECT COMPLETION REPORT EVALUATION NOTE

Validation of PCR performance ratings

PCR rating scale:

Score Description

4 Very Good – Fully achieved with no shortcomings

3 Good – Mostly achieved despite a few shortcomings

2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced

1 Poor – very limited achievement with extensive shortcomings

UTS Unable to score/rate

NA Non Applicable

Criteria Sub-criteria

PCR work score

IDEV revie

w

Reasons for deviation/comments

RELEVANCE Relevance of the project development objective (DO) during implementation

4 4

No further comments.

Relevance of project design (from approval to completion)

4 4

No further comments.

OVERALL RELEVANCE SCORE 4 4

EFFECTIVENESS*

Effectiveness in delivering outcomes

Outcome 1: Access to municipal water in targeted cities/towns

Not

rated 3

The appraisal report (phase 1) only included access to water at the impact level. It is not clear from the information why household level access to water services was not included as an indicator in the project.

Outcome 2: Reliable and stable water produced

Not

rated 3

Water production has significantly increased, however the reported physical water distribution issues have likely affected improvements in household level access to water.

Outcome 3: Quality of water - % of samples meeting

Not

rated 3

The PCR mentioned that water quality monitoring has constantly broken down, which poses sustainability concerns and


PCR work score

IDEV revie

w


standards raises questions with regards to the selection process and possibly how the equipment is operated and maintained.

Outcome 4: Waste water treated

Not

rated 3

Despite wastewater treatment capacity have been restored as intended, however actual wastewater treatment was lower due to apparent sewer challenges. Possibly the length of broken sewer lines was much higher than expected and the 5 km rehabilitated sewerage was not sufficient to ensure optimal wastewater flows into the treatment plants. This could affect human health risks and the quality of the natural environment.

Outcome 5: No. of people to benefit from improved service

Not

rated 2

Not clear how performance towards achieving this outcome was measured. The reported low capacity and willingness to pay would put the success (100% achieved) in doubt.

Outcome 6: Reduction in NRW

Not

rated UTS

The understandable lack of reliable measurements of NRW does not allow for progress reporting.

Outcome 7: Revenue collection Efficiency

Not

rated 3

It is clear that the low capacity and willingness to pay for improved water services has affected revenue collection.

Effectiveness in delivering output

Output 1: Number of water treatment works rehabilitated

Not

rated 3 Achieved. Delays before completion have

been well justified.

Output 2: Number of clean water pump units replaced (to distribution system)

Not

rated 3 No further comments.

Output 3: Number of raw water pumping units replaced

Not


Output 4: Number of water reservoirs built

Not

rated 3 No details found on technical specification

of these reservoirs.


PCR work score

IDEV revie

w


Output 5: Number of waste water treatment works rehabilitated /replaced

Not


Output 6: Number of waste water effluent pumping units rehabilitated/replaced

Not


Output 7: Number of raw sewage pumping units rehabilitated/replaced

Not


Output 8: Km of rehabilitated sewer line

Not

rated 3 Based on the information it seems that more

sewer lines could have been rehabilitated in order to maximize inflows into the wastewater treatment plants.

Output 9: km of new trunk sewer constructed

Not


Output 10: Number of Sewer cleaning equipment handed over to municipalities

Not


Output 11: Number of community WASH groups trained

Not


Output 12: Proportion of women in Community WASH Groups

Not



PCR work score

IDEV revie

w


Output 13: Number of school WASH groups trained

Not

rated 4 The number of WASH groups trained in

schools was nearly double the target (193%).

Output 14: Proportion of Girls in School WASH Groups

Not

rated 3 Effective implementation with regards to

ensuring that school girls are well represented in WASH trainings.

Output 15: Number of people trained

Not

rated 4 Exceeded the target.

Output 16: Number of people reached through the project

Not

rated 2 The justification for stating how 2.4 M

people have benefitted is doubtful, given the issues mentioned in relation to household water access and lower wastewater treatment levels than expected.

Output 17: Number of Municipal Operations & Maintenance staff trained

Not

rated 4 The number of O&M staff trained at the

municipal level has exceeded the target (163% achieved)

Output 18: Number of municipal revenue and cost recovery strategies prepared

Not


Output 19: Number of Municipal medium and long term investment plans prepared.

Not


Development objective (DO)

Development objective rating

3 3 Despite the lack of solid indicator-level data presented in the PCR and other reports, it is likely that the WASH interventions carried out in this project have contributed to reaching the development objectives.


PCR work score

IDEV revie

w


Beneficiaries

Beneficiary1: 2.4 million people in the six cities

Not

rated

2 The evaluator poses concerns with regards to the justification and calculation of the total number of beneficiaries.

-

Unanticipated outcomes (positive or negative not considered in the project

logical framework) and their level of impact on the project (high, moderate,

low)

Institutional development

Not

rated UTR No significant outcomes in addition to the

ones included in the log frame.

Gender Not

rated UTR No significant outcomes in addition to the

ones included in the log frame.

Environment & climate change

Not

rated high The reduction of pollution of the natural

environment with untreated municipal wastewater is worth noting here.

Poverty reduction Not

rated UTR No evidence that poverty was reduced.

Private sector development

Not

rated moder

ate There has been some opportunities for entrepreneurs to supply materials for the maintaining the water supply systems.

Regional integration Not

rated UTR No related information found.

Other (specify)

EFFECTIVENESS OVERALL

SCORE

Not

rated 3

EFFICIENCY Timeliness (based on the initial closing date)

2 2 No comments.

Resource used efficiency

3 3 In the PCR a discrepancy exists between the specific rating for this criterion in the narrative (4) and in section V: overall rating (3).

Cost-benefit analysis


Implementation progress (from the IPR)

3 3 No comments.

Other (specify)


PCR work score

IDEV revie

w


OVERALL EFFICIENCY SCORE

SUSTAINABILITY

Financial sustainability

3 2 The uncertainty of financial sustainability of the water supply systems was noted in the IPR as one of the main project risks and financial sustainability was rightly questioned in the PCR. It is not clear to what extend recommendations made in the IPR have been followed in the final phase of the project.

Institutional sustainability and strengthening of capacities

3 3 No comments.

Ownership and sustainability of partnerships


Environmental and social sustainability

4 3 In the PCR a discrepancy exists between the specific rating for this criterion in the narrative (4) and in section V: overall rating (3). The narrative provided in the PCR for this criterion does not match very well with the required assessment of environmental and social sustainability.

*The rating of the effectiveness component is obtained from the development objective (DO) rating in the latest IPR of the project (see Guidance Note on the IPR). The ratings for outputs and outcomes are determined based on the project’s progress towards realizing its targets, and the overall development objective of the project (DO) is obtained by combining the ratings obtained for outputs and outcomes following the method defined in the IPR Guidance Note. The following method is applied: Highly satisfactory (4), Satisfactory (3), Unsatisfactory (2) and Highly unsatisfactory (1).


PCR Work score

IDEV revie

w


BANK PERFORMANCE

Proactive identification and resolution of problems at different stage of the project cycle

Not

rated 4

Use of previous lessons learned from previous

Not

rated 4

operations during design and implementation

Promotion of stakeholder participation to strengthen ownership

Not

rated 3

Enforcement of safeguard and fiduciary requirements

Not

rated 4

Design and implementation of Monitoring & Evaluation system

Not

rated 2

Quality of Bank supervision (mix of skills in supervisory teams, etc)

Not

rated 4

Timeliness of responses to requests

Not

rated 4

OVERALL BANK PERFORMANCE

SCORE

4 3.6

BORROWER PERFORMANCE

Quality of preparation and implementation

Not

rated 3

Compliance with covenants, agreements and safeguards

Not

rated 3

Provision of timely counterpart funding

Not

rated 3

Responsiveness to supervision recommendations

Not

rated 3

Measures taken to establish basis for project sustainability

Not

rated 2

Timeliness of preparing requests

Not

rated 4

OVERALL BORROWER PERFORMANCE

SCORE

3.0

PERFORMANCE OF OTHER STAKEHOLDERS

Timeliness of disbursements by co-financiers

Not

rated UTR

Functioning of collaborative agreements

Not

rated UTR

Quality of policy dialogue with co-financiers (for PBOs only)

Not

rated UTR

Quality of work by service providers

Not

rated 2

Responsiveness to client demands

Not

rated 2

OVERALL PERFORMANCE OF OTHER 2

STAKEHOLDERS

The overall rating is given: Very Good, Good, Fair and Poor.

(i) Very Good (HS) : 4 (ii) Good (H) : 3 (iii) Fair (US) : 2 (iv) Poor (HUS): 1

DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING AND

EVALUATION (M&E)

Criteria Sub-criteria IDEV Score

Comments

M&E DESIGN

M&E system is in place, clear, appropriate and realistic

2

Monitoring indicators and monitoring plan were duly approved

2

Existence of disaggregated gender indicator

3 The project collected gender disaggregated data.

Baseline data were available or collected during the design

2

Other, specify

OVERALL M&E DESIGN SCORE 2.25

M&E IMPLEMENTA-TION

The M&E function is adequately equipped and staffed

2 The M&E function seem to have been substantially lacking.

OVERALL M&E IMPLEMENTATION

SCORE 2

M&E UTILIZATION

The borrower used the tracking information for decision

UTR Unable to evaluate.

OVERALL M&E UTILIZATION SCORE UTR

OVERALL M&E PERFORMANCE

SCORE 2

PCR QUALITY EVALUATION

Criteria

PCR-

EVN (1-

4)

Comments

QUALITY OF PCR

1. Extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various sections

3

2. Extent of objectivity of PCR assessment score

3

3. Extent of internal consistency of PCR assessment ratings; inaccuracies; inconsistencies; (in various sections; between text and ratings; consistency of overall rating with individual component ratings)

3

4. Extent of identification and assessment of key factors (internal and exogenous) and unintended effects (positive or negative) affecting design and implementation

3

5. Adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization

3

6. Extent of soundness of data generating and analysis process (including rates of returns) in support of PCR assessment

2 Not clear how some beneficiary numbers were calculated or on the basis of what assumptions these numbers were estimated.

7. Overall adequacy of the accessible evidence (from PCR including annexure and other data provided)

2 As above.

8. Extent to which lessons learned (and recommendations) are clear and based on the PCR assessment (evidence & analysis)

4

9. Extent of overall clarity and completeness of the PCR

3

Other (specify)

PCR QUALITY SCORE 2.89

PCR compliance with guidelines (PCR/OM ; IDEV)

1. PCR Timeliness (On time = 4; Late= 1) 4 Completed within 6 months from project closure.

2. Extent of participation of borrower, Co-financiers & field offices in PCR preparation

3

3. Other aspect(s) (specify)

PCR COMPLIANCE SCORE 3.50

*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)

References

African Development Bank Group, Project Appraisal Report, Urgent Water Supply and Sanitation Rehabilitation Project, October 2010

African Development Bank Group, Project Appraisal Report Phase 2, Urgent Water Supply and Sanitation Rehabilitation Project Phase 2, September 2013

African Development Bank Group, Policy for integrated water resources management, April 2000

Government of Zimbabwe. 2012 Zimbabwe National Water Policy Ministry of Water resources Development and Management

Government of Zimbabwe. Zimbabwe Agenda for Sustainable Socio-Economic Transformation (ZimAsset) “Towards an Empowered Society and a Growing Economy” October 2013 – December 2018

Project Completion Report for: Urgent Water Supply and Sanitation Rehabilitation Project Zimbabwe, 6th November 2015

Staff Guidance on Project Completion Reporting and Rating, Quality Assurance and Results Department (ORQR), Quality Assurance Division (ORQR.2), August 2012

Various other internal reports, including trip reports, progress reports, mid-term review reports.