Annex1 PCR EVALUATION NOTE 1. Basic Project Data · Annex1 PCR EVALUATION NOTE 1. Basic Project Data Country: Mozambique Project SAP Reference: ... of Malawi, Zambia and Zimbabwe,

Annex1

PCR EVALUATION NOTE

1. Basic Project Data

Country: Mozambique Project SAP

Reference:

P-MZ-DBO-003

Project Title : Vanduzi-Changara Road Rehabilitation Project

Sector : Tranport Appraisal Actua

l

Theme Road Project Costs

(MUA)

20.79 39.71

Amount cancelled (MUA)

ADF: 1.28

Grant: 0.47

Disbursement rate

ADF : 92.4%, TAF : 41%

Project

Financing

(MUA)

PCR Date: August 2011 ADF Loan 16.79 15.51

PCR Evaluation Note Date: February 2013 TAF Grant 0.80 0.33

Government 2.01 23.87

Board

Approval

Date

Dec 15,

1999

Partners (MUA)

Republic of Mozambique: 2.01

Government of Japan: 1.10

Signature

Date

:Effective

Date

Aug 11,

2000

Aug

11,

2000

Project

Completion

Date

Jan 31,

2003

May

1,

2010

Project Officers (Names) – At Appraisal At Completion

Regional Director Chiji Ojukwu

Sector Director G. Gorgis G. Mbesherubusa

Task Manager

H. Nyame-

Mensah

A. Ali

PCR Team Leader

J. Mabombo

PCR Team Members K.S.H.Rao

PCR Evaluation Note- Mozambique vanduzi road Page 2 of 26 A. Ezzine, February 4, 2013

Evaluator/consultant: Mrs. Aoufa Ezzine Peer Reviewer/Task Manager: Mr.

Foday Turay

2. Project Description (summary from Appraisal Report including addendum/corrigendum or

loan agreement)

a. Rationale and Expected Impacts:

Brief and clear description of the project/programme rationale (problem/issue to be addressed),

expected impacts and intended beneficiaries (direct and indirect impacted). Highlight any changes

during implementation.

The aftermath of war in Mozambique left the country with a vastly damaged transport infrastructure,

mainly the road network. The Government of Mozambique (GOM) embarked in 1992 on a 10-year

Roads and Costal Shipping Project (ROCS) to remove the transportation bottlenecks constraining

agricultural production and marketing and to strengthen the infrastructural base of the economy. The

ROCS which was implemented in two phases was to assist the economic recovery programs through

the rehabilitation and maintenance of basic transport infrastructure in selected priority districts and

international/national corridors. The rehabilitation of the project road i.e. Vanduzi-Changara,

component of ROCS 1, was identified by GOM as one of the priority roads with heavy and intensive

inter-urban traffic in 1992, and which would significantly contribute to regional integration and

enhanced transit traffic as this road is an important regional transit route for the landlocked countries

of Malawi, Zambia and Zimbabwe, which use the port of Beira for their foreign trade. Apart from

removing bottlenecks and improving infrastructure facilities, the GOM has also undertaken various

reforms that culminated in the establishment of necessary organizations and environment for

efficient management of road infrastructure. The National Roads Administration (ANE) was

established in 1999, under relevant acts of parliament, with the objective of managing and financing

the maintenance and development of classified road network. As such, the Road Fund (RF) formed

an integral part of ANE. Subsequently, to delink the management and financing functions, the RF

was separated and set up in 2003 under relevant acts of parliament as an autonomous organization

for financing the road maintenance and development of classified roads while the responsibility for

the other roads is vested with the local authorities.

From 1977, when the Bank commenced its operations in Mozambique, a total of fourteen projects

including a supplementary loan and a multi-national project have been approved for a total amount

of UA 295 million; of which UA 73 million were in grants for studies and technical assistance

support. Till date, seven operations have been completed successfully while two projects were not

completed and cancelled and another closed due to internal strife. Currently, Montepuez-Lichinga

Road (original and supplementary loan) and Nacala Road Corridor projects are on-going. The

objectives of the on-going operations are in line with the GOM developmental strategy of overall

economic development with focus on promoting infrastructure and agriculture development. Of the

total amount approved, UA 116.96 million (67.59%) has been disbursed and UA 19.36 million

cancelled.

The financed operations included the construction of the Nampevo-Alto-Molocue road approved in

1977 followed by the Alto-Molocue-Ligonha road in 1979. Both roads could not be completed

because of the civil war and the loans were cancelled. The Bank then funded three road

rehabilitation studies for the Beira-Machipanda, the Vanduzi-Changara and the Pemba-Montepuez

sections. The Bank Group followed the studies with funding the rehabilitation of the Pemba-


Montepuez, the Mussacamua-Colomue and the Vanduzi-Changara road projects. The Pemba-

Montepuez was completed in 2000 and the Mussacamua-Columue is substantially completed.

To date, a total of 730 km of bitumen road on the primary network has been rehabilitated with Bank

assistance and has improved the level of transportation services in the provinces concerned and

increasingly supported the broad based economic development and social service coverage in the

projects’ zone of influence. Further, being an important link in the Beira transport corridor, the

project rehabilitated road would contribute to enhanced regional trade and cooperation especially

between Mozambique and its landlocked neighbouring countries of Malawi and Zimbabwe.

The project expected impacts are to: i) restore economic growth through improving road transport,

ii) protect past road investments by rehabilitating priority roads, iii) contribute to the creation of a

self-sufficient Road Authority capable of efficiently managing and maintaining a completely

rehabilitated road network, iv) consolidate the overall gains made during the implementation of

ROCs 1 and 2, and v) reduce the overall transport costs due to the rehabilitation of the road. The

accompanying increases in land and labour productivity will increase farm and non-farm income for

farmers as well as traders in Tete and Manica provinces. In addition, the delivery of social services

related to education, health, nutrition and food for work programs including community based

development activities supported by a number of NGOs in the two provinces would be facilitated.

b. Objectives and Expected Outcomes:

Concise and clear description of planned objectives and associated outcome targets, and highlight of

any modifications/revisions.

The project will contribute towards achieving the following sector goals: i) improve economic

activities through the rehabilitation and maintenance of transport infrastructure to protect past

investments and ii) stimulate and recover the previous traffic levels of the national and rural road

networks.

The objectives of the project are to improve road transport services in the project area and to reduce

road maintenance and vehicle operating costs, by rehabilitating the badly deteriorated existing 270

km long bitumen standard road between Vanduzi and Changara, an important regional transit route

for the landlocked countries of Malawi, Zambia and Zimbabwe which use the port of Beira for their

foreign trade. Rehabilitation of this road section will form an important component of Beira Corridor

development plans being taken up by the South African Development Community (SADC).

The project expected outcomes are: i) Vehicle operating costs reduced by respectively 10% and 2%

in the year 2003 and 2021 when the road is fully open to traffic in 2003, ii) Roughness of about 2000

mm/km throughout the life of road up to year 2003, iii) Average Annual Daily Traffic (AADT) of

500 vehicles for the project road by 2003.

c. Outputs and Intended Beneficiaries:

Concise and clear description of outputs and associated targets, and highlight of any revisions.

The project outputs are: i) Construction works consisting in the rehabilitation of the 270 km long

Vanduzi - Changara road, ii) Consultancy services to supervise the works, iii) Audit report and iv)

Institutional support to ANE (technical assistance and four fellowships for post graduate training).

The project intended beneficiaries are the population of the northern half of Manica province

consisting of Barue and Guro districts. However, the general area of influence due to the

international role of the road is Tete and Manica Provinces.


d. Principal Activities/Components:

Concise and clear description of principal components and activities, and highlight of any revisions.

The project has the following four components:

(i) Civil Works comprising rehabilitation of the existing badly deteriorated bitumen surfaced road of 6.0

m wide carriageway, and 1.0 m shoulders on each side for the total distance of 270 Km between Vanduzi

and Changara;

(ii) Consultant services for the supervision of the civil works; this included the recruitment of an

experienced firm of consultants for supervision services of the civil works. The responsibility of the

consultant included also the following: assisting the Executing Agency in tendering and tender

evaluation process, contract award process for the civil works, administer the construction contract,

inspect the works, supervise the necessary quality control testing performed by the contractor, track the

progress and the cost, preparation and submission of monthly, quarterly and final completion reports, and

maintain close relation with the Executing Agency and relevant ministries,

(iii) Project audit services. The purpose of the audit service is to ensure that proceeds of the loan were

utilized solely for the purpose for which they were intended; and

iv) Institutional support to ANE in the form of a) short-term technical assistance to the newly formed

ANE in the fields of planning, contract administration, pavement engineering and materials. In all four

technical assistances were proposed to be recruited by the ANE to fill each of these fields; and b) training

to provide four fellowships for post graduate training in the above four fields for ANE staff. The

proposed training was intended to enable the selected staff members to assume positions of higher

responsibility within the organizational structure of ANE.

3. Evaluation of Design and Implementation (Evaluator assessment of actual vs. envisioned)

a. Relevance of Project Design (and Readiness for Implementation) (The evaluator assesses,

independent of the PCR assessment, the Relevance of the project Design):

The evaluator will also comment on the PCR conclusion regarding this section, and also judge the

relevance of project design, covering realism (complexity-implementation capacity & environment);

clarity and measurability of objectives and results chain; adequacy of assessment risks,

environmental and social safeguards, and implementation arrangements).

The evaluation confirms the rating satisfactory for the project design and readiness for

implementation.

The appraisal report presents a Log Frame with clear outcomes statement and benchmarks.

However, the following outcomes are missing: reduction in travel time and road accidents. The risks

were well identified and adequate measures (national road administration and road fund set up) were

put in place. The responsibility for project implementation was clearly defined with the ANE as

implementing agency. Therefore, no Project Implementation Unit was created and implementation

was under GOM structure. Also, the risk of contractors’ capacity to mobilize equipment was

captured at appraisal in the Log Frame but no mitigation measures were identified.

b. Project Cost (including Borrower Contribution), Disbursements, and Adherence to Schedules (as

relevant to project performance): Assessment of the extent of completeness and fairness of the

the PCR in covering key project implementation aspects –project activity and outputs costs (estimate

- actual; reasons for any changes); disbursements (planned - actual; reasons for any gaps and delays);

timeliness of project completion ( and reasons for any delays):


The PCR treated adequately the project costs disbursement and timeliness.

The estimated project cost at appraisal was UA 20.79 million (including physical and price

contingencies). The ADF loan of UA 16.79 million and Grant – UA 0.80 million (was to cover

84.37% of the total rehabilitation and supervision cost, Japan 5.52% (UA 1.10 million) and GOM

would contribute the remaining 10.11%, equivalent to UA 2.01 million, as counterpart funding. At

project completion, the actual contribution of the Bank was UA 15.84 million (39.9%), the

contribution of ADF being UA 15.51 million (39.1%) and Grant – UA 0.33 million (0.8%) while

GOM provided UA 23.87 million representing 60.1% as counterpart contribution. At the completion

of the project, the ADF loan balance of UA 1.28 million and grant balance of UA 0.47 million was

cancelled because all project components were funded. Overall total project cost was UA 39.71-

million, which is 91% higher than the appraisal estimate of UA 20.79 million. The cost overrun is

due to the 88 months delay in project implementation leading to higher costs than appraisal and

unexpected appreciation of the Rand against US dollar.

c. Implementation Arrangements, Conditions and Covenants, and related Technical Assistance:

Assessment of the extent of completeness and fairness of the PCR in covering these aspects –

implementation arrangements (planned – actual); performance of consultants; compliance with

project covenants; attached technical assistance (PCR availability if TA is at least UA 1 million):

The PCR treated adequately implementation arrangements, compliance with covenants and

consultants performance. Procurement activities were defined according to the Bank guidelines and

ensured transparent bidding process for contractors.

The Borrower’s performance to fulfill the loan conditions was satisfactory, which is evidenced by a

timely i) appointing a project coordinator whose qualifications and experience were acceptable to the

Bank, ii) setting up of a National Road Administration (ANE) to manage and plan road works and

maintenance, setting up a separate autonomous agency i.e. Road Fund (RF) for financing road

maintenance and development and iv) taking timely measures for protection and enhancement of

environment during the construction period.

The consultants supervised well the project. The ANE hired the 4 short term Consultants but failed

to provide in a timely manner and before project completion four fellowships for post graduate

training as proposed at appraisal.

d. Monitoring & Evaluation Design, Implementation, & Utilization (Evaluator assessment):

Assessment of planned and actual M & E design, implementation and utilization:

Design: --Extent to which the project M & E system was clear, adequate and realistic to

generate and analyze pertinent data:

Implementation: Extent to which relevant data was collected: indications in PCR of M&E

implementation and its effectiveness

Utilization: Extent of use of data generated for decision-making and resource allocation:

indications in PCR of M&E use

The evaluation does not confirm the rating satisfactory for the overall project monitoring and

evaluation (M&E) and rates it fair. It was designed at appraisal to be done through project

implementation progress reports, environmental monitoring and audit reports. No baseline data was

collected by Borrower at Appraisal and Bank did not enforce it. No performance indicators were

conceived at appraisal and no monitoring other than works supervision and audit reports were

performed during project implementation.

e. Other Issues (such as Safeguards, Fiduciary):


Assessment of the extent of the PCR’s treatment of compliance with environmental safeguards,

fiduciary requirements, etc:

The PCR treated adequately the Borrower compliance with environmental and fiduciary issues. The

Borrower complied with the Bank’s fiduciary and environmental policy requirements. ANE’s Cross-

cutting Issues Office closely monitored the project environmental issues during construction period.


4. Evaluation of Performance (Evaluator assessment)

a. Relevance of project objectives (Assessment of the ex-ante and ex-post relevance (including

during implementation): Ex-ante and ex-post relevance of objectives (at appraisal & evaluation) in

terms of consistency with country development priorities, the Bank’s country or regional strategy

and corporate priorities. The assessment of the ex-ante relevance of objectives will also cover the

extent to which the objectives were realistic given the available inputs and timeframe.

The evaluation rates highly satisfactory the relevance of the project objectives at appraisal and at

post completion. The Project is in conformity with the country’s development priorities and is

consistent with the Bank’s country’s and regional strategies and Bank’s corporates priorities. The

project remains relevant as the Vanduzi-Changara road is an important regional transit route for the

landlocked countries of Malawi, Zambia and Zimbabwe which use the port of Beira for their foreign

trade.

b. Effectiveness in Delivering Outputs: Extent of Delivery of Outputs (or key conditions in the case

of Program Loans): Assessment of the extent of PCR coverage and discussion of project outputs --

planned and actual outputs and reasons for any deviations;

The evaluation confirms the PCR rating satisfactory for the project effectiveness in delivering

outputs. All project activities were realized satisfactorily (270 km of road rehabilitation, audit reports

and technical assistance to ANE) except that ANE did not sponsor fellowships for 4 post graduates

to undertake trainings under the Grant.

c. Effectiveness in Achieving Outcomes: Assessment of the extent of achievement of the expected

outcomes (from the log frame) using direct/indirect evidence –achievements or expected

achievements. In the absence sufficient data (as direct evidence), indirect evidence (such as outputs

and processes pertinent to the chain of causality) should be used especially in assessing the extent to

which outcomes/objectives are expected to be achieved. The absence of sufficient data for assessing

effectiveness should be indicated (and described in details under PCR quality). The PCR’s rating

should also be reflected in this section.

The PCR confused the concepts of outcomes and treated them inadequately. At completion, the PCR

provided EIRR as an outcome when it discussed the outcome of reduction of variable operating costs

(voc), travel time and maintenance costs, which is wrong. Also, the PCR considered project delays

and good quality of work and cost effective project as outcomes which is also wrong.

The review put the right outcomes according to the Log Frame at appraisal and added the reduction

in travel time as an outcome. The review put the related figures at completion based on the

documentation (PCR and Aide Memoires).

The review rates satisfactory the effectiveness in achieving outcomes. The Annual average daily

traffic (AADT) grew in average, by 2% between 2001 and 2010 and was at 450 in 2010 (and not 500

as expected at appraisal) , travel time decreased by 50% and road network in fair/good condition

increased from 20% in 1999 to 70% in 2009.

d. Efficiency in Achieving Outputs and Outcome : Assessment of the extent to which the outputs and

outcomes were timely, cost-efficient and delivered in the most efficient way. The PCR’s rating

should be commented on. Evidence will include the rates of return (ERR; FRR), extent of timeliness

(time over/under-run), cost savings/over-run and indications of efficient project implementation. In

commenting on the PCR’s rating, the extent of use of these sources of evidence should be taken into


account. In the presence of insufficient evidence, an appropriate rating should be applied.

The project efficiency is rated satisfactory since the economic investment rate of return (EIRR) at

completion is 14,4% (29% at appraisal) and is greater than 12%, the opportunity cost of capital in

Mozambique. At completion, the EIRR is lower because the cost overrun is 91%.

e. Project/Development Outcome (PDO): Assessment of the extent to which the relevant objectives

(outputs; outcomes) were efficiently/timely achieved or likely to be achieved. This should lead to a

calculated rating, reflecting the ratings of outputs/relevance, outcomes/effectiveness and

Timeliness/efficiency. Two PO ratings will be provided: (i) as per the PCR operational definition

(Outputs; outcomes; timeliness) and (ii) as per OPEV guidelines (Relevance; effectiveness;

efficiency (including timeliness). The evaluator can override the calculated rating, and provides the

justification accordingly. The three components are equally weighted.

The evaluation note agrees with the PCR rating satisfactory for the project/development outcome. As

explained in section 4.c above, the project outputs achievements (road rehabilitation, technical

assistance) led to savings in voc and transport time, traffic increase and road condition improvement.

The project is efficient since the EIRR is 14,4% at completion and is greater than 12%, the

opportunity cost of capital. However, the project cost overrun is 91% and project delay is 88 months

which justifies the rating satisfactory (only).

f. Risk to sustained achievement of Project Outcomes (At the time of evaluation, the risk that the

project outcomes (or expected outcomes) will not be maintained (or achieved):

While commenting on the risk factors, stated in the PCR, a solid rationale (including additional

factors that are important in decreasing/increasing risks) should be provided for the rating of the

overall risk to development outcome. The risk should be assessed as high (1); substantial/high (2);

moderate (3) and negligible (4).

The review rates the project overall risk as moderate. 70% of road maintenance works are funded by

the Road Fund and the remaining 30% is funded by Donors. The GOM undertook the appropriate

measures to sustain project achievements: the National Road administration is running, the Road

Fund is set up autonomously and the axle load control is enforced.

g. Additional Outcomes/Impacts (positive and negative, not captured in the logframe): These include

gender, safeguards, institutional, social and socio-economic issues. Assessment of the extent of the

importance of additional and important outcomes stated or not covered in the PCR. The review

should comment on the PCR’s coverage of these outcomes.

The project had a very positive impact on private sector development. 15 small buses are currently

operating between Vanduzi and Chimoro against 8 before project.

8 buses are operating between Vanduzi and Ringwe against 3 before project implementation.

Utilization of boarding house at Vanduzi has significantly increased after the road rehabilitation. The

rehabilitated road is likely to contribute significantly in terms of socio-economic development in the

area as well as increase in regional trade.

h. Performance of Borrower: Commenting on the fairness of PCR’s rating, and re-rating

performance over the entire project cycle (design; implementation; closure) focusing on the

indications of evidence in the PCR (see detailed components in annex):.


The evaluation confirms the Borrower performance as satisfactory. The Borrower implemented well

the project. This is evidenced by the level of outputs and outcomes achievement. The Borrower

provided the audit reports in a timely manner and followed all fiduciary requirements. However,

Borrower did not collect any baseline data and there were 88 months delays in project completion

due to the inefficiency of the first contractor who could not mobilize equipment on time and ended

services 15.5 months after start up. The works were suspended between October 2003 and August

2007 at which date the second contractor resumed the works.

i. Bank Performance (preparation/appraisal –partly ensuring QAE: supervision quality): :

Commenting on the fairness of PCR’s rating, and re-rating performance over the entire project cycle

(design; implementation; closure) focusing on the indications of evidence in the PCR (see detailed

components in annex):

The evaluation confirms the PCR rating the Bank’s performance as satisfactory for the following

reasons: The Bank designed and supervised the project that realized its outputs and achieved its

outcomes satisfactorily. The Bank made sure that all procurement activities are undertaken in

conformity with its rules and procedures. However, the Log Frame lacked the following outcomes:

reduction in travel time, reduction of car accidents, and performance indicators reflecting the

regional integration focus of the project such as increase in trade volume. Also, the supervision

missions lacked environmental and gender experts.

k. Overall Assessment: The evaluator’s summary claim on the overall performance of the

project/programme, and in the case of this being different from that of the PCR, the basis for the

difference(s) should be highlighted. And if the accessible evidence (from PCR and other sources) to

the evaluator is insufficient, then a partly ‘satisfactory rating (to be reviewed)‘ should be given until a

PPER can be done.

The evaluation confirms that the overall project performance is fair. The project realized both outputs

and outcomes but with 91% cost overrun and 88 months delay.

5. Key Lessons, and Recommendations (Evaluator assessment)

a. Lessons Learned: Concise statement on agreement/disagreement with any or all of the PCR’s

lessons. Key and pertinent (and generic) lessons from PCRs, reworded and/or new to be listed here.

The evaluation notes that lessons and recommendations are mixed in the PCR. The PCR provided

recommendations that it entitled lessons. The evaluation proposes the following:

i) Using only bidding contractors and not changing them during implementation is key to avoid

project procurement issues and delays;

ii) Release of counterpart funds in a timely manner and according to Bank’s rules is necessary to

avoid delays in project implementation; and

iii) Transfer of know how to PIU staff is key to ensure project sustainability and that project outputs

are used at their fullest potential.

b. Recommendations: Concise statement on agreement/disagreement with any or all of the PCR’s

recommendations. Key and pertinent recommendations (requiring further action by the borrower


and/or Bank) from PCRs, reworded and/or new to be listed here.

i) Changing contractors negatively affected procurement and subsequently delayed implementation.

It is advisable that in future reliable and consistent contractors with a good track record be engaged.

ii) Suitability of pavement design i.e. crushed stone base or cement stabilized base, should be

assessed after giving due consideration to the geological conditions, terrain, availability of

appropriate construction materials, for the road to be rehabilitated

iii) Although ANE has a dedicated unit for Project implementation, necessary skills need to be

developed for efficient management and monitoring of projects for timely completion within the

planned budget.

iv) All payments should be made according to the contract agreement to avoid unpleasant and costly

situation of termination of projects on such grounds.

v) Baseline data including traffic count should be collected before project start and project

monitoring should be regular until the hand-over of the project to the Borrower (immediately after

expiry of defects liability period). The Borrower should take-over the responsibilities thereafter.


ASSESSMENT SUMMARY

6. Ratings: PCR OP

EV

Rev

iew

Reason for Disagreement/Comments

Relevance: 4 4 The project remains relevant as the rehabilitated road

is an important transit road

Effectiveness in delivering

outputs:

3 3 Expected outputs were realized except that 4 staffs

from ANE did not receive any fellowship for training

Effectiveness in achieving

outcomes:

Efficiency in achieving outputs

& outcomes -Timeliness:

0 0 88 months delay which is more than the double of the

expected project implementation duration

Efficiency in achieving outputs

& outcomes –Disbursement

3 3 92, 4% for the loan disbursement which is greater than

80% the average rate disbursement. 42% for the Grant

because the remaining activities were funded by

GOM

Overall efficiency in achieving

outputs & outcomes

3 3 EIRR is 14,4% which greater than 12% the

opportunity cost of capital. 91% cost overrun

Risk to Development outcome: 3 3 The risk is moderate since reforms such as the

establishment of the National Road agency and the

Road fund are set up to ensure road management and

funding. Also axle load control is enforced

Borrower Performance: 3 3 Weak contracting capacity

Bank Performance: 3 3 Not all project outcomes were captured in Log Frame

at appraisal.

Supervision missions lacked environmental and

gender experts

Quality of PCR: 3 3 Outcomes are inadequately treated in PCR.

This is a summary of PCR and OPEV ratings together with reasons for disagreement/comments. For the

last column, the appropriate section of the PCR review should be cross-referenced to avoid detailed

discussions. Where the evaluator is unable to validate the PCR rating for any criterion, adequate

justification should be provided. As a result, of this, the overall project rating could be ‘partly

satisfactory’.

7. Comments on PCR Quality and Timeliness

PCR quality assessment will be based on some or all of the following criteria and others (as

appropriate):

Quality and completeness of the PCR evidence and analysis to base ratings

Objectivity/fairness of PCR assessment

Internal consistency of PCR assessment: inaccuracies; inconsistencies (in various sections;

between texts and ratings; consistency of overall rating with individual component ratings)

Identification and assessment of key factors (internal and exogenous) and unintended effects

(positive and negatives) affecting design and implementation:

Adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization

Soundness of data generating and analysis processes (including rates of returns) in support of


PCR assessment

Overall adequacy of the accessible evidence (from PCR including annexure and other data

provided)

Extent to which lessons learned (and recommendations) are clear and based on the PCR

assessment (evidence & analysis):

Overall clarity and completeness of the PCR

Other (to be specified)

The PCR quality will be assessed as highly satisfactory/exemplary (4), satisfactory (3),

unsatisfactory (2) or highly unsatisfactory (1).

Comments on the extent of compliance of the PCR with PCR guidelines and pertinent OPEV

guidelines:

PCR Timeliness; rated as timely (exemplary = 4) or late/unsatisfactory (1)

Extent of participation of Borrower, co-financiers and Bank field office, rated as

high/exemplary (4), or substantial/significant (3), or moderate (2), or negligible (1)

Other aspects (to be specified)

The PCR quality is satisfactory. Outcomes are inadequately treated in PCRs. Also, savings in voc are

not provided at completion although they are one important outcome of the project objectives. The

PCR did not use the outcomes of the Appraisal report Log Frame.

8. Priority for Future Evaluative work: Project for Performance Evaluation Report, Impact

Evaluation, Country/Sector reviews or Thematic Evaluation Studies:

- Project is part of series and suitable for cluster evaluation

- Project is a Success Story

- High priority for impact evaluation

- Performance evaluation is required to sector/country reviews

- High Priority for thematic or special evaluation studies (Specify )

- PPER is required because of incomplete validation rating

Major areas of focus in future evaluation work:

a)

b)

c)

Follow Up Action by OPEV:

Division Manager Clearance Director Signing Off


Data sources for validation:

Task managers/Bank staff interviewed/contacted (in persons, by phone & email)

Documents/reports and database

Attachment: PCR Evaluation Note Validation Sheet of Performance Ratings


Appendix 1

PROJECT COMPLETION REPORT EVALUATION NOTE

Validation of PCR Performance Ratings

PCR Rating Scale:

Score Description

4. Very Good – Full achieved with no shortcomings

3. Good – Mostly achieved despite a few shortcomings

2. Fair – Partially achieved. Shortcomings and achievements are roughly balanced

1. Poor – Very limited achievement with extensive shortcomings

UTS Unable to score/rate

NA Non Applicable

Criteria Sub-Criteria

PCR

Work

Score

OPEV

Review

Reason for deviation/

Comments

Relevance of

project

Objectives &

Design

Relevance of project Objectives

1. Project objectives were

relevant to country

development priorities.

4 4

Validated. The project was

relevant to the country's

sector goal of improving

economic activities through

rehabilitation and

maintenance of transport

infrastructure so as to

protect past investments and

to stimulate and recover the

previous traffic levels of the

national and rural road

networks.

2. Project Objectives

could in principle be

achieved with the project

inputs and in the expected

time frame.

2 2

Validated. Project

implementation was delayed

by 88 months due to weak

institutional capacity and

problems in contractors

recruitment.

3. Project Objectives were

consistent with the Bank’s

country or regional

strategy

4 4

The project was consistent

with the Bank's lending

program and sector

priorities of the Bank's

Country Strategy Paper

(CSP) for Mozambique for

the period 1998-2001


consistent with the Bank’s 4 4

The project was consistent

with the Bank's corporate


corporate priorities priorities of transport

development, regional

integration, and sustainable

economic growth.

5. Ex-post Relevance of

objectives n/a 4

The project remains relevant

as the Vanduzi - Changara

road is an important

regional transit route for the

landlocked countries of

Malawi, Zambia and

Zimbabwe which use the

port of Beira for their

foreign trade.

Relevance of project Design -Project Design & Readiness for Implementation

1. The log frame presents

a logical causal chain for

achieving the project

development objectives.

3 3

The Log Frame project matrix

is presented in a logical

framework approach that was

fairly comprehensive.

2. The log frame expresses

objectives and outcomes

in a way that is

measurable and

quantifiable.

3 3

Validated. However, the revue

notes that the following

outcomes are missing :

reduction in travel time and

reduction in road accidents.

3. The log frame states the

risks and key assumptions. 2 3

Risks related to project

implementation such as

availability of construction

material and contractor’s

capacity to mobilize equipment

were captured (unlike what is

mentioned in PCR). In

addition, project design is

based on lessons learned from

previous projects and

mitigation measures. These

lessons include: i) regular

supervision missions and

undertaking of procurement

seminars/workshops and ii)

procurement of supervision

consultant prior to the

commencement of the project

for increasing the chances of

timely project.

4. Project complexity was

matched with country

capacity and political

commitment.

3 3

A Project Coordinator was

designated to oversee the

Project and ANE capacity was

enhanced with recruitment of

4-short-term consultants.

5 Project design includes

adequate risk analysis. 3 3

Project design includes

adequate risk analysis and

mitigation measures. It ensured


project sustainability by

ensuring axle load control and

road maintenance funding. The

revue notes that for this

section, PCR did not put the

right arguments that should

deal with risks related to

project sustainability and not to

project implementation risks.

6. Project procurement,

financial management,

monitoring and/or other

systems were based on

those already in use by

Government and/or other

partners.

3 3

All procurement, financial

management, and monitoring

systems used by the project

were in line with government

procedures as well as with

Bank rules and procedures.

7. Responsibilities for

project implementation

were clearly defined.

3 2

Responsibility for project

implementation was clearly

defined at appraisal but not

enforced. No PIU was created

and project implementation

was under the GOM structure

8. Necessary

implementation

documents (e.g.

specifications, design,

procurement documents)

were ready at appraisal.

3 3

The necessary implementation

documents in respect of road

designs and environmental

mitigation measures were

ready at appraisal. Procurement

rules and procedures were in

place at appropriate time.

9. Monitoring indicators

and monitoring plan were

agreed upon during

design.

3 2

Project monitoring at design

concerned works monitoring

and audits. No monitoring of

the performance indicators

related to traffic, costs and time

was agreed upon at appraisal.

10. Baseline data were

available or were collected

during design.

2 1 No baseline data were

collected or available.

Project Design &

Readiness for

Implementation score

3 3 As rounded average of scores

above

Relevance for objectives

and design score 4 4

As rounded average of scores

above

Achievement

of Project

Outputs*

1. A 270 Km existing

bitumen surfaced road

between Vanduzi and

Changara rehabilitated

with 6.0m wide

carriageway and 1.0

shoulders on each side.

3 3

The road between Vanduzi and

Changara (270 km) was

rehabilitated with 6.8 m (and

not 6) bitumen carriageway

and 0.5 m (and not 1)

shoulders on either side.

Design was reviewed during

implementation.


2. Undertaking quality

supervision on

rehabilitating 270 km of

existing bitumen road in

accordance to the designs.

3 3

Quality supervision of road

project was good

3. Audit reports 3 3

Acceptable Audit reports were

provided.

4. Technical Assistance

for providing 4 short -term

consultants to ANE and

fellowships for post

graduate training for

selected ANE staff.

2 2 4 short-term TA were recruited

by ANE; one for each of the

areas of planning, contract

administration, pavement

engineering and materials.

ANE did not sponsor

fellowships for post graduates

under the grant. Bank rejected

two ANE proposals, the first

one because 3 top management

positions would be vacant

during project implementation

and the other one consisting in

training 20 juniors was

proposed late and was rejected

to avoid to extend last project

disbursement date

OVERALL PROJECT

OUTPUT SCORE 3 3

Achievement

of Project

Outcomes

1. Reduced travel time

3 3

The revue notes that PCR did

not provide the value of this

outcome at completion and

rather gave the EIRR which is

wrong.

The review gives the travel

time at completion. It has

decreased by 50% which has

facilitated easy and fast travel

from these areas to far-off

places like Tete and Maputo.

2. Vehicle operating costs

reduced respectively by

10% and 2% in 2003 and

2021 when the road is

fully opened to traffic in

2003.

3 n/a

This outcome is put by the

review cf appraisal report but

PCR did not provide any value

in terms of variation between

appraisal and completion

figures. PCR used voc in

EIRR calculation.

3. Roughness of about

2000 mm/km throughout

the life of road up to year

2003.

Not

scored 3


review cf appraisal report but

PCR did not provide any value.

PCR mentions in annex that

road network in good/fair

condition has increased from


20% in 1999 (at the time of

appraisal) about 70%in 2009.

4. AADT of 500 vehicles

for the project road by

2003.

Not

scored 3


review. AADT grew in average

2% per year between 2001 and

2007, and 4% between 2007

and 2010 therefore 2% in

average between 2001 and

2010 and was at 450 in 2010

because of road deterioration

due to long project

implementation. It is expected

to increase once roads are

rehabilitated.

OVERALL PROJECT

OUTCOME SCORE 3 3

PCR score is for the wrong

outcomes. The revue has

values for most of the right

outcomes at completion

Gender n/a

Private Sector

Development 3 3

15 small buses are currently

operating between Vanduzi

and Chimoro against 8 before

project.

8 buses are operating between

Vanduzi and Ringwe against 3

before project implementation.

Utilization of boarding house

at Vanduzi has significantly

increased after the road

rehabilitation., the rehabilitated

road is likely to contribute

significantly in terms of socio-

economic development in the

area as well as increase in

regional trade.

Resettlement n/a

HIV/AIDS prevention n/a 3

The Contractor recruited a

service provider to address

social HIV/AIDS issues.

Condoms and pamphlets were

distributed.

Efficiency in

achieving

outputs &

outcomes

Timeliness (in adhering to

the original closing date) 0 0

88 months delay which more

than the double of the expected

project duration

Rates of returns (Specify

if applicable) 3

14, 4% at completion (instead

of 29% as expected at

appraisal) but both rates are


higher than 12% the

opportunity cost of capital in

Mozambique. 91% cost

overrun.

OVERALL PROJECT

EFFICIENCY 2

Risk to

Sustained

achievement

of Project

Outcome**

Key Risks (Specify

common risk factors to be

a basis for rating)

3 3

Project is at moderate risk. Its

sustainability depends on

provision of proper

maintenance and axle load

control. As part of the project,

weigh bridge at Vanduzi has

been refurbished and started

functioning from June 2010,

which is effectively ensuring

axle load control on the project

road. Road Fund is the

financing agency for road

maintenance activities and its

revenues are derived from fuel

levy, transit charges, bridge

tolls, etc. Currently RF is

financing about 70% of the

planned maintenance

requirements; the remaining

being contributed by the

donors under the-ongoing

Integrated Road Sector

Program (PRISE) supported by

various donors under Sector

and Budget Support (SBS)

approach. Efforts are underway

to increase RF contribution to

maintenance activities under

2010-2014 PRISE with the

ultimate aim of financing the

entire classified road

maintenance requirement by

the RF.

* The output ratings have to be weighted by the relative output costs (see PCR format).

The Overall rating is given Very Good, Good, Fair and Poor. Overall rating is the sub-criteria

average.

** Overall risk to sustained achievement of project outcome is rated as follows:

(i) High (H) : 1

(ii) Substantial/significant ( S) : 2

(iii) Moderate (M) : 3

(iv) Negligible (N): 4

(v) UTS (vi) NA


Criteria Sub-Criteria

PCR

Work

Score

OPEV

Review

Reason for disagreement/

Comments

Bank

Performance Design and Readiness


relevant to country

development priorities.

4 4

Validated. The project was

relevant to the country's sector

goal of improving economic

activities through rehabilitation

and maintenance of transport

infrastructure so as to protect past

investments and to stimulate and

recover the previous traffic levels

of the national and rural road

networks.

2. Project Objectives could

in principle be achieved with

the project inputs and in the

expected time frame.

2 2

Project design did not take into

consideration the country’s

contractors capacity and

equipment mobilization which

caused significant delays in project

implementation



country or regional strategy

4 4

The project was consistent with the

Bank's lending program and sector

priorities of the Bank's Country

Strategy Paper (CSP) for

Mozambique for the period 1998-

2001



corporate priorities

4 4

The project was consistent with the

Bank's corporate priorities of

transport development, regional

integration, and sustainable

economic growth.

5. The log frame presents a

logical causal chain for

achieving the project

development objectives.

3 3

Validated-

6. The log frame expresses

objectives and outcomes in a

way that is measurable and

quantifiable.

3 2

Validated. However, the following

outcomes are missing in the Log

Frame and the PCR did not

mention them: reduction in travel

time, reduction of car accidents,

increase in trade volume. No

Regional integration indicators

despite the Project RI focus

7. The log frame states the

risks and key assumptions. 2 3

Risks related to project

implementation such as

availability of construction

material and contractor’s capacity

to mobilize equipment were

captured (unlike what is stated in

PCR) and still caused substantial

delays. Also, the project design is


based on lessons learned from

previous projects and mitigation

measures (regular supervision

missions, procurement

seminars/workshop and

supervision consultant prior to the

commencement of the project for

timely project implementation)

8. Project complexity was

matched with country

capacity and political

commitment.

3 3

Project design included a Project

Coordinator to oversee the Project,

the recruitment of 4-short-term

consultants to enhance ANE

capacity and 4 fellows to get

training.

9. Project design includes

adequate risk analysis. 3 3

Project design ensured project

sustainability by ensuring axle load

control and road maintenance

funding

10. Project procurement,

financial management,

monitoring and/or other

systems were based on those

already in use by

government and/or other

partners.

3 3

Validated. All based on Bank’s

procedures and in use by

Government


project implementation were

clearly defined.

3 3

Responsibility for project

implementation was clearly

defined at appraisal with ANE as

executing agency.

12. Necessary

implementation documents

(e.g. specifications, design,


were ready at appraisal.

3 3

The necessary implementation

documents in respect of road

designs and environmental

mitigation measures were ready at

appraisal. Bank’s Procurement

rules and procedures were in place

at appropriate time.


and monitoring plan were

agreed upon during design.

3 2

Project design lacked monitoring

indicators such as travel time, car

accidents, trade volume.

Monitoring design was basic

(works construction and audit

reports)

14. Baseline data were

available or were collected

during design.

2 1

The Bank did not make sure that

baseline data are available at

appraisal

Project Design and

Readiness Sub-Score 3 3


Criteria Sub-Criteria PCR

Score

OPEV

Review

Justification/ Reason for

deviation/Disconnect/

Comments

Bank

Performance Supervision

1. Bank complied with:

Environmental

Safeguards 3 3

The Bank prepared the

road infrastructure project

with consideration of the

environmental safeguards.

The ANE's Cross-cutting

Issues has actively

participated during the


processes.

Fiduciary

Requirements 3 3

The Bank ensured that its

fiduciary rules and

procedures were adhered to

through project

implementation in granting

its ""No Objections"" to

Government's requests.

Project Covenants 3 3

All project covenants were

fulfilled.

2. Bank provided quality

supervision in the form of

skills mix provided and

practicality of solutions.

2 2

The Bank undertook 19

(nineteen) supervision

missions, including 4 desk

supervisions, during the


period. The skills mix was

not always adequate.

Missions were composed

of experts in the transport

sector and also socio-

economists. The missions

would have benefited from

inclusion of environmental

and gender experts.

3. Bank provided quality

management oversight. 3 3 Validated

PCR was delivered on a

timely basis 1 1

PCR prepared with 16

months delay

Supervision Sub-Score 3 3

OVERALL BANK

PERFORMANCE SCORE 3 3


average.


(i) Very Good (HS) : 4

(ii) Good ( H) : 3

(iii) Fair (US) : 2

(iv) Poor (HUS): 1


Score

PCR-EVN

Validation



Comments

Borrower

Performance Design and Readiness



are clearly defined.

3 3 Validated.

2. Necessary

implementation

documents (e.g.

specifications, design,


are ready at appraisal.

2 2

Procurement documents

were in place but Borrower

was not familiarized with.


and monitoring plan are

agreed upon and baseline

data are available or are

being collected

1 1

Huge delays in procurement

and subsequently in project

start up. Borrower did not

collect any baseline data

Project Design and

Readiness Sub-Score 2 2

Implementation

1. Borrower complied

with:

Environmental

Safeguards 3 3

ANE's cross-cutting issues

office closely monitored the

implementation of the

project environmental

safeguards.

Fiduciary

Requirements 3 3

Borrower followed most of

the fiduciary requirements

of the Bank and its own

rules and procedures

throughout project

implementation.

Project Covenants 3 3

Borrower fulfilled all

project covenants in a

timely manner.

2. Borrower was

responsive to Bank

supervision findings and

3 3

Borrower generally agreed

to the Bank's supervision

missions findings and


recommendations. recommendations.

3. Borrower collected and

used of monitoring

information for decision-

making.

3 3

Borrower used monitoring

information from project

supervision reports for

decision-making

Implementation Sub-

Score 3 3

OVERALL

BORROWER

PERFORMANCE

SCORE

3 3


average.

(i) Very Good (HS) : 4

(ii) Good ( H) : 3

(iii) Fair (US) : 2

(iv) Poor (HUS): 1

M &E Design, Implementation & Utilization


Score

PCR-EVN

Validation



Comments

M & E

Design M & E Design

2

Basic monitoring (works

supervision and audit

reports). No monitoring of

Performance indicators

Design Score 3 2

M & E

Implementati

on

Implementation Score 2 2 same

M & E Use

Use Score


OVERALL M & E

PERFORMANCE

SCORE

2 2

PCR Quality Assessment

Criteria PCR-EVN

(1-4) Comments

QUALITY OF PCR

1. Extent of quality and completeness of

the PCR evidence and analysis to

substantiate the ratings of the various

sections:

2

PCR confused outcomes and

outputs. PCR wrongly

discussed EIRR as outcome.

PCR mixed lessons and

recommendations.

2. Extent of objectivity PCR assessment

scores 3

PCR Assessment scores are

objective

3. Extent of internal consistency of PCR

assessment ratings; inaccuracies;

inconsistencies (in various sections;

between texts and ratings; consistency of

overall rating with individual component

ratings)

3 Ratings are consistent

4. Extent of identification and assessment

of key factors (internal and exogenous) and

unintended effects (positive and negatives)

affecting design and implementation:

3

PCR discussed additional

outcomes that were not

captured in Logframe

5. Adequacy of treatment of safeguards,

fiduciary issues, and alignment and

harmonization

3 All adequately treated

6. Extent of soundness of data generating

and analysis processes (including rates of

returns) in support of PCR assessment:

3 EIRR based on HDM

7. Overall adequacy of the accessible

evidence (from PCR including annexure

and other data provided)

3

EN based on supervision

mission reports and Bank

reports.

8. Extent to which lessons learned (and

recommendations) are clear and based on

the PCR assessment (evidence & analysis):

3 PCR mixed lessons and

recommendations

9. Extent of overall clarity and

completeness of the PCR 2

voc variation should have

been calculated and

mentioned since it is one of

the project outcomes.

Outcomes wrongly treated

in PCR.

Other (to be specified)

PCR Quality Score 3 Good overall

PCR Compliance with Guidelines (PCR/OM; OPEV)


1. PCR Timeliness (On time = 4; late = 1) 1 PCR was written 16 months

after project completion.

2. Extent of participation of Borrower, Co-

financiers & field office in PCR

preparation***

3

ANE project coordinator

assisted the PCR mission

during its visit

3. Other aspect (Specify) n/a

PCR Compliance Score 3 validated

*** rated as high/exemplary (4), or

substantial/significant (3), or moderate (2),

or negligible (1)