Embed Size (px)
Citation preview
Document of
The World Bank
Report No: ICR00003710
IMPLEMENTATION COMPLETION AND RESULTS REPORT
(TF-11062)
ON A
GRANT
IN THE AMOUNT OF US$16.00 MILLION
FROM THE MULTI-DONOR TRUST FUND FOR KHYBER PAKHTUNKHWA AND
FEDERALLY ADMINISTERED TRIBAL AREAS AND BALOCHISTAN
TO THE
ISLAMIC REPUBLIC OF PAKISTAN
FOR A
REVITALIZING HEALTH SERVICES IN KHYBER PAKHTUNKHWA PROJECT
August 31, 2016
Health, Nutrition and Population Global Practice
South Asia Region
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
CURRENCY EQUIVALENTS
(Exchange Rate Effective April 12, 2016)
Currency Unit = Pakistani Rupee (PKR)
US$1.00 = PKR 104.80
PKR 1.00 = US$0.01
FISCAL YEAR
July 1 – June 30
ABBREVIATIONS AND ACRONYMS
ADB Asian Development Bank KP Khyber Pakhtunkhwa Province BHU Basic Health Unit M&E Monitoring and Evaluation CPR Contraceptive Prevalence Rate MDTF Multi-Donor Trust Fund CPS Country Partnership Strategy MDTF-1 Round 1 of the KP/FATA/Balochistan
Multi-Donor Trust Fund DALY Disability Adjusted Life Years MICS Multi-Indicators Cluster Survey DHIS District Health Information System NPV Net Present Value DHMT District Health Management Team NGO Non-governmental Organization DoH Department of Health ORAF Operational Risk Assessment
Framework EPP Emergency Project Paper PSLM Pakistan Social And Living Standards
Measurement Survey ERR Economic rate of return PC-I Planning Commission Form I ESMP Environmental and Social
Management Plan PCNA Post-Crisis Needs Assessment
FATA Federally Administered Tribal Areas PDO Project Development Objective FCV Fragile, conflict, and violence PIFRA Project for Improvement of Financial
Reporting and Auditing FM Financial Management PMU Project Management Unit
GoKP Government of Khyber
Pakhtunkhwa PPHI People’s Primary Health Care Initiative
GoP Government of Pakistan PSLM
Pakistan Social And Living Standards
Measurement Survey HSRU Health Sector Reform Unit RHC Rural Health Center IO Intermediate Outcome SBA Skilled Birth Attendance ISR Implementation Status and Results
Report
TDP Temporarily Displaced Persons
JSDF Japan Social Development Fund
Senior Global Practice Director: Timothy Grant Evans
Country Director: Patchamuthu Illangovan
Practice Manager: Rekha Menon
Project Team Leader: Tayyeb Masud
ICR Team Leader/Author: Naoko Ohno
Pakistan
Revitalizing Health Services in Khyber Pakhtunkhwa Project
CONTENTS
Data Sheet
A. Basic Information ........................................................................................................ i B. Key Dates .................................................................................................................... i C. Ratings Summary ........................................................................................................ i D. Sector and Theme Codes ........................................................................................... ii
E. Bank Staff ................................................................................................................... ii F. Results Framework Analysis ...................................................................................... ii
G. Ratings of Project Performance in ISRs .................................................................. vii H. Restructuring (if any) ............................................................................................... vii I. Disbursement Profile ................................................................................................ viii
1. Project Context, Development Objectives and Design ........................................................... 1
2. Key Factors Affecting Implementation and Outcomes .......................................................... 5
3. Assessment of Outcomes ...................................................................................................... 12
4. Assessment of Risk to Development Outcome ..................................................................... 22
5. Assessment of Bank and Borrower Performance ................................................................. 23
6. Lessons Learned.................................................................................................................... 26
7. Comments on Issues Raised by Grantee/Implementing Agencies/Donors........................... 28
Annex 1. Project Costs and Financing .......................................................................................... 29
Annex 2. Outputs by Component.................................................................................................. 30
Annex 3. Economic and Financial Analysis ................................................................................. 32
Annex 4. Grant Preparation and Implementation Support/Supervision Processes ....................... 35
Annex 5. Beneficiary Survey Results ........................................................................................... 37
Annex 6. Stakeholder Workshop Report and Results ................................................................... 38
Annex 7. Summary of Grantee's ICR and/or Comments on Draft ICR ........................................ 39
Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................................... 51
Annex 9. List of Supporting Documents ...................................................................................... 52
Map 53
i
A. Basic Information
Country: Pakistan Project Name: Pakistan: Revitalizing
Health Services in KP
Project ID: P126426 L/C/TF Number(s): TF-11062
ICR Date: 08/26/2016 ICR Type: Core ICR
Lending Instrument: ERL Grantee: GOVERNMENT OF
PAKISTAN
Original Total
Commitment: US16.00 million Disbursed Amount: US$5.77 million
Revised Amount: US$10.20 million
Environmental Category: B
Implementing Agencies:
Department of Health, Government of Khyber Pakhtunkhwa
Cofinanciers and Other External Partners:
B. Key Dates
Process Date Process Original Date Revised / Actual Date(s)
Concept Review: 04/15/2011 Effectiveness: 04/12/2012
Appraisal: 06/30/2011 Restructuring(s):
06/10/2014
06/29/2015
10/28/2015
Approval: 04/12/2012 Midterm Review: 04/09/2013 04/09/2013
Closing: 06/30/2015 12/12/2015
C. Ratings Summary
C.1 Performance Rating by ICR
Outcomes: Unsatisfactory
Risk to Development Outcome: Modest
Bank Performance: Moderately Satisfactory
Borrower Performance: Moderately Unsatisfactory
C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)
Bank Ratings Borrower Ratings
Quality at Entry: Moderately Satisfactory Government: Moderately Unsatisfactory
Quality of Supervision: Moderately Satisfactory Implementing
Agency/Agencies: Moderately Unsatisfactory
Overall Bank
Performance: Moderately Satisfactory
Overall Borrower
Performance: Moderately Unsatisfactory
ii
C.3 Quality at Entry and Implementation Performance Indicators
Implementation
Performance Indicators
QAG Assessments (if
any) Rating
Potential Problem Project at
any time (Yes/No): Yes
Quality at Entry
(QEA): None
Problem Project at any time
(Yes/No): Yes
Quality of Supervision
(QSA): None
DO rating before
Closing/Inactive status:
Moderately
Unsatisfactory
D. Sector and Theme Codes
Original Actual
Sector Code (as % of total Bank financing)
Health 100 100
Theme Code (as % of total Bank financing)
Health system performance 70 70
Nutrition and food security 10 10
Population and reproductive health 20 20
E. Bank Staff
Positions At ICR At Approval
Vice President: Annette Dixon Isabel M. Guerrero
Country Director: Patchamuthu Illangovan Rachid Benmessaoud
Practice Manager/Manager: Rekha Menon Julie McLaughlin
Project Team Leader: Tayyeb Masud Tayyeb Masud
ICR Team Leader: Naoko Ohno
ICR Primary Author: Naoko Ohno
F. Results Framework Analysis
Project Development Objectives (from Project Appraisal Document)
To improve the availability, accessibility, and delivery of primary and secondary health care services at the
district level.
Revised Project Development Objectives (as approved by original approving authority)
n.a.
iii
(a) PDO Indicator(s)
Indicator Baseline Value
Original Target
Values (from
approval
documents)
Formally
Revised
Target Values
Actual Value
Achieved at
Completion or
Target Years
Indicator 1: People with access to a defined basic package of health, nutrition, and
reproductive health services (number) – core indicator
Value
(quantitative or
qualitative)
0 90% 3,956,119 3,816,585
Date achieved 04/12/2012 06/30/2015 05/21/2013 10/30/2015
Comments
(including %
achievement)
Surpassed against original target and achieved against revised. The Emergency Project
Paper (EPP) provided only a percentage target (90%) without baseline data. The target
value in terms of numbers was set after the Midterm Review in 2013 without
restructuring (Implementation Status and Results Report [ISR] No. 4). At the end, total
number of people with health services totaled 3,816,585, which reached 96% of the
revised target of 3,956,119. If the original target was meant to be 90 percent of the total
population, which is 2,737,386, 139 percent of the original target was also achieved.
Indicator 2: Percentage of children with severe acute malnutrition provided adequate nutrition
services
Value
(quantitative or
qualitative)
0 50% 20% (all target
districts) 16% (two districts)
Date achieved 04/12/2012 06/30/2015 06/28/2014 04/30/2015
Comments
(including %
achievement)
Not achieved against original target and partially achieved against revised. At the time
of restructuring in 2014, the target was substantially decreased as the project scope was
downsized due to the delay in rolling out the provincial nutrition program in all target
districts. Thus, 32% was achieved against the original target (16/50) and 80% against
the revised target (16/20).
Indicator 3: Births (deliveries) attended by skilled health personnel
Value
(quantitative or
qualitative)
24% 39% 30% 48%
Date achieved 04/12/2012 06/30/2015 06/28/2014 04/30/2016
Comments
(including %
achievement)
Surpassed. The EPP provided ‘a 15 percentage point increase over the baseline data’ as
the final target, with no numerical set value for each district; however, the first ISR
used 39% as the original target and 24% as the baseline as the average figure for target
districts. The baseline is from the MICS 2008. At the 2014 restructuring, the target was
then decreased to 30% as the scope downsized. The data from the Pakistan Social And
Living Standards Measurement (PSLM) Survey 2014-15 shows the average figure of
48% for five districts in 2014-15. If the PSLM 2012-13 had been available at the 2014
restructuring, the team could have used the data from the survey at 38% in 2012-13 as a
revised baseline and come up with a realistic revised final target. Even with the revised
baseline of 38%, the final figure of 48% is considered achieved against both targets.
Indicator 4: Contraceptive prevalence rate (any modern method)
Value
(quantitative or
qualitative)
14.3% 20% – 15%
iv
Date achieved 04/12/2012 06/30/2015 – 04/30/2015
Comments
(including %
achievement)
Not achieved. The EPP provided ‘a 15 percentage point increase over the baseline data’
as the final target; however, the ISR used 20% as the original target and maintained it
until the end. Against the 20% target, only 12% was achieved (0.7/5.7). The numbers
are for facility-based service delivery only (that is, it does not include community
outreach or private sector provision). The baseline is from the MICS 2008.
Indicator 5: Community satisfaction with health care services delivery by the public sector
Value
(quantitative or
qualitative)
38% 53% 45% 38%
Date achieved 04/12/2012 06/30/2015 06/28/2014 04/30/2016
Comments
(including %
achievement)
Not achieved. The EPP provided ‘a 15 percentage point increase over the baseline
data’ as the final target, with no numerical set value for each district; however, the ISR
used 53% as the original target and 38% as baseline (PSLM 2008). The EPP used the
data from PSLM, however, it should be noted that the wording used for the indicator is
slightly different between PSLM and the one in the EPP. At the 2014 restructuring, the
target was then decreased to 45% as the project scope was downsized. There was no
comprehensive community survey carried out. The data from the Pakistan Social and
Living Standards Measurement (PSLM) Survey 2014-15 shows the average figure of
38% for five districts in 2014-15. If the PSLM 2012-13 was available at the 2014
restructuring, the team would have used the data from the survey at 35% in 2012-13 as
a revised baseline. Even with the revised baseline of 35%, the final figure of 38% is not
achieved.
(b) Intermediate Outcome Indicator(s)
Indicator Baseline Value
Original Target
Values (from
approval
documents)
Formally
Revised Target
Values
Actual Value
Achieved at
Completion or
Target Years
Indicator 1: Number of districts contracted out for management of services
Value
(quantitative
or qualitative)
0 6 4 4
Date achieved 04/12/2012 06/30/2015 08/12/2015 10/31/2015
Comments
(including %
achievement)
Not achieved against original target and achieved against revised. Out of six districts,
the contractor for one district (D. I. Khan) refused to sign the contract due to the short
implementation period. Hence, the target value was decreased in ISR No. 8 from 6 to 5.
Eventually, another district (Kohistan) was bifurcated during implementation, and the
contract ended in one year. The final target was again changed from 5 to 4 in ISR No.
9. Thus, 66% was achieved against the original target (4/6) and 100% met against the
revised target (4/4).
Indicator 2: Health personnel receiving training (number) – added in ISR as core indicator
Value
(quantitative
or qualitative)
0 100 1,200 1,365
Date achieved 04/12/2012 06/30/2015 11/16/2015 04/30/2015
Comments
(including %
Surpassed. The EPP included a different indicator related to training, as follows:
“Training needs assessment and strategy for the district completed within six months
v
achievement) from the contract date, % of trainings conducted according to plan”. At the first ISR,
this indicator was modified to “Health personnel receiving training (number).” This
was a core indicator monitored by the team throughout implementation, with the
original target of 100%. ISR No. 8 stated that 1,365 persons were trained out of 2,290.
The end target of 1,200 showed up for the first time in the final ISR (No. 9).
Indicator 3: Percentage of hubs established and assessed as fully functioning by the
Department of Health
Value
(quantitative
or qualitative)
0 100 – 100
Date achieved 04/12/2012 06/30/2015 – 04/30/2015
Comments
(including %
achievement)
Achieved.
Indicator 4: Health facility utilization rate: visits per person per year
Value
(quantitative
or qualitative)
0.50 1.00 – 0.54
Date achieved 04/12/2012 06/30/2015 – 04/30/2015
Comments
(including %
achievement)
Not achieved.
Indicator 5: Health facility utilization rate (HFUR) by gender – added in 2014
Value
(quantitative
or qualitative)
– 1.00 – 0.54 Male
0.63 Female
Date achieved – 06/30/2015 – 04/30/2015
Comments
(including %
achievement)
Not achieved. The HFUR disaggregated by gender was newly added at the 2014
restructuring. But no disaggregated baseline data was available at appraisal nor
provided during implementation.
Indicator 6: Health facilities reconstructed, renovated, and/or equipped (number) – core
indicator
Value
(quantitative
or qualitative)
0 20 10 126
Date achieved 04/12/2012 06/30/2015 06/28/2014 04/30/2015
Comments
(including %
achievement)
Achieved. The EPP did not include baseline or target for this indicator. The first ISR
set the baseline of 0 and target value of 20. At the 2014 restructuring, the target value
was reduced to 10.
Indicator 7: Number of district headquarters hospitals refurbished
Value
(quantitative
or qualitative)
0 3 – 1
Date achieved 04/12/2012 06/30/2015 – 04/30/2015
Comments
(including %
achievement)
Not achieved.
Indicator 8: Health facilities adequately refurbished – core indicator
vi
Value
(quantitative
or qualitative)
0 project targeted
facilities 20 52
Date achieved 04/12/2012 06/30/2015 05/21/2013 04/30/2015
Comments
(including %
achievement)
Not achieved against original target and achieved against revised. The EPP entered the
target in percentage, but the first ISR states 100 in number. After the 2013 Mid-term
Review, the target was then revised to 20 in ISR No. 3. The project achieved 260% of
the revised target (52/20).
Indicator 9: Timely disbursement of funds to a consultant/nongovernmental organization
implementing contracting out
Value
(quantitative
or qualitative)
n.a. 90% 100% 25%
Date achieved – 06/30/2015 06/28/2014 10/30/2015
Comments
(including %
achievement)
Not achieved. At the 2014 restructuring, the target was increased. Until ISR No. 8 this
indicator was performing (75%) but it deteriorated toward the end of the project. 27%
was achieved against the original target (25/90) and 25% against the revised target
(25/100).
Indicator 10: Biannual meetings held for the Provincial Steering Committee
Value
(quantitative
or qualitative)
0 2 – 2
Date achieved 04/12/2012 06/30/2015 – 04/30/2015
Comments
(including %
achievement)
Achieved.
Indicator 11: Number of health facilities submitting monthly reports on time to the district
Value
(quantitative
or qualitative)
50 90 – 90
Date achieved 04/12/2012 06/30/2015 – 10/30/2015
Comments
(including %
achievement)
Achieved.
Indicator 12: Establishment within two months from the contract date and operationalization of
the District Health Management Team
Value
(quantitative
or qualitative)
1 6 4 4
Date achieved 04/12/2012 06/30/2015 11/12/2015 11/30/2015
Comments
(including %
achievement)
Not achieved against original target and achieved against revised. Out of six districts,
the contractor for one district (D.I.Khan) refused to assume the responsibility due to the
short implementation period. Another district (Kohistan) was bifurcated during
implementation, and thus the contract ended in one year. The target value was changed
at the 2014 restructuring from 6 to 4 in ISR No. 6. Thus, 66% was achieved (4/6) and
100% met (4/4).
vii
G. Ratings of Project Performance in ISRs
No. Date ISR
Archived DO IP
Actual Disbursements
(US$, millions)
1 08/28/2012 Moderately Unsatisfactory Moderately Unsatisfactory 3.00
2 03/25/2013 Moderately Unsatisfactory Moderately Unsatisfactory 3.00
3 05/21/2013 Unsatisfactory Unsatisfactory 3.00
4 10/16/2013 Unsatisfactory Moderately Unsatisfactory 3.00
5 02/17/2014 Unsatisfactory Moderately Unsatisfactory 3.00
6 06/28/2014 Unsatisfactory Moderately Unsatisfactory 3.00
7 11/23/2014 Moderately Satisfactory Moderately Satisfactory 4.67
8 05/21/2015 Moderately Satisfactory Moderately Satisfactory 4.67
9 12/11/2015 Moderately Unsatisfactory Moderately Unsatisfactory 5.85*1
Note*1- At project closure, the actual disbursement was US$5.85 million, however, with the refund of ineligible
expenditure of US$79,422 in April 2016, the final actual expenditure is US$5.77 million.
H. Restructuring (if any)
Restructuring
Date(s)
Board-
Approved
PDO Change
ISR Ratings at
Restructuring
Amount
Disbursed at
Restructuring
in US$,
millions
(1) Reason for Restructuring and
(2) Key Changes Made DO IP
06/10/2014 n.a. U MU 3.00 (1) Extremely slow implementation
with major delays in procuring
contractors;
(2) Partial cancellation of US$5.8
million; deletion of Component 2
(US$1 million); change in the Results
Framework, and change of the
implementing unit from the Health
Sector Reform Unit to a new Project
Management Unit with a Project
Coordinator, with oversight from the
Director General Health Services and
requisite financial and administrative
powers.
06/29/2015 n.a. MS MS 4.67 (1) Contracting out started but its
implementation was slow due to
inefficiency at the Project Management
Unit, resulting in delay in payments.
(2) Extension of the Closing Date for
four months until October 31, 2015.
10/28/2015 n.a. MS MS 5.28 (1) To match with the Closing Date of
round 1 of the KP/FATA/Balochistan
Multi-Donor Trust Fund (blanket
extension of the Closing Date for the
viii
Restructuring
Date(s)
Board-
Approved
PDO Change
ISR Ratings at
Restructuring
Amount
Disbursed at
Restructuring
in US$,
millions
(1) Reason for Restructuring and
(2) Key Changes Made DO IP
ongoing projects under the MDTF-1
was carried out by its Secretariat).
(2) Further extension of the Closing
Date by six weeks until December 12,
2015.
I. Disbursement Profile
Note: The original disbursement estimate is not available in the system.
1
1. Project Context, Development Objectives and Design
1.1 Context at Appraisal
1. Country and provincial context. The Government of Pakistan (GoP) faced an emergency
of historic proportions at the time of project preparation. When the war in Afghanistan intensified,
militant groups penetrated into the border area of Pakistan. In 2009, a major military operation was
launched by the GoP in the Khyber Pakhtunkhwa (KP) Province and the Federally Administered
Tribal Areas (FATA) to root out local pockets of militants. Starting from the Swat Valley,
bordering the tribal areas, the Government’s operations gradually moved westward. This offensive
resulted in significant damage to physical infrastructure and services and led to a large number of
temporarily displaced persons (TDPs). An estimated three million people were displaced in KP
and FATA in 2009 alone. About 7 percent of the displaced families moved into camps and the
remaining TDPs occupied schools or public buildings or moved in with host families, mostly in
the Swabi and Mardan Districts of KP. The crisis affected not only the TDPs but also those who
remained in the original locations, some of whom were just as poor and vulnerable as the TDPs.
Compounding this, the 2010 floods caused further destruction, large-scale internal
migration/displacement, and massive loss of livelihoods. Even after the successful completion of
military operations, large parts of FATA and KP failed to see a return of major economic activity.
Though Pakistan as a country was not officially categorized as an “IDA Fragile and Conflict-
affected State” by the World Bank, the recent World Bank Board Paper on IDA18 on Fragility,
Conflict, and Violence (FCV) in May 2016 recognizes that FCV are increasingly affecting middle-
income countries such as Pakistan and Nigeria, which display “pockets of fragility” at sub-national
level.1
2. Prior to 2008, health indicators for KP had been gradually improving over time but
remained poor in comparison to neighboring countries in the region. Intra-provincial inequities in
service provision and health status were of particular concern the government. Health facilities in
KP lacked equipment, medicines, and other essential supplies. The frequent and continuous
emergencies/crises faced by the province severely impacted health care provision. Militants
attacked facilities and carried out vandalism (theft of expensive equipment), killings, and
kidnappings of health personnel. Provision of health services was also hampered by the lack of
qualified personnel, vacant posts, and high levels of absenteeism. The population of the province
was not satisfied with the quality of health services delivered by the public sector institutions. For
example, only 8 percent of parents of children with diarrhea visited public sector first-level care
facilities (Basic Health Units (BHUs)) and Rural Health Centers (RHCs)) as against 64 percent of
1 The World Bank. 2016. “IDA18 – Special Theme: Fragility, Conflict, and Violence”. Washington, DC. USA
(http://imagebank.worldbank.org/servlet/WDSContentServer/IW3P/IB/2016/06/03/090224b084391d73/1_0/Render
ed/PDF/IDA18000specia0onflict0and0violence.pdf)
2
parents visiting private practitioners (Pakistan Social and Living Standards Measurement Survey
(PSLM) 2007–08).
3. It is also important to note that Pakistan was undergoing significant political changes with
emphasis on devolution of authority and provincial autonomy. The 18th Amendment to the
Constitution in 2010 devolved authority from the federal government to the provinces in about 40
areas, including health. The federal government’s role to manage and implement health programs
was reduced, while the provinces had to take on a new leadership role in an expanded mandate in
sectors where they had limited capacity and experience.
4. Rationale for Bank assistance. The proposed World Bank assistance for the project was
justified as it was fully supporting key strategies of post-crisis assistance in KP identified by the
government as well as development partners. In the aftermath of the militancy crisis in KP and
FATA, in September 2010 (following the earthquake), a Post-Crisis Needs Assessment (PCNA)
undertaken by a donor consortium outlined strategic directions and priorities to assist with crisis
management and mitigation. The assessment identified key crisis drivers and the subsequent
priority areas that needed to be addressed to support a coherent and durable peace-building
strategy, including: (a) enhancing the responsiveness and effectiveness of the state to restore
citizen trust; (b) stimulating employment and livelihood opportunities; (c) ensuring provision of
basic services; and (d) carrying out counter-radicalization and reconciliation. A World Bank-
administered Multi-Donor Trust Fund (MDTF) for the first round (MDTF-1) was established at
the request of the GoP and development partners to respond to the crisis in KP, FATA, and
Balochistan.2 The MDTF served as a funding mobilization mechanism for the implementation of
the PCNA3 and also provided a coordinated financing mechanism for the government’s activities
as well as investment projects and programs in three post-crisis provinces/areas. The project, one
of 11 MDTF-1 projects across nine sectors, was prepared to support KP under the following
MDTF-1 strategy pillars: Pillar 1 - Restoring Damaged Infrastructure and Disrupted Services;
Pillar 2 - Improving Local and Provincial Service Delivery; and Pillar 4 - Capacity Building and
Institutional Strengthening.
5. Furthermore, the project was in line with Pillar 2 of the Country Partnership Strategy (CPS)
for FY2010–13, “Improving Human Development and Social Protection,” which recognized the
need to enhance delivery of health, nutrition, and population services. The design of the project
also took into account key concerns identified in the CPS for the health sector, namely: (a) better
governance and management of the delivery of basic health services; (b) coverage and quality of
essential health services, especially in disadvantaged areas; and (c) developing service delivery
2 The PCNA did not include Balochistan; Balochistan was added to the scope of the MDTF later and a Balochistan
Development Needs Assessment was carried out under the MDTF in 2012.
3
There were four strategic pillars under the MDTF: Pillar 1: Restoring Damaged Infrastructure and Disrupted
Services; Pillar 2: Improving Local and Provincial Service Delivery: Pillar 3: Supporting Livelihoods and Creating
Employment Opportunities; and Pillar 4: Capacity Building and Institutional Strengthening. In the first phase, ten
donors, that is, Australia, Denmark, European Union, Finland, Germany, Italy, Sweden, Turkey, the United
Kingdom, and the United States, have contributed to US$164.9 million to finance stand-alone projects or program
activities, including those co-financed by the government, bilateral, or multilateral agencies. For details, see the
website https://www.pakistanmdtf.org/index.php.
3
models that would help the country sustain service delivery levels when these systems came under
duress due to natural and man-made disasters by providing support for emergency services at the
community and facility level.
1.2 Original Project Development Objectives (PDOs) and Key Indicators
6. The Development Objective of the project, as stated in the Emergency Project Paper (EPP)
as well as in the Grant Agreement, was “to improve the availability, accessibility, and delivery of
primary and secondary health care services at the district level.”
7. The PDO indicators were:
(a) People with access to a defined basic package of health, nutrition, and reproductive
health services
(b) Percentage of children with severe acute malnutrition provided adequate nutrition
services
(c) Contraceptive prevalence rate for any modern method
(d) Births (deliveries) attended by skilled health personnel
(e) Community satisfaction with health care services delivery by the public sector
1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and
Reasons/Justification
8. The PDO was not changed. However, as a result of the 2014 restructuring which downsized
the project scope, the final target values for three out of the five PDO indicators were decreased.
Also, two intermediate outcome (IO) indicators, namely, ‘health personnel receiving training
(number)’ and ‘health facility utilization rate disaggregated by gender,’ were added.
1.4 Main Beneficiaries
9. The Project sought to benefit the populations of the six districts of KP affected by the crises
(militancy and floods). The total population of the six target districts, based on the 2008 Census
data, was 3,041,540: Battagram (307,278), Buner (506,048), D. I. Khan (852,995), Dir Lower
(717,649), Kohistan (472,570), and Tor Ghar (185,000). The selection of target districts was based
on criteria agreed on between the World Bank team and the Department of Health (DoH), including
crisis-affected districts, poor health- and socioeconomic indicators, and whether other funding was
available from the provincial Annual Development Plan or other development partners.
1.5 Original Components (as approved)
10. The project consisted of three components:
Component 1: Revitalizing Health Care Services (total estimated cost - US$11.0 million)
(a) Reorganization of primary health care centers into hubs and delivery of a
comprehensive package of health care services.
(b) Outsourcing the management of all the facilities in hubs to private firms/non-
governmental organizations (NGOs) through a competitive process. The selected
4
firms/organizations were to be responsible for a comprehensive package of care to the
communities through application of the hub approach.
(c) Improvement in district headquarters hospitals to enable optimal functioning as
referral-level hospitals.
Component 2: Rehabilitation of Health Infrastructure (total estimated cost - US$1.0 million)
(a) Rehabilitation of health facilities damaged during the crisis to enable service delivery
(no new construction of infrastructure was proposed under this project).
Component 3: Establish and Operationalize a Robust Monitoring and Evaluation System at
the District and Provincial Levels (total estimated cost - US$4.0 million)
(a) Strengthening and operationalization of monitoring and evaluation (M&E) systems to
guide project implementation at the district level and dissemination of results through
province-wide analysis.
(b) Supporting capacity building and operationalization of a District Health Information
System (DHIS) and periodic third-party evaluation of the project in selected districts
including baseline and end line surveys to assess results.
1.6 Revised Components
11. Due to a very slow startup and poor implementation, the project was restructured on June
4, 2014 (see detailed explanation in Section 2 below). US$2.06 million was cancelled from
Component 1, decreasing the total component costs for the district management contracts to
US$8.94 million for the shorter implementation period of contracting out. Component 2 involving
civil works was cancelled entirely. Component 3 funding was also reduced from US$4.0 million
to US$1.26 million as operationalization of DHIS did not happen due to the dissolution of the
M&E Cell. The remaining activities were the operationalization of M&E systems, both at
provincial and district levels through a Provincial Steering Committee and District Health
Management Team (DHMT).
1.7 Other Significant Changes
12. Substantial project restructuring after a long period of problematic status (June 4,
2014). In addition to the changes in components mentioned above, the restructuring entailed the
following: (a) replacing the implementing unit, the Health Sector Reform Unit (HSRU) in DoH
with a new Project Management Unit (PMU) under the direct administrative control of the
Secretary of Health and with requisite proper financial and administrative powers; (b) revision of
the Results Framework by adding one IO indicator (health facility utilization rate by gender),
totaling 12 IO indicators and new target values for three out of the five PDO indicators and three
out of the 11 IO indicators; and (c) changing the definition of “incremental operating costs” to
include the salary and allowances of the project coordinator as a civil servant.
13. Restructuring to extend the Closing Date (June 29, 2015). The processing of the 5-year
extension of the MDTF-1 (Closing date of December 31, 2015) was ongoing at the time of
restructuring but had encountered delays. A decision was, therefore, made by World Bank
management to grant a 4-month extension for all ongoing MDTF-1 supported projects to October
31, 2015 to allow sufficient time to complete the MDTF-1 extension process.
5
14. Restructuring to extend the Closing Date (October 28, 2015). As an extension of the
MDTF-1 continued to be delayed, all MDTF projects were extended a second time by six weeks
to December 12, 2015 through an ‘omnibus’ extension processed by the MDTF Secretariat
internally, which was approved on October 28, 2015. Finally, after the MDTF-1 five-year
extension was processed and became effective in November 2015, the Economic Affairs Division,
Government of Pakistan, and the MDTF Secretariat decided that any further extensions of projects
under the MDTF-1 beyond December 12, 2015 would be handled on a case-by-case basis,
depending on their implementation status. A decision was then taken that this project would be
closed without further extension because of its previous delays, lack of counterpart funding, delays
in payments, and an overall unsatisfactory implementation status (see details in Section 2).
2. Key Factors Affecting Implementation and Outcomes
2.1 Project Preparation, Design, and Quality at Entry
15. The project, as well as all other projects financed by the MDTF-1, was prepared under
OP/BP 8.00 - Rapid Response to Crises and Emergencies. Its design, therefore, was a response to
emergency circumstances and took into account the significant risks that such circumstances
entailed.
16. Project design. The project design was well thought-out considering the prevailing
environment and constraints the project would have to face. Lessons from local and global
experience were taken into account. A number of pilot experiences had highlighted the
considerable potential of the contracting out mechanism for quickly improving access and
availability of primary health care services in underserved and remote districts as well as in a
fragile security context.
17. Two local experiences, in particular, heavily influenced project design, namely: the
Battagram4 model implemented in one KP district and the approach used by the People’s Primary
Health Care Initiative (PPHI). The World Bank had provided assistance to the Battagram model
through the Japan Social Development Fund (JSDF) under the ‘Revitalizing and Improving
Primary Health Care in Battagram District Project’ in the context of Pakistan’s 2005 earthquake
that damaged the country’s health infrastructure. A second initiative involved contracting with the
PPHI in 13 districts of KP for the management of BHUs. Both models employed contracting out
of health service delivery to NGOs, which had full autonomy with regard to financial matters and
authority to recruit to fill vacant positions. Health committees and quality improvement teams at
facilities were also established under both models to foster community involvement. The following
additional advanced features were included in the Battagram model: (a) greater financing and
hiring authority for a NGO/contractor than the PPHI had; (b) management of all health facilities
including dispensaries, mother and child health centers, BHUs, and RHCs; (c) inclusion of vertical
4 Battagram is an underdeveloped district in KP located in a mountainous setting with an estimated population of
361,000 (2004–05). Save the Children USA implemented the Battagram model with the financing from the JSDF.
6
programs and related indicators that focused on preventive services; and (d) a structured and
transparent system of performance-based incentives. These improvements arguably contributed to
the successful implementation of the Battagram model. The Battagram model thus had
demonstrated that a contracting approach was appropriate to the needs created by the post-
earthquake emergency as well as in areas where the provision of public services was disrupted due
to conflicts.
18. The Battagram model was, however, further refined for the project by incorporating lessons
learned from implementing a contracting out approach in the health sector in Afghanistan. The
Afghanistan model also involved contracting out NGOs for the delivery of health services in
unsecured areas. A key feature of the Afghanistan model, however, was the development of a
comprehensive health care package, a basic package of health services delivered to the entire
population. Accordingly, the project was designed to apply the Battagram model to a larger
number of KP districts, with the delivery of a comprehensive health care package for the entire
district population, similar to the Afghanistan approach.
19. The PDO statement was likely more complex than it needed to be for a project essentially
focusing on a single set of interventions. The use of three terms, i.e., ‘availability’, ‘accessibility’,
and ‘delivery’ of health care services was meant to demonstrate that the project was attempting to
ensure that all these aspects were receiving focus. However, none of these terms was clearly
defined and there was no clear alignment of PDO indicators to the specific PDO terms; this made
measuring the achievement of the PDO more challenging than was necessary (see Section 3.2).
20. Risks and mitigation. The risks and their mitigation measures associated with the project
preparation and implementation were adequately identified in the Operational Risk Assessment
Framework (ORAF) in the EPP. The highlighted risks were: fragile security, insufficient capacity
in the Department of Health (DoH), and weakness in the monitoring and evaluation (M&E)
capacity of the DoH. Likely substantial delays in the approval of the Planning Commission Form
I (PC-I) within the Government was not identified as a risk. The proposed mitigation measures for
each risk were the following: for security risks, as mentioned earlier, contracting out service
delivery to an NGO/contractor had been shown to be appropriate when public service provision
was disrupted and mobility was limited, particularly in post-conflict and fragile environment. In
order to complement limited capacity in the DoH, a multilayered supervision mechanism was
incorporated, including the use of an independent third party for data validation and periodic
supervision, monitoring by district health offices, and community involvement, along with the
DoH’s regular supervision. In addition, the World Bank team provided intensive hands-on
fiduciary support to the DoH in recognition of the DoH’s lack of previous experience with a World
Bank project. For the M&E capacity risk, the project specifically supported capacity building and
operationalization of the DHIS under Component 3 and the establishment of the DHMT under the
contracting-out sub-component in Component 1. The project also planned for a baseline survey in
the first year of implementation because of the lack of baseline data for many indicators at project
appraisal.
21. Unfortunately, all identified and not identified (possible delays with approval of PC-I)
major risks cited above materialized during implementation. While the proposed mitigation
measures were appropriate, their effectiveness was largely compromised by the fragile post-crisis
operating context or these measures did not have a chance to be implemented. For instance, a
7
multilayered supervision mechanism was proposed in the EPP to compensate for the inadequate
capacity of DoH. The World Bank task team repeatedly visited Peshawar providing hands-on
training and guidance to DoH, but it was not sufficient to quickly and fully address the lengthy
process for procurement and subsequent release of government funds. Other mitigation measures,
such as hiring a third party for data validation, community involvement and monitoring by district
health offices, could be applied only after the initiation of implementation of contracting of health
services, and were therefore not applicable to resolving the DoH capacity constraints.
22. Project preparation. Project preparation took three months, with the Project Concept
Note approved on April 15, 2011, appraisal completed by June 30, 2011 and negotiations
completed on July 21, 2011. Regrettably, the World Bank’s regional vice president approval was
only granted nine months later, on April 12, 2012, as a result of significant delays within the
Government in the approval of the PC-I, which was a condition for World Bank approval. In light
of the fact that the MDTF-1 end date was set as December 31, 2015, the project period was
essentially shortened to three years from four as a result of these delays by the GoP.
23. Government’s commitment. The buy-in from the GoKP was mixed at project preparation.
On one hand, prior positive experience from local contracting out initiatives had raised GoKP
interest and confidence to expand the model to the entire province. The GoKP’s commitment was
apparent from its large share of promised counterpart financing, i.e., provision of US$45 million
of the total project cost of US$61 million for the recurrent costs of running the health facilities in
the target districts. It was widely recognized at the time that this was the only project leveraging
such large counterpart financing, which in turn became a precedent for other MDTF-1 projects. In
other words, the success of the contracting out model in the project was dependent on the
availability of the government counterpart financing due to its heavy reliance. On the other hand,
despite this commitment, a similar commitment was not evident when it came to the internal
approval process of PC-I, which took nine months, though it is understood that the approval
process was long pending at the federal level. Though the World Bank accelerated its internal
processing using the emergency procedures, the Government did not respond with the same level
of urgency. Furthermore, the abolishment of the M&E Cell in DoH at the time of project
effectiveness also put into question the GoKP’s willingness to implement the project.
2.2 Implementation
24. The project followed emergency operational procedures under OP/BP 8.00 that allowed
for the completion of implementation readiness criteria after the start of implementation. Since
the start of implementation was severely delayed as a result of the approval delay, most
departments in the Government could not start any activities. Importantly, this long waiting period
for PC-I approval was not utilized to increase the readiness of project implementation. Some of
the key activities required for project start-up included placement of key project staff, and
preparation of an Operations Manual and an Environmental and Social Management Plan (ESMP).
Regrettably, no work was done to advance any of these activities until implementation started.
Also, the implementation bottlenecks/risks identified in the ORAF were not addressed while
waiting for PC-I approval. In retrospect, a stronger start of implementation would have been
possible if there had been a more proactive and a less bureaucratic approach taken to enhance
implementation readiness, such as limited capacity in DoH, while waiting for PC-I approval.
8
25. Limited capacity in DoH/HSRU and unfamiliarity with World Bank procedures led to
extensive implementation issues from the beginning. These included: limited contract
management experience, low enforcement of guidelines and rules, lack of internal communication
within DoH, and lengthy processing for routine project tasks. This was further exacerbated by
continuous movement of key staff and frequent changes in leadership and subsequent
unpredictability on the direction of the contracting out approach. Such instability in leadership and
the challenging political economy slowed down day-to-day decision making due to risk aversion.
All these problems significantly delayed each step necessary for the completion of the contracting
out of health services and hiring of an M&E firm. For instance, the finalization of the request for
proposal for contracting out was done by February 2013 with a six-month delay. Other
procurement steps such as the establishment of a technical evaluation committee, finalization of
contracts, and signing of contracts, faced similar delays, in spite of repeated visits and warnings to
the Government by the World Bank task team.
26. After agreed milestones/actions were missed several times, the World Bank considered the
option of possible project suspension and even cancellation. The World Bank task team had also
recommended project restructuring early on but it was not agreed to by the GoKP at that time. At
the Mid-Term Review in April 2013, the World Bank formally questioned, in its management
letter to the GoKP, the viability of the project and the GoKP’s willingness to implement the project.
This letter warned about a possible suspension and a subsequent cancellation of the project, if the
outstanding tasks, such as submission of a revised ESMP, progress report on the procurement
process, and TORs for an M&E firm as an alternative arrangement for the abolished M&E Cell,
were not addressed by the agreed dates. The project had been already rated as Moderately
Unsatisfactory for a year and was further downgraded to Unsatisfactory. Other than the initial
disbursement of US$3 million made to the Designated Account, there was no disbursement for
more than a year, and the utilization of funds was extremely low.
27. Project restructuring was finally undertaken in June 2014 to address poor project
implementation. This restructuring was explicitly supported by the Chief Minister of the KP
Province in March 2014 who promised to ensure successful and unhindered project execution. In
addition to the partial cancellation of US$5.8 million, the major revision was to change the project
implementing entity to remove administrative hurdles and internal bureaucracy that had caused
delays in procurement, from the HSRU to an independent and new Project Management Unit
(PMU), set up specifically for project implementation, headed by a full-time Project Coordinator,
with oversight from the Director General of Health Services of the DoH. The Project Coordinator
was given the full authority that had previously been assigned to the Project Director at the HSRU.
However, importantly, the PMU had to build up its capacity and had no prior experience with
World Bank-financed operations.
28. The June 2014 restructuring led to some improvements in implementation. Service
provision contracts were finally signed between the DoH and NGOs/contractors in five out of the
six districts, with full expectations for a later extension of the implementation period. The
successful bidder for the sixth district (D.I. Khan) declined to sign the contract due to a shortened
implementation period from the original bidding documents. In the other five districts, the
contracting out activities finally started, showing results on the ground. Lastly, due to the
bifurcation of one of the target districts, Kohistan, and subsequent weakened district administrative
9
capacity, the contractor faced difficulties in continuing operations. As a result, that contract was
allowed to expire after the initial period of one year.
29. Even after the contracting out of services started, the project continued to face a set of
problems. These included: (a) delays in payments to the contractors; (b) lack of contract
management capacity; (c) communication issues between the PMU and contractors; and (d) delays
in reporting from the contractors. The most critical issue was substantial delays in payment
processing for different suppliers and the NGOs/contractors because of layered reviews and
comments sought for the payments within the DoH. Delays of more than nine months in
government budget transfer to the contractors resulted in them either stopping their activities in
the districts or using their own funding to continue service provision, with expectations of later
reimbursement and eventually extensions of the contract period. The DoH also made
reimbursement of the MDTF funds equally difficult for the NGOs/contractors; they claimed that
there were more than 15 steps to go through to receive the MDTF funds. The implementation
problems were largely due to lack of capacity in the PMU, which was led by a part-time Project
Coordinator without full administration power and staff who received little support from the DoH.
Numerous court cases over staff appointments in the HSRU/PMU led to the lack of a fully
empowered Project Director/Coordinator. Frequent turnover in the Secretary Health position
throughout implementation – seven secretaries in three years – also had a detrimental impact on
project performance.
30. There is some evidence that once service provision started, real project benefits emerged
on the ground – availability and accessibility of services improved and large segments of the
population started utilizing services. Data suggests that the project approach had the potential to
quickly respond to people’s needs by improving health service delivery. The majority of the IO
indicators in the target districts started to show progress and appeared to be on the right trajectory
to achieve the PDO. There was also support from the local political leadership and administration
in the districts. As discussed above, the project Closing Date was extended twice, in June and
October 2015, for total of five and half months, initiated by the MDTF Secretariat, due to the
delayed extension process for MDTF-1. Throughout the extension process, World Bank
management’s view towards the project was favorable, as visible project benefits were being
observed on the ground. The ISR archived in May 2015 rated both IP and DO as Moderately
Satisfactory. After the 5-year extension of the MDTF-1 was granted in November 2015, a decision
was made on a case-by-case basis for each ongoing project regarding further extension, depending
on implementation status. Despite strong requests to continue the project from politicians, district
authorities, and beneficiaries in the target districts (in Battagram and Tor Ghar Districts people
came out for a protest against the closure of the project), the World Bank decided to close the
project. By this time, the last ISR drafted in November 2015 already recognized that it was no
longer satisfactory and proposed the downgrading to Moderately Unsatisfactory, noting
continuing extensive delays in the release of government funds to the NGOs/contractors and
uncertain commitment by the GoKP. On this basis, the World Bank submitted a note to the MDTF
Advisory Committee in late November 2015, proposing that no extension be granted to the project.
The Advisory Committee, representing the federal government, the three provincial governments,
and all the contributing donors endorsed this way forward.
10
2.3 Monitoring and Evaluation, Design, Implementation, and Utilization
31. The overall rating for M&E is Modest.
32. M&E design. The five PDO indicators and 11 IO indicators were originally selected for
the project to measure availability, accessibility, and delivery of primary and secondary health care
services at the district level. The World Bank team also included three IDA core indicators in the
Results Framework, namely, people with access to a defined health package (PDO), health
personnel trained, and health facilities renovated. The training-related IO indicator was originally
phrased as “Training needs assessment and strategy for the district completed within six months
from the contract date, % of trainings conducted according to plan”; however, it was changed to
“Health personnel receiving training (IDA core indicator) during the first ISR. During the June
2014 restructuring, one IO indicator (health facility utilization rate by gender) was added, totaling
12 IO indicators. Sources of data for the indicators were to be: (a) project progress reports, (b)
DHIS; and (c) surveys. Source of data for the majority of indicators were either from (a) or (b)
mentioned above. Three PDO indicators were to rely on (c) survey data, namely, PDO indicator 3
- skilled birth attendance (SBA), PDO indicator 4 - contraceptive prevalence rate (CPR), and PDO
indicator 5 - community satisfaction. As was implemented in other provinces of Pakistan, the
provincial-level Multi-Indicators Cluster Survey (MICS) and the Pakistan Social and Living
Standards Measurement (PSLM) Survey were the data source. The use of population-level
indicators, such as SBA and CPR, turned out to be too ambitious in the post-crisis context. The
baseline data for some indicators, such as PDO indicators 1 and 2 as well as health facility
utilization rate (IO indicator), was not available at appraisal; therefore, data collection was planned
in the first year of implementation.
33. In recognition of weaknesses in the existing M&E capacity and system in the DoH and
security constraints, the following multilayered M&E strategies were proposed under Component
3: (a) strengthening the DoH M&E Cell; (b) establishing DHMTs to review the progress at the
district level every quarter; (c) hiring an independent consultancy firm for baseline, midterm, and
end line data collection; (d) strengthening the DHIS and external validation of DHIS data; and (e)
having periodic supervision by a third-party consultant.
34. M&E implementation. Despite a well-designed M&E approach, its implementation was
uneven at best. Abolishment of the M&E Cell in the DoH at the time of project effectiveness was
a major stumbling block. The M&E Cell had been envisaged as a ‘clearing house’ for all
monitoring data that would be working closely with the HSRU to ensure that the contractors were
performing according to their contracts. Subsequent proposals by DoH to hire an M&E firm to
assist the HSRU in establishment of a baseline and quarterly monitoring of the project indicators
were not effective as delays in hiring meant the firm came on board only a few months before
project closure. Baseline data collection never took place during implementation. Without baseline
data, the Bank team struggled to set and revise the target values for each indicator. Also, the DHIS
was not fully operationalized as envisaged due to the dismantling of the M&E Cell. Moreover, the
MICS, the population-based survey, did not take place during the project implementation period.
As a result, much of the data for indicators were not available to assess the progress towards PDO
and thus adjust indicator targets during project implementation.
11
35. The target values for three out of five PDO indicators and three out of 12 IO indicators
were formally revised at the June 2014 restructuring, adjusting them to the downsized scope of the
project. In retrospect, the Results Framework could have been also simplified during the
restructuring as the baseline data collection was not done even two years after project
effectiveness. For one PDO indicator and four IO indicators, the target values were modified only
in ISRs without going through the restructuring process (see details in the Datasheet).
36. M&E utilization: Project data were in general not used to strengthen M&E capacity and
inform decision-making in the DoH. Despite limitations around M&E in the DoH, however, the
project did manage to use some data, mainly drawn from DHMT reports and the findings of third-
party surveys. These data pointed to some improvement in the use of services.
2.4 Fiduciary and Safeguard Compliance
37. Financial Management (FM) is rated as Moderately Unsatisfactory. As a whole, FM
arrangements were implemented effectively. An experienced FM team remained onboard
throughout the project life. An adequate financial system was in place that provided accurate and
timely financial information. Financial reports required under the legal agreement were submitted
to the Bank on time, which were reviewed and found acceptable. External audits of the project
were conducted by the Supreme Audit Institution, which provided reasonable assurance that
project funds had been used for intended purposes. Internal audits were conducted by KPMG—
the internal audit reports did not highlight any major internal control exceptions. Delays, however,
occurred in release of counterpart funding, which severely constrained the project’s progress and
had the NGOs/contractors use their own funding to continue service provision. These delays were
a result of a lack of understanding of the funds flow mechanism by the district officials responsible
for transfer of government funds to the NGOs/contractors.
38. Procurement. Extensive delays in the procurement of the contracting firms was the main
obstacle to the overall progress of the project. Each step in the procurement process, such as
preparation and issuance of Expressions of Interest, preparation of Requests for Proposals,
constitution of the technical evaluation committee, and negotiations with the contracting firms,
took significant amounts of time, resulting in repeated delays in the achievement of agreed target
dates. In recognition of limited experience with World Bank operations in the DoH, the World
Bank team provided frequent visits and hands-on instructions to the HSRU regarding procurement
actions. Major delays, however, persisted until after the restructuring in June 2014, when it was
decided to transfer implementation functions from the HSRU to a new PMU. Under the new PMU,
five out of six contracts were signed. However, lack of experience in contract management in the
PMU, along with a part-time Project Coordinator without full sanctioned authority, hindered the
solution of day-to-day issues and smooth communication flow among stakeholders. Procurement
of an M&E firm also faced a significant delay. The World Bank evaluation of procurement of
goods found shortcomings in the procurement practice at the PMU, including delays in the
evaluation process, poor procurement record maintenance, and inconsistency in the Requests for
Quotation. Procurement during project implementation is, therefore, rated as Unsatisfactory.
39. Safeguards. The project was classified as Environmental Category ‘B’, in accordance with
OP 4.01. No other safeguard policies were triggered. The GoKP prepared an ESMP for the project
in accordance with the local regulatory requirements as well as World Bank safeguard policies.
12
The Environmental and Social Screening and Assessment Framework was prepared by the World
Bank in accordance with the provisions of OP/BP 8.00 for emergency operations. The ESMP also
included similar plans and guidelines, particularly for health care waste management, to be
implemented during the operation of the selected primary health care facilities in the province.
40. The overall performance rating of the environmental and social safeguards is rated as
Moderately Unsatisfactory. Implementation of health care waste management at the primary health
care facilities was only partially implemented despite some successful initial steps, including
formulation of a waste management team within each health care facility, awareness raising and
training of relevant staff, and segregation of different types of wastes within the health care
facilities. The key weakness in the ESMP implementation pertained to the final disposal of the
wastes from the health care facilities, and none of the facilities covered under the project could
manage to establish any appropriate system/mechanism to safely dispose of the infectious waste.
Another weakness of the project was the irregular preparation of the ESMP quarterly progress
reports, and hence the safeguard progress was not always communicated to the Bank regularly and
on time. The lack of regular reporting further exacerbated weak safeguards implementation as the
task team was unable to visit the project sites in view of the security concerns, at least during the
early years of project implementation.
41. With regard to social safeguards, the EPP in its annex identified several constraints to
health service delivery in KP, such as demand-side barriers (cost, perceived quality of services,
poor accessibility, social barriers) as well as supply-side barriers (low morale of staff and shortage
of female doctors). However, it concluded that these shortcomings would be addressed by the
project. Selection of target districts was deemed appropriate, as it was based on appropriate
socioeconomic and health indicators. The EPP had recommended close monitoring of health
service utilization by the marginalized population such as the poor and women. Social safeguards
however appeared to have received little attention during implementation, as there is no reference
in ISRs or Aide-Memoires, even after the contracting out started. Establishment of a grievance
redressal mechanism was substantially delayed, but it was featured in all four districts where
contracting out took place.
2.5 Post-Completion Operation/Next Phase
42. It can be noted that despite serious implementation problems, politicians and the GoKP
appear to have recognized the increased satisfaction of the targeted population with the health
service provision through the contracting out process implemented by the project. Therefore, the
GoKP, even after different political parties came to power, has decided to expand the contracting
out initiative to all KP districts from April 2016 for 18 months, to be financed by the government
development budget.
3. Assessment of Outcomes
3.1 Relevance of Objectives, Design, and Implementation
43. Relevance of PDO: High. The PDO to improve the availability, accessibility, and delivery
of primary and secondary health care services at the district level was and remains pertinent to the
context in KP and the GoKP’s priorities.
13
44. The current CPS 2015–19 for Pakistan supports the priority of the GoKP to improve service
delivery, enhanced focus, and improved management of health, nutrition, population, and
sanitation services with good progress toward the Millennium Development Goals. The CPS also
seeks to address sources of fragility and conflict, with an emphasis on restoring trust between
citizens and the Governments of KP, FATA, and Balochistan through improved government
service delivery using the MDTF.
45. Relevance of Design: Substantial. The project’s planned activities were pertinent to the
post-conflict situation on the ground. The use of NGOs for the contracting out approach in delivery
of health services is deemed appropriate for such a fragile and difficult operating environment.
The project design was informed by previous local experiences in Battagram District as well as
the PPHI and further improved by incorporating the Afghan approach. The design provided
contractors with flexibility in management of health facilities both with regard to staffing and
logistics to ensure improved coverage of the assigned population and management and
administration of the existing community-based and outreach programs.
46. Relevance of Implementation: Modest. On the one hand, the selection of the
implementing entity appeared appropriate given that it was fully staffed and had been functioning
since 2002 with a reasonable track record of implementation and management of the health sector
reform agenda. However, given the emergency nature of this project and the relatively short
implementation period and the fragile context, the use of this entity was likely not appropriate, as
was proven during implementation. As mentioned previously, there were numerous and
continuous implementation challenges throughout the project despite the fact that the
implementing agency was changed to the PMU. The limited capacity at DoH/HSRU and the PMU
persisted and improvements made were not sufficient to reverse the damage done.
3.2 Achievement of Project Development Objectives
47. As discussed in previous sections, project implementation was severely delayed for the
first two years and essentially nothing was accomplished during this time. Following restructuring,
there was some progress but the time left for actual implementation of contracting out was too
short. Because the issue of delayed release of government funds to the contractors was unsolved,
a further extension of the Closing Date was not granted. Therefore, it was not possible for the
project to realize its full potential and thus achieve the PDO. Table 1 below summarizes progress
against the PDO level indicators. Clearly, achievements against the original project targets were
minimal, but after the scaling down of the majority of the original targets during the June 2014
restructuring, and the consequent initiation of contracting out, the project did achieve some partial
results. (Note: given that target values for many of the indicators were changed at restructuring,
the tables that follow are organized into Phase 1 and 2 periods to more clearly reflect changes
before and after restructuring.)
48. The overall achievement of the PDO assessed by the indicators in the Results Framework
is rated as Negligible and Modest in Phase 1 and 2, respectively. Out of five PDO indicators, two
were achieved in both Phase 1 and 2 (see Table 2 below). Out of the 12 IO indicators, eight were
achieved and four were not achieved either in Phase 1 or 2 (for details, see Annex 2).
14
Table 1. Summary of PDO/IO Indicators Achievements
Phase 1 - Achievement against
the Original Target
Phase 2 - Achievement against the
Revised Target
PDO (5) IO (12) PDO (5) IO (12)
Surpassed – HS (96%+) 2 0 2 2
Achieved – S (85–95%) 0 5 0 6
Partially achieved – MS (65–84%) 0 1 1 0
Not achieved – MU (41–64%) 0 6 0 4
Not achieved – U, HU (0–40%) 4 0 2 -
Percentage (%) achieved (HS or S) 2/5 = 40% 5/12 = 41% 2/5 = 40% 8/12 = 66%
Unsatisfactory Negligible Moderately Unsatisfactory
Modest
Note: HS = Highly Satisfactory; S = Satisfactory; MS = Moderately Satisfactory; MU = Moderately
Unsatisfactory; U = Unsatisfactory; HU = Highly Unsatisfactory
49. Assessment of the achievement of PDO. The PDO addressed three dimensions of health
services—its availability, accessibility, and delivery of primary and secondary health services.
Without a description in the EPP, the assessment of the achievement of the PDO in the ICR was
challenging due to the following reasons: (a) definitions of three PDO areas, i.e., availability,
accessibility, and delivery of health services, are somewhat overlapping, which can be defined in
different ways; (b) some PDO indicators could include more than one PDO area; and (c)
assignment of each indicator to the PDO areas could be also different depending on the definition
of each PDO area. With the above ambiguity in mind, the ICR attempted to assign all PDO and IO
indicators to each of three dimensions of the PDO to make the case for what has been achieved by
the project (see Table 2 and details below).
Table 2. Summary of PDO Achievement
PDO area PDO indicators
(achievement in Phase 1
/Phase 2)
Intermediate outcome
indicators
(achievement in Phase 1
/Phase 2)
Rating (only HS or S
considered to be
achieved)
PDO area 1:
Availability of health
services (readiness in
service provision in
the supply side)
#3 (skilled birth
attendance) - HS/HS
#4 (contraceptive rate) –
U/U
#2 (health personnel
trained) – S/S
#6 (health facilities
renovated) – S/S
#7 (No. of DHQ
hospitals refurbished) –
U/U
#8 (health facilities
refurbished) – MU/HS
Phase 1 – Modest PDO indicator = 1/2
IO indicator = 2/4
Total = 3/6(50%=MU)
Phase 2 - Substantial PDO indicator = 1/2
IO indicator = 3/4
Total = 4/6(67%=MS)
PDO area 2:
Accessibility of health
services (financial
accessibility, physical
accessibility, and
adoptability)
#1 (access to a defined health services) –
HS/HS
#2 (clinical nutrition
services provided –
secondary care) – U/MS
#4 and 5 (health facility
utilization rate) – U/U
Phase 1 - Negligible PDO indicator = 1/2
IO indicator = 0/1
Total = 1/3 (33%=U)
Phase 2 - Negligible PDO indicator = 1/2
IO indicator = 0/1
Total = 1/3 (33%=U)
PDO area 3: Delivery
of health services #5 (community
satisfaction) – U/U
#1 (No. of districts
contracted out) – MS/S
Phase 1 - Modest PDO indicator = 0/1
15
(perspectives of end-
users & change of the
‘delivery’ mode of
health services to the
contracting out
approach)
#3 (% of hubs
established) – S/S
#9 (timely disbursement
of funds) – U/U
#10 (provincial steering
committee) – S/S
#11 (health facilities
monthly reporting) – S/S
#12 (DHMT) – MS/HS
IO indicator = 3/6
Total = 3/7(42%=MU)
Phase 2 - Substantial PDO indicator = 0/1
IO indicator = 5/6
Total = 5/7(71%=MS)
50. ‘Availability’ of health services is defined to include services such as human resources,
facilities, and medicines. In the ICR, PDO indicator 3 – Skilled birth attendance (SBA) and PDO
indicator 4 – Contraceptive prevalence rate (CPR) are assigned to assess the level of availability
of health services achieved under the project. Both CPR and SBA are recognized as service
utilization indicators, provided availability of contraceptive methods and counseling for CPR and
trained medical personnel for SBA is secured. However, given “utilization” is not part of PDO,
these two indicators are assigned to "availability" for the assessment purpose, due to availability
as a prerequisite for service utilization.
51. PDO indicator 3 – Skilled birth attendance. This indicator is a measure of a health
system’s ability to provide adequate care for pregnant women.5 With the latest figure from the
PSLM 2014-15 at 48 percent, this indicator is considered to have surpassed the original and revised
targets. The release of data from the PSLM 2012-13 came too late for the 2014 restructuring for
more accurate setting of baseline and targets. In fact, using the district population estimates
(weighted) from the 1998 Census, the average rate of SBA for five districts in PSLM 2012-13 is
38 percent, which is already 15 percent higher than the original baseline of 24 percent. The PSLM
2014-15 points to a substantial improvement in SBA for five target districts at 48 percent. The
increase in target districts is higher than the overall improvement for rural areas of KP in the same
time period (see Figure 1 below). It should be noted that it is not possible to isolate the attribution
of the contracting out activities in the increase of SBA because the survey data such as PSLM and
MICS does not differentiate between sources of SBA, i.e., the public or private sector.
5 http://www.cpc.unc.edu/measure/prh/rh_indicators/specific/sm/percent-of-deliveries-attended-by-skilled-health
16
Figure 1. Comparison of skilled birth attendance in target five districts of KP
Source: PSLM 2012-13 and 2014-15
52. PDO indicator 4 - Contraceptive prevalence rate for any modern method. This indicator
measures a coverage of contraceptive use for family planning programs at the population level,
taking into account all sources of supply and all contraceptive methods. 6 In spite of its importance,
the use of the CPR for the project could have been avoided for many reasons. First, the increase in
CPR requires efforts from both the supply and demand sides (such as voucher program and
information campaign), as the uptake of family planning methods is heavily influenced by
religious beliefs, personal decisions, and/or fertility preference. The project was able to address
only the supply side through government facilities and Lady Health Workers. Second, with little
change in CPR in Pakistan in the last decade, a project with a short implementation period would
likely not have been able to boost the CPR. Finally, given the nature of the survey data (MICS), it
was not possible to measure the attribution of the contracting out activities to the increase of CPR
through the public sector. As previously noted, a simpler indicator, such as the number of family
planning consultations provided or additional number of new family planning acceptors in
government facilities, would perhaps have been more suitable for the project given the limited
availability of data and difficult operating environment.
53. ‘Accessibility’ of health services has three dimensions, namely, financial affordability,
physical accessibility, and acceptability (behavioral and cultural aspects). Even if services are
available, only when ‘accessed’ by end users, would available health services be consumed and
thus coverage indicators improve. Hence, availability is a prerequisite to accessibility. In the ICR,
PDO indicators 1 and 2 are assigned to assess the increase in ‘access’ to health services under the
project.
54. PDO indicator 1 - People with access to a defined basic package of health, nutrition,
and reproductive health services (number). At the end of the project, the total number of people
with access to a defined package of health services was 3,816,585, 96 percent of the revised target
6 http://www.cpc.unc.edu/measure/prh/rh_indicators/specific/fp/cpr
0
10
20
30
40
50
60
70
Provincialrural avg.
Lower Dir Buner Battagram Tor Ghar Kohistan
2012-13 2014-15
17
of 3,956,119. The original target was defined as 90 percent, but unclear about its denominator. If
it was meant to be 90 percent of the total population, which is 2,737,386, 139 percent of the original
target was achieved. The reason for the final figure exceeding the estimated total population is
unclear; however there are a several possible explanations: (a) the figures might include both new
and repeated patients, (b) population might have increased since the 1998 Census; and (c) people
outside the target districts might have also utilized services. Some NGOs/contractors organized
medical camps for remote areas to provide better access to underserved populations. It is
noteworthy that the project benefited 65 percent of total MDTF-1 beneficiaries, the single largest
contribution among all the MDTF-1 funded projects. The baseline for the indicator was set as zero
in the EPP. It should be however noted that there was some form of health service provision in the
target districts that was not defined as a comprehensive package of services even before the project
implementation.
55. PDO indicator 2 - Percentage of children with severe acute malnutrition provided
adequate nutrition services. Achievement of the indicator suffered due to the delayed province-
wide rollout of the nutrition program. The final figure of 16 percent is therefore only for two
districts. The baseline for the indicator was also set as zero in the EPP. There had been no provision
of clinical nutrition services for severe malnourished children at secondary health facilities before
the project.
56. ‘Delivery’ of health services usually considers the perspectives of end-users to measure
health care delivery performance. PDO indicator 5 measured client satisfaction towards health
services provided by the public sector.
57. PDO indicator 5 - Community satisfaction with health care services delivery by the
public sector. Exit interviews and surveys were the source of data for this indicator in EPP.
However, as no comprehensive exit interviews were carried out, the two data points of PSLM
(2012-13 and 2014-15) are compared to calculate the level of achievements. The indicator
improved only 3% in the target districts, from 35% in 2012-13 to 38% in 2014-15, which is
somewhat aligned with the level of overall improvement for rural areas of KP in the same time
period (see Figure 2 below). Among the target districts, there were substantial increases in three
districts, i.e., Lower Dir, Tor Ghar, and Kohistan, while reductions were observed in two districts.
It should be noted, however, that the wording and definition of this indicator and the one
consistently used in PSLM are slightly different. The PSLM uses the “Percent distribution of
household satisfaction towards Basic Health Unit”, while the PDO indicator used a broad term,
i.e., “satisfaction with the government health services”. Given the PSLM was used as baseline in
the RF, the PDO indicator should have followed the same wording and definition.
18
Figure 2. Comparison of community satisfaction towards Basic Health Unit in five target districts
Source: PSLM 2012-13 and 2014-15
58. Data issues for PDO indicators 3–5. These three health outcome indicators, skilled birth
attendance, modern contraceptive prevalence rate, and community satisfaction, were not measured
adequately during project implementation due to lack of an on-time population-based survey.
While statistically accurate, the use of a population-level indicator and relying on a population-
based survey for obtaining results may not be ideal for projects operating in the post-crisis and
fragile context. In fact, the expected MICS did not occur during or even right after the project
period. The PSLM 2012-13 was available in April 2014, but could not be utilized in the mid-course
correction in the Results Framework during the June 2014 restructuring the timing of data release
was too late for the revised PC-I processing/approval for restructuring. The PSLM 2014-15 results
were released only after project closing. Besides, inclusion of service utilization in the private
sector in a population-based survey for service coverage indicators would also affect measuring
the attribution of the project interventions. When the unavailability of a population-based survey
data became obvious during implementation, these service coverage/utilization indicators at
population level could have been modified to ones for measuring a service volume change that
would have easier access to data.
59. Administrative data from the district. Administrative data shows that the project
interventions did work, making crucial health services available to people living in four remote
and underserved districts. The administration data from Battagram District (Figures 3-5 below),
for instance, clearly indicates substantial improvements in availability and utilization of health
services on the ground after the contracting out started. This indicates that using the administrative
data and absolute numbers (not percentage) could have been simpler and made it easier to show
the impacts brought by the project, rather than using ambitious population-level indicators and
relying on the survey data given the aforementioned constraints, especially given under the
emergency operation and short project implementation.
0
10
20
30
40
50
60
70
80
Provincialrural avg.
Lower Dir Buner Battagram Tor Ghar Kohistan
2012-13 2014-15
19
Figure 3. Comparison of monthly average number of visits to outpatient department (OPD) in Battagram
District from 2013 to 2016
Source: Integrated Health Services, Battagram District, March 2016
Figure 4. Comparison of performance of key health indicators (monthly average) in Battagram District from
2013 to 2016
Source: Integrated Health Services (IHS), Battagram District, March 2016.
Figure 5. Comparison of presence of medical officers in BHUs in Battagram District from 2015-2016
Source: KP Health Roadmap Stocktake, February 2016
20
3.3 Efficiency
60. The overall rating for the Efficiency of the project is Negligible.
61. The EPP included the economic and financial analysis in its annex; however, the main text
did not discuss or summarize the analysis. The analysis did not attempt to quantify project benefits
due to lack of project location specific data and thus the results became generic. The expected
benefits of the project, covering such areas as reduced disease burden and improved life
expectancy, were considered to be very substantial.
62. Allocative efficiency. Allocative efficiency is rated as Modest. Component 1 of the project
was supposed to generate most direct benefits, while Components 2 and 3 could generate indirect
benefits by strengthening healthcare system. The project rightly focused on cost effective
interventions which is proven to work in a difficult environment, i.e., using the contracting out
approach to revitalize health services in post disaster/conflict areas. The project disbursement
shows that the primary emphasis of the project was in revitalizing health services, accounted for
87.7 percent of the project disbursement. The health interventions supported by the project were a
set of promotive and preventive as well as primary health care services and assessed against the
international evidence of their cost-effectiveness. Also, the project adequately allocated the funds
to the rehabilitation of health facilities under Component 2 and strengthening M&E system under
Component 3, given the constraints identified during project preparation. Annex 3 attempts to
evaluate potential benefits intended to be brought by the project.
63. However, while allocative efficiency was substantial in theory, the actual performance
under project components was unsatisfactory; the entire project scope was substantially reduced
because of extremely slow implementation progress. Component 2 was dropped entirely due to
non-performance, and budget allocated for Components 1 and 3 was reduced to 80 percent and 30
percent, respectively, due to delay in procurement process. Furthermore, the government
counterpart funds needed to run the contracting out services was not paid on time to the
NGOs/contractors. Therefore, the actual impact of the project activities were significantly reduced
as opposed to the intended benefits laid out in the EPP.
64. Implementation efficiency. Implementation efficiency was Negligible, given overall
extremely slow implementation. The implementation delays like contracting out of services for
two years led to reduction of the total project amount by 36 percent at the 2014 restructuring.
Besides, at project closing, only 57 percent of the remaining amount or 36 percent of the original
mount was disbursed. As noted previously towards the end of the project there were slight
improvements as the contracting out activities had started. The increase in the total number of
services suggests that health facilities are becoming more efficient based on decreasing cost per
service. The project also provided an opportunity to train 1,365 persons, which possibly
strengthened the quality of care and linkage between outputs and outcomes.
65. Another critical implementation inefficiency was the overall decision making process, both
on the GoKP and the Bank management. Senior government officials in the GoKP showed their
commitments and promised their willingness to improve implementation performance. Their
intentions expressed to the Bank management were however not enforced at the implementing
21
agency level. On the other hand, at the Bank, in spite of repeated requests from the task team for
decisive actions against stagnant implementation, decisions were either not taken or delayed.
3.4 Justification of Overall Outcome Rating
66. As indicated in the summary table below (Table 3), the overall outcome rating of the project
is estimated to be Unsatisfactory.
Table 3. Project Overall Outcome Ratings
Phase 1 (against the original
target values; between
Effectiveness and the June 2014
Restructuring)
Phase 2 (against the revised
target values; between the
Restructuring and the Closing
Date)
Relevance Substantial
Objective High
Design Substantial
Implementation Modest
Efficacy Modest Modest
PDO 1 (availability) Modest Substantial
PDO 2 (accessibility) Negligible Negligible
PDO 3 (delivery) Modest Substantial
Efficiency Negligible
Value (a) 2 2
Total % disbursed (b) 0.51 0.49
Final score (a*b) 1.02 0.98
Final outcome rating 1.02 + 0.98 = 2.0 Unsatisfactory
3.5 Overarching Themes, Other Outcomes and Impacts
(a) Poverty Impacts, Gender Aspects, and Social Development
67. The project design did not explicitly take into account impacts on poverty. There is no
data available to directly measure positive or negative impacts on poverty during the project period.
Contracting out health services, however, appears to have had important impacts on poverty,
gender, and social development. First, the World Bank task team made a conscious decision during
project preparation to target districts based on criteria it developed, including crisis-affected
districts with poor socioeconomic and health indicators, as the EPP required. The targeted districts
are located in hilly areas with difficult geographical access to health facilities. People living with
such hardships had been chronically underserved with basic social services previously. The project
made some reliable and quality health services available to people in these target districts—
availability of female doctors led women to seek health services at the facility for the first time.
Free medical camps held at remote areas drew hundreds of people in a few hours. People now
understand the importance of the availability of such basic health services in their lives and their
expectations have been raised. In fact, people in Battagram and Tor Ghar Districts were
empowered and expressed their concerns by staging a demonstration to urge continuation of
provision of health services when they were informed about project closure.
22
(b) Institutional Change/Strengthening
68. The project was specifically designed to strengthen institutional capacity in the government
and increase the government’s ownership/confidence in the contracting out model. This intention
was clearly exhibited in the choice of the implementing agency, that is, the HSRU of the DoH, and
by the leveraging of a large government recurrent budget along with the development budget. Such
exemplary efforts to attempt to use the government system and build longer-term sustainability
unfortunately did not prove optimal during project implementation. The use of the government
system may not have been the right choice under an emergency operation, as institutional
strengthening and ownership building takes considerable time even in the best of situations. At the
end of the difficult project implementation period, however, it is nonetheless the case that the
GoKP is attempting to institutionalize the project experience and has decided to roll it out to the
entire province using its own budget.
(c) Other Unintended Outcomes and Impacts (positive or negative)
69. Not applicable.
3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops
70. A series of stakeholder/beneficiary interviews, anecdotal stories, and news articles from
the field consistently point to the significant appreciation by beneficiaries of project interventions
on the ground, especially in the districts where quality and continuous health service provision had
been severely lacking. During the interviews as part of ICR preparation, there were several requests
from the district government officials for the continuation of the project.
4. Assessment of Risk to Development Outcome Rating: Moderate
71. The overall risk to the development outcome is considered Moderate. While project
implementation faced a series of setbacks, as the previous sections describe, the project provided
opportunities for the GoKP to observe the advantages of employing the contracting out model and
making the GoKP accustomed to using its own resources for contracting out. This should be noted
as the project’s major achievement—it worked as a pilot/catalyst for the GoKP to increase its
commitment to the contracting out model in the difficult political and economic environment of
KP Province. However, moving forward, for this initiative to be successful, the implementation
obstacles that the project faced, especially in the areas of contract management and funds flow,
must be addressed. It is also important that knowledge and capacity accumulated in the PMU
around the contracting out - contract management, managing district-level relationship, accounting,
etc. - be transferred to any new implementing agency.
23
5. Assessment of Bank and Borrower Performance
5.1 Bank Performance
(a) Bank Performance in Ensuring Quality at Entry
Rating: Moderately Satisfactory
72. The World Bank performance at entry is rated as Moderately Satisfactory. The PDO and
project design were highly relevant to the context at appraisal and the PCNA/MDTF objectives.
Building on the successful implementation of the JSDF Battagram project and other previous local
and regional operations, the project design was technically sound. Using the government system
for implementation of the contracting out and leveraging large government recurrent budget turned
out to be too ambitious and likely the wrong choice in the context of the emergency situation.
Although from the perspective of fostering institutional building and increased ownership within
the DoH and GoKP, the choice for this implementation arrangement should be lauded. The choice
of PDO indicators was too ambitious considering the post-crisis and fragile context and access to
credible data source. The major operational risks were identified in ORAF, including the limited
capacity for implementation and monitoring by the DoH, and measures to mitigate the risks were
put in place, such as identification of additional skills and staff required and hiring of a third party
to support M&E. However, slow PC-I approval processing within the federal government was not
identified as a critical risk, though the task team was fully aware of it. Besides, a long-waiting
period was unfortunately not utilized for enhancing limited capacity. In light of the shortened
implementation period, the closing date extension should have been considered immediately after
the nine month delay, but this was not a likely option because the project timeline was bound by
the MDTF-1 end date. Considering such shortcomings under the MDTF, the World Bank/MDTF
Secretariat should have actively requested the GoKP to explore ways to expedite the Government’s
internal processing, such as (a) possible streamlining the PC-I approval process for all the MDTF-
1 projects as a response to the condensed Bank internal procedures under OP/BP 8.00; and (b)
considering financial incentivizes for the GoKP for expedited PC-I processing under the
emergency operation. In fact, the Government of Pakistan approved the simplified PC-I procedure
in December 2011 to facilitate quick processing and approval of MDTF projects by delegating the
authority of project approval to the relevant provincial/ regional authorities. However, such
changes took place after the project’s PC-I processing was started, thus all delays occurred for the
project’s PC-I. Lastly, while OP/BP 8.00 permitted the project to move ahead even when certain
implementation arrangements were not yet in place, a high price was paid for doing so. For
example, putting in place aspects of the procurement process critical to contracting out could have
ensured faster start-up of project implementation.
(b) Quality of Supervision
Rating: Moderately Satisfactory
73. The quality of overall World Bank supervision is rated as Moderately Satisfactory. The
amount of effort that the World Bank task team put forth was commendable. Despite the
heightened security situation and difficult the post-crisis and fragile context, frequent visits and
close contact with the DoH/HSRU led to proactivity by the World Bank task team in identifying
problems. Considering the lack of experience in World Bank operations at the DoH, the repeated
fiduciary support provided by the World Bank task team was also proactive and productive. The
24
regular reviews conducted found that in some cases expenditures needed to be adjusted according
to World Bank policies. The World Bank task team also maintained a close contact throughout
project implementation with the Country Management Unit and the MDTF Secretariat within the
Bank to timely update them and raise critical implementation issues for a decision. The Bank
management also raised issues at the highest level of the GoKP including the Chief Minister,
Health Minister, Finance Minister, and Additional Chief Secretary to jointly address
implementation bottlenecks.
74. Unfortunately, these efforts were not sufficient to turn the project implementation around;
the impacts were largely compromised by the post-crisis and fragile operating environment. In
addition, the following weaknesses in the World Bank supervision can be identified: (a) decisive
actions, including a warning of possible suspension/cancellation of the project in light of prolonged
implementation delays, were either delayed or not taken up by the Bank management; (b) the
Results Framework was not proactively revised even at the 2014 restructuring, other than
downsizing the end target values; and (c) the World Bank should have not allowed the
NGOs/contractors to provide their own funding for the continuation of service provision (see
Section 2.2). This was clearly a violation of the contracts between the GoKP and the contractors,
which promised quarterly payments to the contractors. The newspapers have featured the stories,
which posed reputational risks to the World Bank. Failure by the GoKP’s to secure timely release
of adequate funds for the contractors was a serious matter that should have prompted the threat of
a series of remedies, such as withholding of disbursement, declaring misprocurement, and
suspension/cancellation of the project. It could have sent a strong signal to the GoKP to address
this critical project matter promptly. Lastly, in retrospect, it is unfortunate that the project was
closed just as it was beginning to show some results. As mentioned previously, the Bank’s
decision-making on this issue varied. However, at the end, as there was no indication that the
critical pending funds release issues would be resolved quickly, the Bank’s decision is
understandable.
(c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory
75. The overall rating for Bank performance is Moderately Satisfactory, based on rating of
Moderately Satisfactory both for quality at entry and for quality of supervision.
5.2 Borrower Performance
(a) Government Performance
Rating: Moderately Unsatisfactory
76. The Government’s buy-in during preparation was mixed. The level of interests and
ownership at the senior level of the GoKP to the project was high, after their successful experience
in Battagram and the PPHI, as indicated by the large amount of project counterpart funding. In the
meantime, despite this being an emergency project, the long PC-I approval process within the
government resulted in a shortened project implementation period. As stated previously, the
Government of Pakistan approved the simplified PC-I procedure in December 2011 to facilitate
quick processing and approval of MDTF projects; however, such changes took place after the
project’s PC-I processing was started, thus the project’s PC-I suffered from substantial delay.
25
During implementation, when the project faced a series of implementation issues in the DoH, the
GoKP remained committed to the project and senior management along with the Bank
management intervened at crucial times. Such commitment and ownership at the highest level of
the government, however, did not get translated into actions at the implementing agency level,
mainly due to political complexity and loose enforcement of internal government rules. The major
implementation issue was a failure to promptly address project processing delays at the
DoH/HSRU, which in turn led to unnecessary implementation delays. For instance, even after the
Chief Minister issued an order to immediately release the government recurrent budget to the
NGOs/contractors, it took six months to execute this release. Numerous court cases and political
interference over staff appointments in the HSRU/PMU, leading to the lack of a fully empowered
Project Director/Coordinator, were also detrimental to project implementation. In addition, there
was frequent turnover in the Secretary Health position (seven secretaries in the three years of
project implementation), which definitely affected the prospects for timely implementation. Given
these considerations, the GoKP performance is rated as Moderately Unsatisfactory.
(b) Implementing Agency or Agencies Performance Rating: Moderately Unsatisfactory
77. The DoH was the implementing agency of the project. Within the DoH, first the HSRU
and later an independent PMU were responsible for managing project implementation. Burdened
by KP’s weak government institutions and political complexity typically seen in post-crisis
environment, in addition to the short project implementation period, the DoH struggled to
implement their first World Bank project. It was also their first experience with managing a large-
scaled contracting out initiative. Limited understanding of contracting management as well as the
Bank procedures was the main reason for initial implementation delay at the HSRU. Frequent
meetings and support were extended by the Bank team to mitigate such shortcomings. The DoH,
however, was not able to resolve deficiencies critical to project implementation. These included
the provision and continuity of key staff, the processing of government documents, procedural
delays coupled with additional processing steps and differences in interpretation of policies, and
timely decision making on day-to-day operations. These implementation issues were further
exacerbated by the frequent change of Secretary Health and Project Director/Coordinator. All led
to an almost two-year delay in the procurement process for contracting NGOs/contractors and
substantial delays in release of funds. The NGOs/contractors had to deal with more than 15
steps/reviews within the DoH to receive the reimbursement of the MDTF funding. Also, the PMU
could not ensure on-time release of the government recurrent budget to the NGOs/contractors. As
a result, some NGOs/contractors chose to utilize their own funds to continue project activities
while waiting for funds to be released, up to nine months in one instance. In addition, the PMU
was not proactive in managing relationship on the ground for smooth initiation of the contracting
out activities. Thus, the Inception Meeting with all stakeholders was to take place immediately
after the signing of the contract for each district; however, none was organized by the PMU. The
NGOs/contractors had to deal with such issues by themselves, which took a few months to settle.
Lastly, the abolishment of the M&E Cell in DoH at the time of project effectiveness put into
question the willingness to implement the project. The performance of implementing agency is
therefore rated as Moderately Unsatisfactory.
(c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory
26
78. The overall rating for overall Borrower performance is Moderately Unsatisfactory, based
on rating of Moderately Unsatisfactory both for Government Performance and for Implementing
Agency Performance.
6. Lessons Learned
79. The following key lessons have been identified.
Lessons for Bank Operations in the post-crisis and fragile context
80. Project design and evaluation must be based on a realistic assessment of prevailing
implementation arrangements and risks. Especially for projects implemented in a fragile and
low capacity context, realistic assessment of the implementation risks is needed. Besides,
mitigation measures should be implementable and significant enough to improve the situation
when identified risk takes place. Project design, including the Results Framework, and
implementation arrangements need to be carefully reviewed based on the risk level and adjusted
accordingly taking into account the lead time required for setting up basic implementation
arrangements. Because of institutional and capacity-related weaknesses, sufficient project
implementation timeframe should be planned out. There is also a need for developing greater
tolerance for failure or implementation hurdles. The Bank increasingly engages in FCV contexts,
with the new concept of “pocket (or sub-national) of fragility” in the next IDA18, the ratings and
the need for inter rating reliability should be reconsidered, when it comes to evaluation of project
performance. In performance evaluation, the post-crisis/FCV context should be also factored in,
and moreso given that the amount of efforts made by the task team in such an environment is
usually far greater than projects implemented in a non-crisis setting. In spite of the extensive
support and additional interventions, the results are not always encouraging, constrained by low
capacity, heightened security, and weak institutions.
Lessons for Emergency Operations
81. Emergency projects are unlikely to meet their objectives in the absence of well-
established and reliable institutional capacity. Use of existing government systems in
emergency operations is challenging. In particular, efforts to combine two sets of objectives, that
is, attempting to achieve tangible results in a short period while also aiming for long-term capacity
building and institutional strengthening of existing government systems can prove problematic,
especially in countries with very low institutional capacity.
82. Essential procurement arrangements must be in place to facilitate a smooth start-up. Given the need to make rapid progress, emergency projects sometimes make compromises with
respect to implementation readiness. As in this project, insufficient readiness on the part of the
implementing agency later can hinder smooth initiation of project activities. Adequate
procurement arrangements are particularly important in contexts where reliance on contracting is
the only likely implementation strategy.
Lessons for Operations in KP
27
83. Political economy analysis is essential when preparing a project for contexts like KP.
One underlying dynamic affects project implementation in KP, namely, political complexity.
Political interference can negatively or positively influence every level of implementation. Lessons
from one Bank-financed project in KP in the past,7 which appears to have involved political
complexity and conflicting government response due to tangled stakeholder relationships, include
the following: (1) direct discussions need to be held with all potential stakeholders during project
preparation to ensure a greater degree of transparency and a broad buy-in for the proposed
operation, (2) official interests should fully reflect the combined interests and decisions of all
stakeholders to avoid confusion and potential disruption in preparation and implementation.
Lessons Learned for Contracting Out of Health Services
84. The contracting out of health services to NGOs/contractors can be an effective
approach in hard-to-reach areas and/in security-compromised areas. Despite the many
difficulties encountered in implementation, the project made some progress in demonstrating the
viability of contracting out the delivery of health services in remote or security-compromised areas
where traditional provision has not been effective. This potential success is heightened if
NGOs/contractors who manage health facilities have considerable autonomy/flexibility with
regard to the financial and human resource aspects of providing services.
85. Proper contract management by the implementing agency is critical to the success or
failure of contracting out initiatives. Responsibility for contract management entails preparing
and processing contracts, regular monitoring of contractors’ outputs as defined in the contract, and
meeting with contractors regularly to identify and resolve implementation issues. A contract
management specialist may not be adequate for undertaking these multiple tasks if the number of
contracts is large. Poor contract management capacity by health ministries is common worldwide.
Successful contract management in countries appears to have the following key characteristics:
provision of sufficient resources, use of local consultants, and having a manageable number of
large contracts.8
86. Clear internal rules of business must be in place and internalized by the lead
implementing agency. Lack of clear understanding of policies and the absence of clear rules of
business obviously mitigates the likelihood of project success. It is particularly important in
projects depending on contracting out that, prior to implementation, streamlined procedures for
flow of funds are clearly understood and in place within the government to avoid payment delays.
Otherwise the contracting process quickly comes to a halt.
87. Frequent and effective communication among all stakeholders is essential to address
and remove concerns regarding the value of contracting out. Contracting out government
services can cause some concerns among the existing players. For example, parts of the DoH and
some district authorities were not confident that they could hold NGOs accountable for results
without close control over resources and inputs. This prompted unnecessary review processes for
7 The World Bank. Note of Cancelled Operation for “Khyber Pakhtunkhwa and FATA Emergency Recovery
Project” (Report No: NCO00002137). 8 Loevinsohn, Benjamin. 2008. “Performance-Based Contracting for Health Services in Developing Countries: A
Toolkit.” World Bank.
28
payments to the contractors at the DoH. It also appears that some District Commissioners, who are
in charge of directly releasing government recurrent budget to the contractors, were unsure what
the contractors were really doing and thus delayed the payments. Such problems stem from
unfamiliarity with managing contracts and apprehension about the concept of contracting out and
can only be overcome by extensive communication between all stakeholders.
88. Contracts should only be signed if there is a sufficient implementation period for the
approach to succeed. NGOs/contractors, repeatedly voiced their concern that one year and a few
months was just too short to realize the full impacts of the contracting out process. The Bank task
team could have proactively suggested that the implementation period in 2014 be extended as part
of restructuring. NGOs/contractors need to make substantial investments in setting up new
activities in a district and establishing smooth relationships with the district authority, local
politicians and leaders, and service providers. This argues for ensuring that the implementation
period for contracting out of services be sufficient to achieve satisfactory results.
7. Comments on Issues Raised by Grantee/Implementing Agencies/Donors
(a) Grantee/Implementing agencies
89. The comments provided by the Government of Pakistan are included in Annex 7. The
comments have been addressed by the ICR team in the main text of ICR.
(b) Cofinanciers/Donors
(c) Other partners and stakeholders
29
Annex 1. Project Costs and Financing
(a) Project Cost by Component (in US$ million equivalent)
Components
Appraisal
Estimate
(US$, millions)
Revised
Estimate*1
(US$,
millions)
Actual/Latest
Estimate (US$,
millions)
Percentage
of Appraisal
Percentage
of Revised
Amount
Component 1: Revitalizing
Health Care Services - a 56.0 26.94 8.36 14.9% 31.0%
MDTF 11.0 8.94 5.05 46.0% 61.6%
Counterpart financing2 45.0 18.0 3.31 7.0% 18.0%
Component 2: Rehabilitation
of Health Infrastructure - b 1.0 0.0 0.0 0% 0%
Component 3: Establish and
Operationalize a Robust
Monitoring and Evaluation
System at the District and
Provincial Levels -c
4.0 1.26 0.72 18.0% 36.0%
Total Project Costs (a+b+c) 61.0 (including
MDTF 16.0)
28.2 (including
MDTF 10.2)
9.08 (including
MDTF 5.773)
14% (only
MDTF - 36%)
32% (only
MDTF – 56.6%)
Note: *1 - US$5.8 million was cancelled in June 2014, reducing the total grant amount to US$10.2 million.
Note: *2 – The government counterpart financing was used only for Component 1 as recurrent budget to run
government health facilities.
Note: *3 - At project closure, the actual expenditure was US$5.85 million, however, with the refund of ineligible
expenditure of US$79,422 in April 2016, the final actual expenditure is US$5.77 million.
(b) Financing
Source of Funds Type of
Cofinancing
Appraisal
Estimate
(US$,
millions)
Revised
Estimate
(US$,
millions)
Actual/Latest
Estimate
(US$,
millions)
Percentage of
Appraisal
MDTF for Crisis-Affected Areas of
KP/FATA/Balochistan – 16.00 10.20 5.77 36.0%
Government recurrent health budget Counterpart
financing 45.00 18.00 3.31 18.0%
30
Annex 2. Outputs by Component
Component 1: Revitalizing health care services. (original estimated cost - US$11.0 million;
revised - US$8.94 million; actual expenditure - US$5.05 million). Originally, six districts were
selected for contracting out, based on criteria developed by the task team. Management of health
service was outsourced to the NGOs/contractors, through a competitive selection process. Out of
six districts, the contractor for D.I. Khan refused to assume responsibility due to the short
implementation period. Kohistan was bifurcated during implementation and thus the contract
ended in one year. The primary health care centers (BHUs and RHCs) in each district were
reorganized into ‘hubs’ to enable efficient delivery of a comprehensive package of health care
services. The health facilities damaged during the crisis were well repaired and renovated to enable
service delivery. A training needs assessment was carried out in close coordination with the
Provincial Health Services Academy with successful training of 60 percent of the health personnel.
Intermediate Results Target Actual Remarks
1 Number of districts contracted out for
management of services
4 4 Not achieved against
original target and
achieved against
revised
2 Health personnel receiving training
(number)
1200 1365 Target achieved—
total persons were
2290
3 Percentage of hubs established and
assessed as fully functioning by the DoH
100 100 Target achieved
4 Health facility utilization rate: Visits per
person per year
1.00 0.54 Not achieved
5 HFUR by gender* 1.00 0.54 Male
0.63 Female
Not achieved
6 Health facilities constructed, renovated,
and/or equipped (number)
10 126 Target achieved
7 Number of district headquarters hospitals
refurbished
3 1 Not achieved
8 Health facilities adequately refurbished 20 52 Not achieved against
original target and
achieved against
revised
Component 2: Rehabilitation of Health Infrastructure (original estimated cost - US$1.0
million; revised - US$0.0 million; actual expenditure - US$0.0 million). The component was
entirely cancelled at the restructuring in June 2014, without any disbursement made. Hence, there
is no output under Component 2.
Component 3: Establish and Operationalize a Robust Monitoring and Evaluation System at
the District and Provincial Levels. (original estimated cost - US$4.0 million; revised - US$1.26
million; actual expenditure - US$0.72 million). The component operationalized M&E systems to
guide project implementation at the district level by timely dissemination of the monthly reports.
The Provincial Steering Committee with the additional chief secretary as the chairperson reviewed
the project activities in all target districts every six months according to the agreed plan of the
project. The operationalization of the DHMT enabled timely reviewing, monitoring, and
31
facilitation of project implementation at the district level. Expected support to operationalization
of DHIS did not get materialized, due to the dissolution of the M&E Cell.
Intermediate Results Target Actual Remarks
9 Timely disbursement of funds to a
consultant/NGO implementing
contracting out
100 25 Not achieved—delay in the
release of funds worsened
toward the end of the
project
10 Biannual meetings held for the Provincial
Steering Committee
2 2 Target achieved
11 Number of health facilities submitting
monthly reports on time to the district
90 90 Target achieved
12 Establishment within two months from
the contract date and operationalization of
the DHMT
4 4 Not achieved against
original target and achieved
against revised
32
Annex 3. Economic and Financial Analysis
The Emergency Project Paper (EPP) describes both direct and indirect benefits arising from the
primary health care and hospital services. The same framework is used as an exit point in the
economic and financial analysis. No economic rate of return (ERR) or net present value (NPV)
were estimated. The scope of analysis for this report is, due to severe data constraints, limited to
the review comments on (a) cost-effectiveness of the interventions supported by the project; (b)
cost-benefit considerations; (c) efficiency considerations; and (d) equity considerations.
Cost-effectiveness of the interventions supported by the project
Based on analysis by Institute for Health Matrix and Evaluation (IHME) “Global Burden of
Diseases, Injuries, and Risk Factors Study 1990-2010” wherein results for Pakistan were published
in August 2013; the health conditions addressed by project interventions account for about 42.4%
of the disease burden in Pakistan9. The figure below represents 25 top leading causes of diseases
burden in Pakistan in 2010 and the changes that have occurred since 1990.
The top 25 causes of DALYs are ranked from left to right in order of the number of DALYs they
contributed in 2010. Bars going up show the percent by which DALYs have increased since 1990.
Bars going down show the percent by which DALYs have decreased.
9 GBD Profile: Pakistan (http://www.healthmetricsandevaluation.org)
33
Cost effectiveness of interventions has been established in peer-reviewed literature.10 The Table
below presents cost (US$) per DALY averted by interventions supported by the project. Given
Pakistan’s GDP per capita, targeted interventions were cost effective. An intervention is cost-
effective if the cost per disability-adjusted life-year (DALY) avoided is less than three times the
national annual GDP per capita. It is highly cost-effective if it is less than the national annual GDP
per capita. World Health Organization’s Choosing Interventions that are Cost–Effective (WHO-
CHOICE) project recommended these under thresholds based on per capita national incomes
approach.11
Table: Cost per DALY averted by the project interventions
Project component US$/DALY averted
Standard maternal and child health package 24-585
Package of prenatal and delivery care 92-148
Expanded Program on Immunization (EPI) 8
Tetanus Toxoid vaccine 14
Acute Respiratory Infections (Facility) 24-424
Diarrhoea (Oral Rehydration Therapy) 132
Integrated Management of Childhood Illness
Malnutrition
Breast feeding support programme 3-11
Growth monitoring and counselling 8-11
Vitamin A supplementation 6-12
Communicable disease treatment and control
Control of tuberculosis (DOTS) 5-35
Although funding to this component was relatively small (US$5.13 million), probably there were
improvements of maternal and child health status and reduction of the prevalence of life-
threatening communicable diseases. There were exponential improvement in health results in
terms of disease burden, death averted and lives saved. With regards to value for money, the project
added value by the increase in access and use of health services. It was complex and difficult to
quantify the gross value of the project intervention due to lack of required data.
Cost-benefit considerations
10 Laximinarayan, R. (2006). Advancement of global health: key massages from the disease control priority project.
The Lancet. Pp. 1193-1208. 11 http://www.who.int/bulletin/volumes/93/2/14-138206/en/
Choosing interventions that are cost-effective [Internet]. Geneva: World Health Organization; 2014. Available from:
http://www.who.int/choice/en/ [cited 2016 April 10].
34
The concept of the “value of statistical life” (or life-year) is the basis for quantifying the benefit of
better health in monetary terms. The “value of statistical life”12 would be US$4,095 if it is
considered at least five times higher than GDP per capita in Pakistan. The project spent on average
1.95 million per year and if we consider value US$4,095 it would only have to achieve an average
of 476 additional life years annually to “break even”. This threshold is very feasible, given that
the project made available of health and nutrition services to 3,816,585 individuals. Improved
access to health care and use of preventive care also improved productivity through avoiding lost
productivity due to preventable illnesses and related premature deaths. The benefits would
substantially exceed the costs, even if only 1% of those put on treatment attained one additional
year of life as a result. The project appears to have achieved a very favorable cost-benefit ratio.
Efficiency considerations
A project is considered efficient if it helps to achieve the same health gains at lower cost (or,
equivalently, greater health benefits for the same cost). Data on gains are not available. It is
difficult to identify areas where project achievements could have been realized more cost
efficiently. The major investments were generally made at the primary and hospital level care.
Moreover, the funded services are helping to address conditions that represent well over 42% of
the disease burden in Pakistan, so resources were directed to high-priority interventions.
In qualitative terms, key project activities under component 1 and 3 offered scope for efficiency
gains. The project provided opportunity to train 1,365 persons and this activity possibly
strengthened quality of care and linked between outputs and outcomes. The activities could have
generated better value for money as there was no implementation of service delivery before starting
this project.
The total cost of operationalizing monitoring and evaluation system over the project was US$0.72
million. All targeted facilities (90) submitted monthly reports on time to district. The potential of
improving efficiency was there. Thus the strengthened M&E system and DHIS would easily pay
for themselves many times over if the systems can be leveraged to achieve even small (e.g., 1%)
efficiency gains on an annual basis.
Equity
Selection of intervention districts was based on poor socio-economic and health indicators. The
interventions in health benefitted girls and women from poorer households and promoted gender
and economic equity in the remote districts. The resources were used in vulnerable areas. The
benefits had a strong equity dimension. The project could have had a much stronger M&E focus
with respect to the impact of activities on different socioeconomic groups though the EPP
recommended to close monitoring of health service utilization from equity perspective.
12 OECD. The Value of Statistical Life: A Met-Analysis. (2012). ENV/EPOC/WPNEP(2010)9/FINAL
Viscusi, W. Kip and Aldy, Joseph E., "The Value of a Statistical Life: A Critical Review of Market Estimates
Throughout the World" (2002). Harvard Law School John M. Olin Center for Law, Economics and Business
Discussion Paper Series. Paper 392
35
Annex 4. Grant Preparation and Implementation Support/Supervision Processes
(a) Task Team members
Names Title Unit Responsibility/
Specialty
Lending/Grant Preparation
Tayyeb Masud Task Team Leader, Health Specialist SASHN Task team leader
Inaam ul Haq Senior Health Specialist SASHN Health specialist
Kees Kostermans Lead Public Health Specialist SASHN Health specialist
Tekabe Ayalew Belay Senior Economist SASHN Economist
Naoko Ohno Operations Officer SASHN Operations
Maria Gracheva Senior Operations Officer SASHN Operations
Martin Serrano Senior Counsel LEGES Legal
Chau-Ching Shen Sr. Financial Officer CTRFC Financial
Javaid Afzal Senior Environmental Specialist SASDI
Environmental
safeguards
Chaohua Zhang Lead Social Development Specialist SASDS Social safeguards
Samina Mussarat Islam Social Development Specialist SASDS Social safeguards
Robert Bou Jaoude Program Manager - MDTF SASPK MDTF
Uzma Sadaf Senior Procurement Specialist SARPS Procurement
Syed Waseem Kazmi Financial Management Specialist SARFM Financial management
Anwar Ali Bhatti Financial Analyst SACPK Disbursement
Nasreen Shah Kazmi Team Assistant SASHD Program assistant
Supervision/ICR
Tayyeb Masud Task Team Leader; Senior Health Specialist GHN06 Task team leader
Naoko Ohno ICR Team Leader; Operations Officer GHN06 ICR team leader/author
Inaam ul Haq Program Leader SACPK Program leader
Uzma Sadaf Senior Procurement Specialist GGO06 Procurement
Qurat ul Ain Hadi Financial Management Specialist GGO24 Financial management
Nasreen Shah Kazmi Team Assistant SACPK Program assistant
Muhammad Waqas Mushtaq Consultant SACPK MDTF
Mohammad Omar Khalid Consultant GENDR Environmental
safeguards
Ambreen Tariq Consultant GHN06 ICR
L. Richard Meyers Consultant GHN06 ICR
Shakil Ahmed Senior Health Economist GHN06 ICR (Economic analysis)
(c) Staff Time and Cost
Stage of Project
Cycle
Staff Time and Cost
No. of staff weeks Total cost (USD)
(including travel and
consultant costs)
Lending TF BB TF BB
FY11 7.04 0.00 23,697 0
FY12 15.53 0.00 55,538 0
36
FY13 3.76 0.00 7,390 0
TOTAL 26.33 0.00 86,626 0
Supervision/ICR TF BB TF BB
FY12 1.85 0.00 13,993 0
FY13 7.96 0.00 18,038 0
FY14 30.62 7.39 135,050 10,223
FY15 16.9 9.06 52,237 26,472
FY16 18.71 1.71 72,293 3,075
TOTAL 76.04 18.16 291,613 39,770
37
Annex 5. Beneficiary Survey Results
Not available
38
Annex 6. Stakeholder Workshop Report and Results
Not available
39
Annex 7. Summary of Grantee's ICR and/or Comments on Draft ICR
Project Completion Report by the Government of Pakistan
PCR – 01 (Revised-2010)
PC - IV
PROJECT COMPLETION REPORT
(PROFORMA FOR DEVELOPMENT PROJECTS)
PLANNING COMMISSION GOVERNMENT OF PAKISTAN
40
PCR – 01 (Revised-2010)
GOVERNMENT OF PAKISTAN PLANNING COMMISSION
****** PROJECT COMPLETION REPORT
(PC – IV PROFORMA)
To be furnished immediately after completion of the project regardless the
project accounts have been closed or not.
1. Name of the Project/Program/Study
Revitalizing Health Care Services in
KP (RHS)
Location
PMU in Peshawar with program areas
in Districts Kohistan, Torghar,
Battagram, Lower Dir, Buner and D I
Khan
2. Sector Health
Sub-Sector Primary and Secondary Health Care
service delivery
3. Sponsoring Ministry/Agency Health Department, KP
4. Executing Agency (s)
Project Management Unit (PMU)
under Directorate Health, KP
5. Agency for Operation & Maintenance after Completion
Health Department, KP
6. Date of Approval & Approving Forum (DDWP/CDWP/ECNEC/PDWP/Other)
• Original ECNEC PKR 2.125 B (USD 25 M)
AUG 2012
• Revised
PDWP(R) PKR 1.520 B (USD 16 M)
JULY 2013
2nd REVISION PKR 968.95 M
(USD 10.2 M) 19th NOV 2014
3rd REVISION PKR 1004.66 M
(USD 10.2 M) 26th JUNE 2015
7. a) Implementation Period
Date of Commenceme
nt
Date of Completion
• As per PC-I April 12, 2012 June 30th 2015
41
• Actual April 12, 2012 12th Dec. 2015
b) Extension(s) in the Implementation Period (if any)
Date Period (Months/Days)
30/10/2015 4 months
12/12/2015 42 days
(Rs. Million)
8. Capital Cost PC-I Cost (approved) Actual Expenditure
Local FE/Loan/ * Grant
Total Local FE Grant
Total
• Original 2125.5 2125.5 570.32 570.32
• Revised 1004.66 1004.66 570.32 570.32
Foreign-aid share as grant from MDTF administered by the World Bank with latest PC-1 exchange rate as $1=Rs. 98.50. (Total grant :Original $25 million Revised $10.20 M
(Rs. Million)
9. Financing of the Project Local FE Grant Total
Federal Share n/a n/a n/a
Provincial Share
Donors/Others 1004.66 1004.66
Total: 1004.66 1004.66
* Foreign-aid grant amounting to $10.20 with exchange rate as $1=Rs. 98.50 In addition district regular health budget is provided to Implementing
partners for districts health service delivery 10. Project Accounts
a) Nature of Account
Type Date of Opening Lapsable/ Non-lapsable
PLA
Assignment Account Aug 2012 Non-lapsable
Current Account
Saving Account
Other
b) Status of Account If closed, mention the date
Not yet
If not closed, mention reasons thereof & tentative closure date
Final Report submitted to the World Bank and account closure
in process.
11. Financial Phasing as per PC-I and Expenditure (Rs. Million)
Year PC-I Phasing PSDP Allocation Releases Expenditure
42
Total FE Grant
Total FE Grant
Total FE Grant
Total FE Grant
1 2 3 4 5 6 7 8 9
2012-13
638.65 638.65 500 500 283.62 283.62 4.34 4.34
2013-14
885.80 885.80 500 500 279.28 279.28 26.75 26.75
2014-15
937.87 937.87 1447 1447 423.94 423.94 390.7
6 390.7
6
2015-16
580.71 580.71 580.71 580.71 156.62 156.62 148.4
6 148.4
6
Total 3043.0
3 3043.0
3 3027.7
1 3027.7
1 1,143.4
6 1,143.4
6 570.3
1 570.3
1
12. Physical Targets and Achievements
S.No. Items (as per PC-I)
Unit Quantity
Actual *
Achievements
* Attach/Annex detailed information for each item separately
13. Item-wise Planned & Actual Expenditure (Rs. Million)
S.No.
Items (As per PC-I)
PC-I Estimates Actual Expenditure
Total Local FEC Total Local FEC
1 Component 1: Improve availability, accessibility and quality of health care services
880.76
880.76
509.61
509.61
2. Component 3: Establish and operationalize a robust M&E system
123.90
123.90
60.69
60.69
Total: 1004.66 1004.66 570.30 570.30
14. Recurring Cost after Completion of the Project
(Rs. Million)
S.No. Components PC-I Estimates* Actual Expenditure*
Total Local FEC Total Local FEC
43
Total:
** Directorate Health, Khyber Pakhtunkhwa will be responsible and will be financed through districts regular health budget to DHOs.
15. Achievement of Objectives
S. No. As Contained in the PC-I Actual Achievement*
1 Component 1: Improve availability, accessibility and quality of health care services
1. Improved human resource management and strengthened Health Care Professionals at all health facilities.
2. Availability of equipments and supplies at Health facilities through MDTF grant
3. Improved Logistic Management system resulted into availability of medicines at all Health Facilities level through Government funds for medicines.
4. Improved Health Facilities infrastructure by provision of renovation, repairs etc through MDTF grant.
5. Highly improved referral mechanism through Hub-Approach resulted into 24/7 availability of ambulance and Health Care Professionals at primary and secondary level of Health Facilities.
6. Rationalization of district health budget. In general, a district health budget comprises 85-90% for salary component and only 10-15% for Non-Salary component. This non-salary component is insufficient to provide necessary service delivery by any means. Under the project, Government funds routed as single line to Implementing Partners and they had full discretion to allocate salary and non-salary budget as per need to achieve the desired service delivery goals. This
44
rationalization of budget resulted into improved results. Key Results
1. Per capita OPD attendance increased remarkably.
2. ANC 2 and postnatal coverage increased.
3. Community satisfaction and restoration of confidence on Public Health system.
2. Component 3: Establish and operationalize a robust M&E system
1. GRM designed and implemented
2. ESMP implemented 3. Health Care professionals
trainings 4. Vehicles and equipments
procured for robust monitoring
16. Year-wise Income from Services/Revenue Generation (Rs. Million)
S. No. As Estimated in the PC-I Actual
N/A
17. RBM Indicators as given in the PC-I
S.No. Input Output Outcome
Targeted Impact
Baseline Indicator
Targets after Completion of Project
Refer annex A for RBM indicators
18. List of Project Directors (PDs) till Completion
S.No. Name & Designation From To
1 Dr Siraj Muhammad 22.02.2013 21.3.2013
2 D Shaheen Afridi 1.4.2013 31.01.2014
3 Dr Azmatullah Khan 1.2.2014 25.03.2014
4 Dr Shahid Yunis 26.03. 2014 21.10.2014
5 Dr Riaz 22.10.2014 02.6.2015
45
6 Dr. Nadeem ahmed 03.06.2015 Present
19. Responsibility/Ownership of Assets (Procured/Acquired/ Developed) after
Completion of the Project
Indicate Agency : Health Department, Khyber Pakhtunkhwa
List of Assets (Moveable/Immoveable) Annex B
20. Impact after Completion of the Project
a) Financial: Project ended and MDTF funding stopped by 12th of Dec 2015. Two districts, Torghar and Battagram have been extended till June 30th 2016, under DG Health KP to run through district regular health budget. All other districts handed over to DHOs as per existing system.
b) Economic: This project was designed to bring efficiencies and halt leakages by improving the management of healthcare delivery services. Besides, prevention was emphasized in the project design which is considered economical by all the experts around the world. The vacant positions in the district budgets were mostly filled and it was designed that at the completion of the project such positions will be regularized. 500 positions were filled during this period.
c) Technological: Appropriate technologies are one of the main outcomes of healthcare delivery system besides improved healthcare. The project was designed to introduce appropriate technologies in terms of management and rationalization of services through Hub approach.
d) Social: Health seeking behaviors of the population increased during the currency of this project. Emphasis was made in the project interventions to contact appropriate levels of healthcare services, by establishing referral system right from the first level of contact with the healthcare delivery system to the secondary level. The project generated 500 jobs of various healthcare cadres through transparent and easy hiring and placement mechanism. (Education, Health, Employment, area Development, etc.)
e) Environmental: Compliance to ESMP guidelines was made mandatory by the implementing partners. Waste management and infection control was heavily invested in areas. Noncompliance to these guidelines could result into termination of contract. As a result hygiene practices were promoted at all levels especially at the secondary care level.
21. Mechanism for Sustainability of Activities after Completion
1. Contract Management Unit: Contracting out of management of health facilities
to private sector has provided intended results in terms of improved service delivery at district level. However, experiences of this project shows that contract management expertise are very important in contracting out or public private partnership. Currently, this expertise is lacking in the department and whatever
46
limited capacities build during this project will be lost with closure of PMU. The department has invited EOI for outsourcing of all primary health facilities in the province through Health Foundation and without fully operational contract management unit it will be very difficult to manage contracts. it is recommended that the department may find ways to retain project key staff which will help build capacity of Health Foundation.
2. Timely Releases: Fund Flow Mechanism was developed for transfer of government funds to implementing partners but there were delays in release of funds which adversely affected service delivery. It is very important to ensure timely releases on basis of pre-financing to implementing partners. Finance department should arrange bridge financing in case of any delay.
3. Orientation of Line departments including AG office: Despite three years of
this project we still find it difficult to make concerned officials in department regarding procurement process, approval process and fund flow mechanism. Despite approved fund flow mechanism duly vetted by AG office and agreement vetted by Law department AG office audit team fails to understand the process and take unnecessary audit Paras. Before launching any donor funded project such orientation of concerned staff of all department is critical to success of the project.
22. Financial/Economic Analysis
S.No. Components As Per PC-I After Completion
a) Financial
Net Present Value (NPV) N/A
Benefit Cost Ratio (BCR)
Internal Financial Rate of Return (IFRR)
Unit Cost Analysis
b) Economic
Net Present Value (NPV) N/A
Benefit Cost Ratio (BCR)
Internal Economic Rate of Return (IERR)
23. Issues Faced during Implementation
There is a visible effect of the project in the district facilities, with the ability to mobilize additional staff on short notice and provision of requisite medicines the working of the facilities has improved considerably. Despite full commitment and hard work by implementing partners, there are some areas which remained under-performed due to various factors.
47
1. Full time Project Coordinator and hiring of key staff at Project management unit resulted into under-performance of component 3 of the project. 2. Hiring of Independent Third party M&E Firm. The firm could only be hired in August 2015, during extension of the project. 3. Delay in the procurement process of the six district management contracts resulted into restructuring of the project and deleted component 2 relating to civil works to rehabilitate health facilities infrastructure. 4. Contract out of DHQs at Lower Dir and Battagram. Due to uncertainty in the project life, DHQs could not be contracted out as envisaged in the project PC-1. 5. Improved Nutrition Services could not be implemented due to uncertainty in the project life as envisaged in the project PC-1. 6. Delays in release of Government Funds and reimbursement of expenditures against budget allocated from MDTF grant.
REASONS FOR PERFORMANCE GAP
1. Court litigation case on selection of Project Coordinator. This issue has been resolved now.
2. Court litigation cases during selection of firms for District Management contracts resulted into five months delay in procurement process.
3. Court litigation cases in District Buner and Lower Dir after executing management contracts in the districts (Strikes by District Health Administration)
4. Third Party M&E firm was advertised twice leading to delay in procurement of the firm.
5. DHQs could not be outsourced due to uncertainty and short left over time of the project.
Due to delay in notification and understanding of district administration on Fund Flow Mechanism leading to delayed transfer of funds to Implementing Partners. Due to Local Government act, a revised Fund Flow Mechanism has been approved ensuring timely transfer of funds to Implementing partners. This issue was timely resolved
24. Lessons learned
a) Project identification: The department of health KP has long since felt the need to bring in efficiencies in the healthcare delivery system through PPP. As part of the reform agenda various models of PPP models were tested. This project was the first of its kind and scale where the whole district primary and secondary care management was outsourced to reputable firms based on the Battagram model supported by Jica/SDPF initiative.
b) Project preparation: Many activities that were seemed appropriate were included initially in the project document, but were later on altered in the revision process of the project document. The intervention logic was relevant however, district level authorities were not taken fully on board during project preparation process which posed great challenges at the implementation stage. The financial flow mechanism was also a challenging job throughout the project life as GOP financial accounting
48
procedures are cumbersome and many levels are involved before it is released.
c) Project approval: Project PC-1 was prepared and approved in June 2011. Project was approved smoothly through ECNEC by April 2012.
d) Project financing: The project had two streams of funding to contractors for implementation of the project a. Component 1 Service delivery; i) Govt. of KP regular health budget
for districts and b. ii) MDTF grant in the project assignment account provided by the
World Bank. Donor released funds to GoKP on advance basis; however, the contract with implementing partners required release of funds on reimbursement basis. In contrast, GoKP regular health budget was provided to Contractors in advance at start of each quarter. In the first Financial Year of the project implementation in districts, GoKP share could only be transferred to contractors after lapse of 10 months; this considerable delay along with reimbursement mechanism for MDTF share has caused serious financial burden on contractors and affected the program activities. Further, the project coordinator had category II powers; it resulted into considerable delay in obtaining category 1 approvals for MDTF share payment to contractors. Lesson Learned:
Fund Flow Mechanism should be approved before start of the project and should be part of approved PC-1
MDTF share should also be in advance to contractors.
Project coordinator should be Category I officer or in case of Category II officer powers of category should be provided.
Counter financing should be there from ADP. It is very critical as the project has faced major issue during closure of the project. As per World Bank policy, World Bank project closure expenditure to be borne by the government. The project requested for grant but not yet released and staff is without salary for the last three months.
e) Project implementation:
Resistance from district health department staff: Implementing any such contracting out project where authority is transferred from traditional government official to private sector such resistance do take place. It was successfully handled in districts but however due to involvement of local political leaders implementing partner withdrew from Kohistan district. As mentioned earlier, the department needs to take on board district authorities including political leaders before launching the project and should be
49
implemented where there is willingness and acceptance to this model.
Court cases in Lower Dir and Buner were filed which delayed the implementation in these districts.
During its implementation six project coordinator were changed due to litigation cases which affected the project performance
The district took considerable time in understanding of Hub-approach and performance based mechanism
As explained earlier delays in release of funds – both Govt. and MDTF share affected its implementation.
PMU was not fully operational as PC was changed frequently and critical technical staff could not be hired.
25. Suggestions for Future Planning & Implementation of Similar Projects
Suggestions have already been given in relevant section, however they are summarized as under;
1. Take on board districts and involve them in preparatory phase 2. Strong local political Commitment is required to steer the process and
implementation 3. Orientation of key officials of concerned departments especially AG office 4. Ensure pre-financing to contractors and approved fund flow mechanism be part
of PC-1 5. Full time Project Director with category I powers 6. Most important: Contractors should be given at least 3-5 years contracts to
demonstrate results.
Submitted by: Signature
Name & Designation
Telephone No.
E-mail Address
Date
LIST OF ANNEXURES
S.NO. DETAIL ANNEXURE
1 Project Indicators & Performance A
2 List of Assets B
3 Final Financial Report (IFR) to the World Bank C
4 Districts Audit Reports D
50
Comments on the draft ICR sent by the Government of Pakistan
51
Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders
Not applicable
52
Annex 9. List of Supporting Documents
(a) World Bank Project Documents
Bank Operational Policies and Business Procedure OP/BP 8.00 (2014)
Pakistan Country Partnership Strategy (1) FY2011–14 (Report No. 53553-PK
(2010) and 65286-PK (2011) and (2) FY2015–19 (Report No. 84645-PK (2014)).
Grant Agreement and Project Agreement (Grant No. TF-11062PK) dated on April
12, 2012
Emergency Project Paper (Report No. 62125-PK) dated on March 26, 2012
Project Restructuring Paper (Report No. RES14560) dated on June 4, 2014, and
Project Restructuring Paper (Report No. RES19312) dated on June 30, 2015
Aide-Memoires (July 2012, Aug 2013, Jan 2013, April 2014, Oct 2014, March
2015)
Implementation Status Reports (ISRs) No. 1 – No. 9
Implementation Completion Memorandum for the JSDF supported Revitalizing and
Improving Primary Health Care in Battagram District, April, 2011.
Note of Cancelled Operation for Pakistan: Khyber Pakhtunkhwa and FATA
Emergency Recovery Project (Report No: NCO00002137) dated on June 29, 2012.
Implementation Completion and Results Report for Pakistan: Competitive Industries
Project for Khyber Pakhtunkhwa (Report No: ICR00003728). Draft dated on May
24, 2016.
Board Paper on “IDA18 – Special Theme: Fragility, Conflict, and Violence”. 2016.
(http://imagebank.worldbank.org/servlet/WDSContentServer/IW3P/IB/2016/06/03/0
90224b084391d73/1_0/Rendered/PDF/IDA18000specia0onflict0and0violence.pdf)
(b) Project Documents and Data
Government of Khyber Pakhtunkhwa. Khyber Pakhtunkhwa Health Sector Strategy
2010–17, December 2010
“Assessment of the Save the Children (US) Performance Based Incentive
Mechanism & Economic Analysis of the Project “Revitalizing & Improving Primary
Health Care in Battagram District” Oxford Policy Management. June 2010.
“Report on Health Facility Assessment”. Contech International Health Consultants.
June 2010.
“Final Report: End-Line Household Survey of the Project “Revitalizing and
Improving Primary Health Care in Battagram District”. Apex Consulting. April
2015.
Bureau of Statistics, Government of Pakistan. Pakistan Social and Living Standards
Measurement Survey (PSLM) 2012-13 and 2014-15 (provincial/district
disaggregation)
53
Map
