Document of The World Bank€¦ · Target Values Actual Value Achieved at Completion or Target Years Comments (incl. % achievement) ACHIEVED. While targets were not set, evidence

Document of The World Bank

Report No: ICR00003642

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-44320 & IDA-47210)

ON

CREDITS

IN THE AMOUNT OF SDR 3.9 MILLION (US$ 6.0 MILLION EQUIVALENT)

TO THE

INDEPENDENT STATE OF SAMOA

FOR A

HEALTH SECTOR MANAGEMENT PROGRAM SUPPORT PROJECT AND ADDITIONAL FINANCING

June 14, 2016

Human Development Sector Unit Health, Nutrition, and Population East Asia & Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective December 18, 2015)

Currency Unit = Samoan Tala

SAT$ 1.00 = US$ 0.39 US$ 1.00 = SAT$ 2.58

FISCAL YEAR (July 1-June 30)

ABBREVIATIONS AND ACRONYMS

Senior Global Practice Director: Timothy Evans

Practice Manager: Toomas Palu

Project Team Leader: Eileen Brainne Sullivan

ICR Team Leader: Neesha Harnam

INDEPENDENT STATE OF SAMOA Health Sector Management Program Support Project

CONTENTS

Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ............................................... 12. Key Factors Affecting Implementation and Outcomes .............................................. 43. Assessment of Outcomes .......................................................................................... 114. Assessment of Risk to Development Outcome ......................................................... 185. Assessment of Bank and Borrower Performance ..................................................... 186. Lessons Learned ....................................................................................................... 217. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 22Annex 1. Project Costs and Financing .......................................................................... 23Annex 2. Detailed Project Components and Outputs by HSP Outcome ...................... 24Annex 3. Economic and Financial Analysis ................................................................. 32Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 37Annex 5. Beneficiary Survey Results ........................................................................... 39Annex 6. Stakeholder Workshop Report and Results ................................................... 40Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 41Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 57Annex 9. List of Supporting Documents ...................................................................... 59Annex 10. List of Indicators and Arrangements for Results Monitoring (PAD) ......... 60Annex 11. Summary of Agreements ............................................................................. 68Annex 12. SWAp Governance and Administration ...................................................... 69

MAP IBRD 33472

i

A. Basic Information

Country: Samoa Project Name:

SAMOA HEALTH SECTOR MANAGEMENT PROGRAM SUPPORT PROJECT

Project ID: P086313 L/C/TF Number(s): IDA-44320,IDA-47210

ICR Date: 6/14/2016 ICR Type: Core ICR

Lending Instrument: SIL Borrower: GOVERNMENT OF SAMOA

Original Total Commitment:

SDR 1.90M

Disbursed Amount:

SDR 3.90M (Includes Additional Financing of SDR 2.00M)

Revised Amount: SDR 3.90M

Environmental Category: B

Implementing Agencies: Ministry of Health (MOH) and National Health Service (NHS)

Cofinanciers and Other External Partners: Ministry of Finance and Trade (MFAT; New Zealand) and Department of Foreign Affairs and Trade (DFAT; Australia), UNFPA, UNICEF and WHO. Pool partners were those providing financing (MFAT, DFAT, IDA, GOS); together with UNFPA, UNICEF and WHO they are referred to as development partners (DPs) in the Joint Partnership Agreement (JPA). B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 05/22/2007 Effectiveness: 10/21/2008 10/21/2008

Appraisal: 04/09/2008 Restructuring(s): 12/03/2013

Approval: 06/19/2008 Mid-term Review: 04/06/2011 12/14/2011

Closing: 12/31/2013 12/18/2015 C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Moderately Unsatisfactory

Risk to Development Outcome: Moderate

Bank Performance: Moderately Unsatisfactory

Borrower Performance: Moderately Unsatisfactory

ii

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Unsatisfactory

Government: Moderately Satisfactory

Quality of Supervision: Moderately Unsatisfactory

Implementing Agency/Agencies:

Moderately Unsatisfactory

Overall Bank Performance:


Overall Borrower Performance:


C.3 Quality at Entry and Implementation Performance Indicators

Implementation Performance

Indicators QAG Assessments

(if any) Rating

Potential Problem Project at any time (Yes/No):

No Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

Yes Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:


D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

Central government administration 25 25

Health 75 75

Theme Code (as % of total Bank financing)

Child health 17 17

Health system performance 33 33

Injuries and non-communicable diseases 17 17

Other communicable diseases 16 16

Population and reproductive health 17 17 E. Bank Staff

Positions At ICR At Approval

Vice President: Keith Hansen James Adams

Country Director: Franz Drees-Gross Nigel Roberts

Practice Manager/Manager:

Toomas Palu Fadia M. Saadah

Project Team Leader: Eileen Brainne Sullivan Muhammad Ali Pate

ICR Team Leader: Neesha Harnam

iii

ICR Primary Author: Neesha Harnam F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document) This ICR uses the PDO stated in Schedule 1 of the Financing Agreement, which reads: “The objective of the Project is to improve the Recipient's effectiveness in managing and implementing the Program based on the use of results from sector performance monitoring.” This is identical to the first-tier objective of the Project Appraisal Document (PAD); the PAD also includes the objective of the Health Sector Plan (HSP) as a second-tier objective, namely “The HSP’s aim in the medium-term (FY2009-2013) is to improve access to, and utilization of effective, efficient and quality health services to improve the health status of the Samoan population.” Revised Project Development Objectives (as approved by original approving authority)No revisions were made to the PDOs. (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

KPI #1 (SWAp): Percent health sector budgets and disbursements conform to policy objectives and HSP priority areas Value quantitative or Qualitative)

Beginning of SWAp 90% >90%

Date achieved 10/21/2008 Year 4 onward 12/18/2015 Comments (incl. % achievement)

ACHIEVED. The SWAp aimed to introduce a mechanism for the government to better align its resources with priorities. This indicator was tracked using the proportion of budget expenditure by the various program activities.

KPI #2 (SWAp): Share of annual outpatient visits by poorest quintile of population (indicator of equity of access – HIES)

Value quantitative or Qualitative)

N/A No target set

Achieved according to proxy data on vaccinations and

antenatal care Date achieved 10/21/2008 12/18/2015

iv



approval documents)




Comments (incl. % achievement)

ACHIEVED. While targets were not set, evidence suggests that improvements were seen in terms of equity of access. The March 2009 summit was supposed to establish the baseline and target for the rest of the program, but due to a lack of data this was not done. However, proxy data on vaccinations and antenatal care show improvements based on DHS data by quintile. The share of population receiving all basic vaccinations generally improved from 2009 to 2014 from 20.0 to 48.7 (lowest); 15.1 to 51.7 (second); 29.3 to 40.5 (middle); 31.2 to 47.1 (fourth); 29.5 to 45.3 (highest). Similarly, the percentage of women receiving antenatal care from a health care provider from 2009 to 2014 generally improved from 86.5 to 89.1 (lowest); 91.6 to 91.8 (second); 95.1 to 92.3 (middle); 93.9 to 96.7 (fourth); 99.1 to 98.2 (highest).

KPI #3 (HSP): Prevalence of diabetes Value quantitative or Qualitative)

21.5% No target set 45.8%

Date achieved 2002 (STEPS Survey) 2013 (STEPS

Survey)


NOT ACHIEVED against HSP target value of 25-50% decrease. Figures reported represent prevalence among adults 25-64 years old. While it is possible that increased access (see KPI #2) may be responsible for the increase in prevalence this is likely only a partial explanation for the observed increase.

KPI #4 (HSP): Perinatal mortality rate (22 weeks gestation to <7 days after birth per 1000 live births/year – WHO definition) Value quantitative or Qualitative)

7 per 1000 live births No target set 8 per 1000 live

births

Date achieved 2009

(DHS)

2014 (DHS)


PARTIALLY ACHIEVED. DHS 2014 definition is 28 weeks gestation to <7 days after birth per 1000 live births/year, and not the WHO definition. The baseline figure may have been artificially low, as a survey of all public health facilities in 2008 place the perinatal mortality rate at 23.9 per 1000 live births (in line with data obtained from the census), which suggests improvement over time. IMR reduced from 20.4/1000 (2006 census) to 15.4/1000 (2011 census); U5MR improved from 24.4/1000 to 19.4/1000 (2011 census). Maternal mortality also declined based on DHS data from 40.2/100,000 (2009) to 26/100,000 (2014).

KPI #5 (HSP): Adolescent birth rate (Annual births to women 15-19 yrs) Value quantitative or Qualitative)

42.60 No target set 39.00

Date achieved 2006

(Census)

2011 (Census)


NOT ACHIEVED against the HSP target for 2018 of 30. Defined as annual births to women 15-19 years old.

v



approval documents)




KPI #6 (HSP): Incidence of water- and food borne infections Value quantitative or Qualitative)

670 No target set 135

Date achieved 2009 (PATIS; Cases of

clinically suspected typhoid)

2014 (PATIS;

Cases of clinically suspected typhoid)


PARTIALLY ACHIEVED. While there was some concern that typhoid cases reported in 2014 may not reflect the total number of cases due to timing of surveillance reports, a steady and substantial decline has been observed over the years of the Project. It is also worth noting that not much improvement was seen among children with diarrhea: 68% (DHS 2009) to 62% (DHS 2014) while the proportion of untreated children with diarrhea increased from 3% (DHS 2009) to 7% (DHS 2014).

KPI #7 (HSP): Injuries in children (under 15 years old) Value quantitative or Qualitative)

237 (population 70,489)

No target set 282

(population 71,890)

Date achieved 2007/2008;

population data is from 2006 census

2012/2013;

population data is from 2011 census


NOT ACHIEVED. Numerator reflects number of hospital admissions reported for children under 13. Denominator reflects population of children under 15. Slight increase seen on a percentage basis. 2010/2011 data had 267 admissions.

KPI #8 (HSP/ICR Indicator): Percentage who currently smoke tobacco Value quantitative or Qualitative)


Date achieved STEPS (2002)

STEPS (2013)


PARTIALLY ACHIEVED against the HSP target for 2018 of 20.2%. While the target has not fully been achieved, tobacco control progress to date has been notable.

KPI #9 (HSP/ICR Indicator): Percentage of those who have drunk (alcohol) in the past 12 months Value quantitative or Qualitative)



STEPS (2013)


ACHIEVED/EXCEEDED against the HSP target for 2018 of 26.5%.

KPI #10 (HSP/ICR Indicator): Percentage of people with high levels of physical activity

vi



approval documents)







STEPS (2013)


ACHIEVED. A high level of physical activity is defined as >3000 MET minutes/week. While specific targets were not set, available evidence from the STEPS survey suggests an improvement over time.

(b) Intermediate Outcome Indicator(s)



approval documents)

Formally Revised

Target Values



IOI #1 (SWAp): MTEF and related Procurement Plan updated and adjusted based on recommendations from sector reviews. Value quantitative or Qualitative)

No MTEF Complete Complete

Date achieved 2008 12/18/2015 12/18/2015 Comments (incl. % achievement)

PARTIALLY ACHIEVED. MTEF 2009 and 2012 covered three-year periods, and the MTEF and related procurement plan for 2016-2019 is currently under preparation. However, the MTEF has not been annually updated since 2012.

IOI #2 (SWAp): Key Sector Partners’ corporate Plans and Government investments aligned with HSP priorities Value quantitative or Qualitative)

Not aligned Complete Complete;

alignment seen

Date achieved 2008 Year 2 onward 12/18/2015 Comments (incl. % achievement)

ACHIEVED. Alignment efforts are undertaken through the Health Advisory Committee which meets regularly. In addition, the MOH has instituted a combined plan that incorporates all investments in the sector.

IOI #3 (SWAp): Percentage of SWAp Program funds released according to agreed scheduleValue quantitative or Qualitative)

0% 100% 100%

Date achieved 9/25/2008 Year 2 onward 12/18/2015


ACHIEVED. Funds were released based on commitments by the government, though there were some delays in disbursement during the course of the Project. It is worth noting that the SWAp incurred budget overruns, which were covered by DPs and Government towards the end of the FY 2015-2016.

IOI #4 (SWAp): Number of reported drug stock-outs by facility Value Unknown <10% of baseline 2%

vii



approval documents)

Formally Revised

Target Values



quantitative or Qualitative) Date achieved 2008 Year 3 onward March 2012


ACHIEVED as 2% is negligible. While there is still no data on stockouts at the national level, data from March 2012 notes a 2% stockout rate in rural areas and 95% availability of essential medicines at all public health facilities at any one time.

IOI #5 (SWAp): Health expenditure as percentage of government budget expenditure Value quantitative or Qualitative)

15% >15% 16.1%

Date achieved 9/25/2008 12/18/2015 10/30/2015 Comments (incl. % achievement)

ACHIEVED.

IOI #6 (SWAp): Stakeholders participation in program planning and implementation reviews Value quantitative or Qualitative)

None Full Full and regular

consultation

Date achieved 9/25/2008 Year 3 onward 10/30/2015


ACHIEVED. Stakeholder participation was evident in Annual Summits and Joint Reviews as documented in an Analysis of Review Missions (Annex 7 of the Evaluation of Samoa Health Sector Management Program conducted in 2013).

IOI #7 (SWAp): Disaggregation of data by sex, age and domicile enhances planning for services


None Full

Surveys disaggregated (STEPS 2013,

DHS 2014) Date achieved 9/25/2008 Year 3 onward 10/30/2015 Comments (incl. % achievement)

PARTIALLY ACHIEVED. STEPS 2013 and DHS 2014 surveys were disaggregated, though it is unclear that this has enhanced planning.

IOI #8 (SWAp): MOH Financial Audits submitted on time and action plan agreed for resolving outstanding issues Value quantitative or Qualitative)

Unknown Complete Complete

Date achieved 2008 Year 1 onward 10/30/2015 Comments (incl. % achievement)

ACHIEVED.

viii



approval documents)

Formally Revised

Target Values



IOI #9 (SWAp): DHS and other statistical reports completed within stated timeframe and made public


Yes with delays, available in public domain

Complete

Yes and with less delays than baseline and made available in public domain

Date achieved 9/25/2008 Year 1 onward 10/30/2015 Comments (incl. % achievement)

ACHIEVED.

IOI #10 (HSP): People aged 18 years and over overweight or obese Value quantitative or Qualitative)

Overweight: 85.62% Obese: 56.0%

No target set Overweight: 89.1%

Obese: 63.1%

Date achieved 2002

(STEPS)

2013 (STEPS)


NOT ACHIEVED against HSP target of: Overweight: 63.9%; Obese: 42% (25% reduction from baseline). In addition, the share of the population meeting the recommended intake for fruit and vegetable consumption (5+ servings/day) decreased from 8% of males and 3% of females (DHS 2009) to 2% and 1% (DHS 2014), respectively.

IOI #11 (HSP/Core): Percentage of children under 1 year received at least one dose of measles vaccine Value quantitative or Qualitative)

63.00%

No target set Comparable data

not available

Date achieved 2009

(DHS)


ACHIEVED against HSP target for children 0-29 months of 90%. This is similar to a core indicator on children immunized. The Health Sector Plan Midyear Review notes that about 85% of children under one-year-old have received immunization against measles in 2012. More recent comparable data was not available due to changes in the national vaccine program. However, annual immunization rates from the UNICEF/WHO joint reporting form suggests improvement over time from 49% (2008/09) to 91% (2013/14).

IOI #12 (HSP): Improved medical waste management Value quantitative or Qualitative)

New HCWM Plan not yet implemented

No target set Improved

Date achieved 2008 12/18/2015

ix



approval documents)

Formally Revised

Target Values




PARTIALLY ACHIEVED. While a HCWM Plan was in place in 2008, a revised Plan was signed in June 2013. Investments in HCWM infrastructure (incinerators, medical waste bins) and training (waste management and the importance of HCWM) were made, though training participation was weak. Regular supervision missions took place to ensure compliance with the HCWM Plan, though it appeared that compliance and funding was limited.

IOI #13 (HSP): Primary care utilization by gender, age, domicile Value quantitative or Qualitative)

No information available. No target set No information

available

Date achieved 2008 12/18/2015 Comments (incl. % achievement)

UNKNOWN.

IOI #14 (HSP/Core): Antenatal care coverage for three visits Value quantitative or Qualitative)

58.40% (4+ visits)

No target set 72.90%

(4+ visits)

Date achieved 2009

(DHS)

2014 (DHS)


ACHIEVED/EXCEEDED against HSP target of 70% (4+ visits). Indicator in the RF originally read antenatal care coverage at least one visit, same as one of the core indicators. This was revised to three visits in later ISRs, but the DHS reports 4+ visits which is reported here. Median age of visit was also earlier – from 6 months to 5.9 months (2009 to 2014 DHS).

IOI #15 (HSP): Proportion of Rheumatic Heart Disease patients complying with treatment Value quantitative or Qualitative)

84% No target set 80%

Date achieved 2008 2015


ACHIEVED. While no target was set, evidence suggests improved compliance that was sustained over time. Data from 2006 and earlier suggests compliance was below 50%. Screening efforts began in 2008, and the proportion of RHD patients complying with treatment has been maintained at >80% since then, reaching a peak of 86% in 2010.

IOI #16 (HSP): Staff mix and distribution according to national standards


N/A No target set

The NHS issued a Workforce

Development Plan (April 2013) which identifies the needs

and gaps to be filled to meet

quality standards. Date achieved 2008 12/18/2015

x



approval documents)

Formally Revised

Target Values




PARTIALLY ACHIEVED. The national standards have been articulated in the Workforce Development Plan. Based on the number of annual practicing certificates issued, there has been an increase in the health professional to population ratios since 2008, particularly for nurses though the ratio for doctors has declined slightly.

IOI #17 (HSP): Evidence of performance monitoring leading to policy and regulatory action to improve health services


N/A No target set

Many frameworks and policies

developed but limited evidence of

improvement in health services.

Date achieved 2008 2015


PARTIALLY ACHIEVED. The Monitoring and Evaluation Framework was developed in 2011, and all health professions have developed standards (2013/2014). A key achievement was the development of an interim HRH database focused on identifying staff gaps by facility and qualification; however, this database is not yet available for use at NHS. Additionally, the Tobacco Control Policy 2010 enforced implementation of and public compliance of the Tobacco Control Act 2008. While there were General Prevention and Health Promotion Policies developed focusing on Health Information Systems (HIS) development and policy, these were not fully exploited to inform policy and develop regulatory action. In terms of performance monitoring, limited data on health outcomes (including NCDs), was collected and reported outside of national surveys, limiting the evidence base available in decision-making.

IOI #18 (HSP): Demonstrated outcomes of training plan by component


N/A No target set

Outcomes demonstrated in

Component 2 (HIS and coding

capacity) and Component 3

(DHS, M&E, HRH)Date achieved 2008 2015


PARTIALLY ACHIEVED. DHS trainings have been instrumental in taking the DHS from initial stages to report writing. Trainees of the M&E study tour developed the M&E Framework. Health information management training has equipped staff to improve HIS. HRH-related trainings resulted in the HRH country profile and establishment of an HRH taskforce.

IOI #19 (Core): Health personnel receiving training (Number) Value quantitative or Qualitative)

0 No target set 2125

(MOH Accounting Documents)

Date achieved 10/21/2008 18 Dec 2015

xi



approval documents)

Formally Revised

Target Values




A sizeable number of health personnel received training. Training included attachments in urology and orthopedics, and in health management. This allowed successful utilization of equipment procured under the SWAp, improvement of documents produced under MOH (Operational Manual, review of protocols and standards), and the development of leadership skills. Training included graphic design (2 people), improving their ability to produce and design multimedia materials for the health sector.

IOI #20 (Core): Health facilities constructed, renovated, and/or equipped (Number) Value quantitative or Qualitative)

0 No target set 13

Date achieved 5/31/2008 12/18/2015 Comments (incl. % achievement)

IOI #21 (Core): People with access to a basic package of health, nutrition, or population services (Number) Value quantitative or Qualitative)

No target set

Date achieved Comments (incl. % achievement)

Dropped from ISR #7 (July 22, 2012) onward.

IOI #22 (Core): Children receiving a dose of vitamin A (Number) Value quantitative or Qualitative)

No target set

Date achieved Comments (incl. % achievement)

Dropped from ISR #7 (July 22, 2012) onward.

G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual

Disbursements (USD millions)

1 10/28/2008 Satisfactory Satisfactory 0.00 2 06/11/2009 Satisfactory Moderately Satisfactory 0.25 3 01/07/2010 Satisfactory Moderately Satisfactory 0.25 4 05/12/2010 Satisfactory Moderately Satisfactory 0.88 5 02/20/2011 Satisfactory Moderately Satisfactory 1.47 6 03/03/2012 Moderately Satisfactory Moderately Satisfactory 2.17

xii

No. Date ISR Archived

DO IP Actual

Disbursements (USD millions)

7 07/22/2012 Moderately Satisfactory Moderately Satisfactory 2.17

8 01/15/2013 Moderately

Unsatisfactory Moderately Satisfactory 2.91



10 10/19/2013 Moderately Satisfactory Moderately Satisfactory 2.91 11 01/21/2014 Moderately Satisfactory Moderately Satisfactory 4.92 12 08/17/2014 Moderately Satisfactory Moderately Satisfactory 4.92 13 03/06/2015 Moderately Satisfactory Moderately Satisfactory 4.92 14 04/29/2015 Moderately Satisfactory Moderately Satisfactory 4.92



H. Restructuring (if any)

Restructuring Date(s)

Board Approved PDO

Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made

DO IP

05/20/2010 S MS 0.88

Additional financing (AF) for an amount of US$3.0 million equivalent was granted for post-tsunami activities within the health sector and reflected in the Program of Work (see Annex 2). No other changes were made.

12/02/2013 MS MS 2.91

Level 2 restructuring. No cost extension of the Project closing date by two years to December 18, 2015. Amendments to the Financing Agreements are dated December 3, 2013

xiii

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. At the time of appraisal, Samoa’s economic reforms, launched in 1993, had resulted in macroeconomic stabilization and comprehensive structural changes with real Growth Domestic Product (GDP) growth of over 3 percent realized in 2007 and steadily rising per capita incomes. However, the fiscal space of the Government and its ability to increase expenditures were constrained, with total revenues and grants expected to decline over the period 2006-07 to 2010-11 from 37.7 percent to 36.6 percent of GDP. 2. Over the past several decades the health status of Samoa’s population saw significant improvement, largely through the provision of basic health services. The country had high emigration rates among the population of working age, resulting in a relatively high dependency ratio mitigated by high remittance levels back to the country. For a number of years, Samoa had struggled with the classic epidemiological transition. The rising burden of NCDs, found increasingly among younger adults, posed serious risks to long term growth prospects. Although most infectious diseases were under reasonable control, acute respiratory infections and rheumatic fever remained significant causes of morbidity and, in the latter case, resulted in a high and growing demand on hospital services. Infectious diseases (with outbreaks of measles and dengue seen in recent years) and maternity-related illnesses accounted for over 30 percent of hospital admissions in 2003/2004, yet did not receive the appropriate level of funding. While Government of Samoa (GOS) policy favored cost-effective preventive services, the bulk of public resources were still directed towards less cost-effective hospital services. In early 2007, the Government developed a new Health Sector Plan (HSP) focused on dealing with: (i) rising non-communicable diseases; (ii) reproductive, maternal and child health; (iii) emerging and re-emerging infectious diseases; and (iv) prevention of injuries, given the significant share of the health burden associated with injury and poisoning among children. In addressing these issues, the Health Sector Plan (2008-2018) identified six priority areas and strategies: (i) health promotion and primordial prevention; (ii) quality health service delivery; (iii) governance, human resources for health and health systems; (iv) partnership commitment; (v) health financing; and (vi) donor assistance. 3. As part of an overall major public sector governance reform program, the GOS began implementation of a purchaser-provider split within the health sector with the passing of the Ministry of Health (MOH) Act 2006 and the National Health Service (NHS) Act 2006. The NHS Act that gave it the mandate for service delivery also created a separate Executive Board to oversee NHS operations aside from the MOH1. Under these arrangements: (i) the MOH was responsible for regulatory oversight of the health sector, including operational budgets of State health entities and human resources, monitoring of health system performance as well as primordial health promotion and prevention services; and (ii) NHS was the major Government agency responsible for the provision of public health care services, including secondary prevention, primary health care (PHC) and hospital services, though the Ministry of Finance (MOF) retained direct control over the budget of NHS.

1 The CEO of MOH sits on the board of the NHS.

2

4. Based on the HSP and in accordance with the shift towards a sector-wide approach (SWAp) across the whole of Government, the GOS requested key development partners (the New Zealand Agency for International Development (NZAID; later MFAT), the Australian Agency for International Development (AusAID; later DFAT) and the World Bank) to establish a health SWAp Program as a basis for assistance to implement the first five-year period of the HSP (FY2009-2013). Further, the GOS was promoting a more coordinated approach as a means for increasing the effectiveness of development assistance. This approach is particularly important given the small island state context where there is limited capacity in an environment vulnerable to external shocks and natural disasters. A summary of the SWAp governance and administration is provided in Annex 12. The GOS had prior experiences with the Education Sector Program II (established in 2006) and the Water Sector Support Program (2005-2010), both of which had SWAp aspects. 5. There was strong rationale for the World Bank’s involvement in the HSP. First, the Bank had previous experience in working in Samoa, having provided 15 credits (US$87.8 million) between 1974 and 2008 for support to various sectors, including health. The preceding World Bank Project (the Samoa Health Sector Project (P064926)) closed at the end of 2006 and had been rated Moderately Satisfactory. Second, the Bank’s global and regional expertise in design and implementation of sectoral programs could be adapted to the Samoan context. Third, by leveraging IDA’s support together with other DPs (NZAID and AusAID), the proposed Program represented a new way of engagement with the health sector in the country and of building institutional capacity. Fourth, the Bank engagement in Samoa at both macro as well as sector levels provided an avenue for linking health sector development efforts within a whole-of-government context. Finally, the Bank’s non-lending knowledge and analytical products effectively complemented the Program context. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 6. This ICR focuses on the PDO stated in Schedule 1 of the Financing Agreement (FA) dated July 23, 2008, which reads: “The objective of the Project is to improve the Recipient's effectiveness in managing and implementing the Program based on the use of results from sector performance monitoring” (FA, p. 5). It is worth noting that in the PAD, a two-tiered2 system of development objectives was employed, the first of which corresponds with that of the FA. The second-tier PDO reflects the objective of the HSP,3 and states that “The HSP’s aim in the medium-term (FY2009-2013) is to improve access to, and utilization of effective, efficient and quality health services to improve the health status of the Samoan population” (PAD, p. 5). The Joint Partnership Agreement (JPA) identifies two objectives (institutional and development objectives) which correspond to the first and second-tier PDOs of the PAD.

2 A pre-review of the Project was conducted by the Independent Evaluation Group on 8/24/2015, who distinguished between the first- (Health Sector Program) and second-tier (SWAp Process) PDOs. The same tiered approach is used in discussed the PDOs here, though the tiers described in the ICR refer to the PDOs as listed in the PAD. 3 While the rationale for a SWAp was explicitly expressed in the HSP, the areas of focus in the HSP were different than the PDO. The HSP notes that the “SWAp for quality health for Samoa will focus on: (i) genuine and effective partnerships; (ii) donor participation, coordination and contribution; (iii) consistent collaborative planning, monitoring and evaluation arrangements; (iv) human resource development for health (HRH) and skills transfer; and (v) a Samoan perspective of health.”

3

7. A complete list of indicators (KPIs and IOIs) used throughout the Project is presented in the ICR Data Sheet and summarized here. Key indicators for assessment of the SWAp (Part B of the results framework in the PAD) are as follows: (i) Percent health sector budgets and disbursements conform to policy objectives and HSP priority areas; and (ii) Share of annual outpatient visits by poorest quintile of population (indicator of equity of access – HIES). Areas of focus and specific KPIs included for assessment of the HSP (Part A of the results framework in the PAD) were: (i) Control of non-communicable diseases (Prevalence of diabetes); (ii) Improved maternal and child health (Perinatal mortality rate); (iii) Universal access to reproductive health services (Adolescent birth rate); (iv) Control of communicable diseases (Incidence of water- and food borne infections); (v) Injury prevention and management (injuries in children under 15 years). The complete results framework as approved and presented in the PAD is provided in Annex 10. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 8. The PDOs were not revised. 1.4 Main Beneficiaries 9. The PAD does not specify a “primary target group,” though the economic evaluation notes that “a main thrust of the Program is to support institutional development for better policy planning, the benefits of which are spread over a range of beneficiaries and therefore difficult to quantify.” The second-tier PDO listed in the PAD specifies that the target population of the project is the Samoan population. Given the link between the first- and second-tiers, the beneficiaries were considered to be the Samoan population. 1.5 Original Components 10. A detailed description of the three components is available in Annex 2 and is outlined here. The Project components listed in the PAD were more comprehensive than those listed in Schedule 1 of the Original FA. Nonetheless, these differences were not found to be substantive. A summary of the Project Components is as follows: Component 1 (Health Promotion and Prevention) aimed to support the transformation of the

health sector from a narrower medical orientation towards a health and wellness model. Central to MOH’s efforts to reorient the sector was health promotion.

Component 2 (Enhancement of Quality Health Care Service Delivery) aimed to support improvement of the various dimensions of the quality of health care, particularly at the primary care level.

Component 3 (Strengthening Policy, Monitoring and Regulatory Oversight of the Health System) aimed to support the MOH with its role of coordination, policy development, monitoring and regulation in the health sector. More specifically, this component aimed to support implementation of the Monitoring and Evaluation (M&E) system to monitor the Program and SWAp performance and the incremental operating costs for Program implementation.

1.6 Revised Components

4

11. Project components were not revised though, as part of the SWAp redevelopment4, a further outcome area was added in 2011, namely “improved risk management and response to disasters, emergencies and climate change.” While no formal amendments were made to the HSP and the Project was not restructured to reflect this outcome area, its addition5 is reflected in subsequent plans and programs of work. 1.7 Other significant changes 12. A summary of Agreements is provided in Annex 11. Amendments to the Financing Agreements, Cr. 4432-WS and Cr. 4721-WS (December 3, 2013), extended the closing date of the project by two years at no cost. The Partnership Agreements were also amended to extend the end date of the partnership. In addition, following the tsunami that occurred in 2009, the Project allocated SWAp resources (obtained from an additional US$3.0 million equivalent of World Bank funding) to post-tsunami activities within the health sector. Additional funds were provided through the Additional Financing Agreement (May 20, 2010) and reflected in the change in the Program of Work (POW). In particular, the additional proportion of costs6 (from 43 percent in 2009-2010 to 67 percent in 2010-2011) allocated to Component 2 partly reflects the incorporation of post-tsunami activities. These activities are described in the February 2010 Aide Memoire, and listed in Annex 2 of this ICR.

2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 13. Soundness of Background Analysis: Project preparation was supported by a grant of $500,000 from the Japan International Cooperation Agency (TF90307) which was largely spent on consultant services. The background analysis conducted was strong in that it provided a good overview of the health issues in Samoa using available evidence on the burden of disease, but there was room for improvement. The Project would have benefited from background analysis that also included an inventory of tools and policies in place and from using this information to refine the Project indicators. For example, IOI #12 of “improved waste management system” lacked a target describing what constitutes improvement. More broadly, the achievement of the PDO was predicated on the assumption that the necessary tools and policies and an adequate monitoring and evaluation system were in place, which proved not to be the case as the Project progressed. Thus, while the background analysis correctly identified the health issues in Samoa, more in-depth analysis would have been beneficial where the PDO and related indicators were concerned. Finally, in terms of approach, alternatives such as traditional project support and Development Policy Lending were considered prior to deciding on the sector-wide approach. 14. Assessment of Project Design: It is important to note that the Project was not a traditional SWAp, as MFAT served as Coordinating Development Partner for most of the SWAp implementation period, though this role was later rotated to DFAT. The Project was aligned with the HSP in approach (SWAp) and scope (overlap in components). The development of HSP had

4 Further details regarding the redevelopment are available in Section 2.2. 5 It should be noted that the addition of an outcome area focused on disasters, emergencies and climate change represented an innovative and proactive approach. 6 Evaluation of Samoa Health Sector Management Program (Health SWAp). Philip Davies, AusAID HRF. May 21, 2013.

5

been a participatory process with the DPs providing input. The three components of the Project broadly overlapped with the first three strategies of the HSP, and the SWAp could be said to align with the last three strategies7. Though Project design followed good practices in SWAp design at the time, the design did not adequately address the implications of policy changes such as the MOH-NHS split, or the link between the results chain and activities as detailed below. 15. A key challenge of the Project was with regard to timing. The concept note review for the SWAp took place in 2007 with consultations taking place beforehand, shortly after the split between MOH and NHS. Project design incorporated several important lessons learned, including (i) the importance of government ownership to the success of SWAps; and (ii) the need for higher than average supervision. However, government ownership within the health sector initially proved to be difficult given the MOH-NHS split, which, while mentioned in the context of reforms within the health sector (PAD, p. 23), was not explicitly addressed in terms of implementation arrangements. In particular, while the implementation arrangements delineate responsibilities of the two agencies, their ability to successfully carry out their duties in the context of the split was not discussed. Closer or more appropriate supervision (tailored to needs as they arose) may have been able to mitigate these issues to some extent, particularly if the skill mix required for supervision had been considered among DPs (see Section 5.1(b)). 16. While the PAD included a results chain linking components to outcomes (PAD, Annex 9, Figure 1), it appeared to assume that the requisite programs, policies and systems were in place at the outset, which turned out not to be the case. Project design could have benefited from an improved results chain, as the link between HSP objectives and strategies, and the components and constituent activities were unclear as noted by an evaluation of the SWAp conducted in 2013.8 It appears that the absence of a clear results chain hampered achievement of the Project going forward, as discussed under Section 3. 17. Assessment of Quality at Entry: The PAD incorporated several lessons learned from two prior projects in Samoa as well as lessons from other SWAps in the region. A lesson learned from the Samoa Health Sector Management Project (P064926) that would have been useful to include was that of clear and measurable project development objectives, including the use of appropriate indicators and early establishment of baselines. SWAp indicators were process indicators, and only a subset of outcome indicators were regularly reported on as part of the Results Monitoring Arrangements presented in the PAD as shown in Annex 10. Monitoring and Evaluation (M&E) design was weak and the results chain could have been strengthened. While the team involved in Project preparation proactively held a Quality Enhancement Review early in the Project cycle (March 5, 2008), documents available at the time did not provide the Review Panel with sufficient information in some areas. For example, a Results Framework was absent. The recommendations of the Panel appear to have been taken into consideration to some extent (e.g. the number of Health Summits was reduced to one instead of two as originally proposed), other issues highlighted by

7 HSP six priority areas and strategies are: (i) health promotion and primordial prevention; (ii) quality health service delivery; (iii) governance, human resources for health and health systems; (iv) partnership commitment; (v) health financing; and (vi) donor assistance. 8 Evaluation of Samoa Health Sector Management Program (Health SWAp). Philip Davies, AusAID HRF. May 21, 2013.

6

the Reviewers appear to have been only partially addressed. For example, the Panel noted the need to establish a clear link between the health system/status in Samoa and interventions proposed in the HSP, and cautioned against possible procurement issues, recommending against being too conservative. Largely due to these shortcomings, Bank performance in ensuring quality at entry was rated Moderately Unsatisfactory (Section 5.1). 18. The Project represented the first health SWAp in the country, and overall risk was considered Moderate. There was high-level government commitment to the Project, and the SWAp modality was seen as a natural next step for the country given its success in Project implementation and the presence of multiple development partners. In general, the assessment of risks and mitigation measures were appropriate. Where the split is concerned, the PAD notes that, “conflict between health sector entities over evolving responsibilities and accountabilities within the sector and a consequent inability for sector entities to maintain an open and participatory sector dialogue” (p. 13) was a high risk, and identified suitable mitigation measures. In hindsight, these mitigation measures may have been inadequate as understanding of responsibilities and accountabilities of the MOH and NHS were still being developed after the split, as well as the dual role of MOH as implementing agency of two components and coordinator of DP inputs to the sector. 2.2 Implementation 19. Implementation progress was initially slow (only about 40 percent of planned funding had been expended by the end of September 2011 or three years after effectiveness), and the Project faced challenges on several fronts. Several major events occurred over the course of the Project, namely an H1N1 epidemic (2009), a tsunami (2009), and Cyclone Evan (2012), presenting a difficult environment for implementation and absorbing capacity of the health sector. The flexibility of the SWAp allowed it to quickly provide additional funds (US$3.0 million equivalent in IDA funds) in response to the tsunami. While ISR #7 (June 2012) noted that the GOS was likely to achieve the PDOs if implementation continued at the pace observed in the past six months, PDO achievement was downgraded to MU a few months later (January 2013) due to delays under Component 2 and disbursement, and a lack of progress on Project indicators. Over-programming of the POW and a resulting shortfall in funds was also seen at that time. The POW was subsequently revised and the Project received a two-year no cost extension for completion of remaining infrastructure works. Progress was seen in the completion of non-infrastructure activities by October 2013, resulting in a rating change to MS (October 2013). However, further challenges (procurement, monitoring and evaluation, overall project management) resulted in a downgrade of PDO rating back to MU prior to Project closing (ISR #15; November 2015). Key factors affecting implementation are described below: 20. Program of Work: Development of the POW has been criticized in prior reviews of the SWAp for having been a bottom-up instead of a top-down process. In this context, it is important to note that the bottom-up approach taken may have been successful if accompanied by a clear process of prioritization. The POW was developed based on solicitation of proposals for activities to be completed within the SWAp envelope, with activities falling broadly within the six HSP outcome areas9 considered eligible for financing. Activities were then prioritized, with component

9 HSP outcome areas are: (i) Improved Healthy Living through Health Promotion and Primordial Prevention; (ii) Improved Prevention, Control, and Management of Chronic and NCDs; (iii) Improved Prevention, Control, and

7

focal points involved in the prioritization process. Many activities unable to be funded in one year were rolled over to the next. The POW thus evolved into an increasingly large and varied set of activities over the years. In general, evidence of systematic prioritization and selection of activities (or specific mix of activities) emphasizing use of data from sector performance monitoring was not clearly seen. While the Medium-Term Expenditure Framework (MTEF) was conducted and did provide some evidence regarding resource allocation, it was not regularly updated. It was also unclear as to why some activities were selected and others were not, and why, when overruns occurred, some activities were dropped while others were retained. In some cases, people were informed that an activity would be funded through the SWAp only to be told later that the activity was being dropped. Similarly, there was limited evidence of meaningful policy dialogue between DPs and GOS. Given these challenges, management of the SWAp went from an initial focus on overall strategy and achievement of outcomes, to a focus on outputs and activities. The risk that the SWAp would deliver a wide array of inputs and outputs to the sector without the desired impact on health outcomes was noted by March 2012 (Aide Memoire). 21. A significant and unexpected event was the redevelopment of the Tupua Tamasese Meaole (TTM) Hospital into a teaching hospital supported under the Government of the People’s Republic of China assistance program (2009/2010). This resulted in additional delays, as efforts were made to ensure that works were not duplicated with those conducted under the SWAp. Further, despite the stated emphasis of the HSP on promotion, prevention and primary care services, there appears to have been a particular disconnect between resource allocation and these priority areas, in part due to the need to develop the facilities at the new TTM Hospital. 22. Limited capacity, high turnover and weak communication: The SWAp was challenged by limited capacity, which had implications in many aspects of the Project. First, the limited capacity of the MOH meant that it was a challenge to address and remedy weaknesses that arose (ISR #9, June 2013). The strong emphasis on implementation of SWAp activities and the short timeframe in which activities needed to be carried out (with the exception of the first POW, subsequent POWs were on an annual basis) meant that there was limited capacity remaining for M&E, which was handled by a single person. Individuals were also required to perform activities beyond their area of expertise. For example, part of the reason for cost overruns was the costing of infrastructure activities by technical specialists in health instead of those with a more appropriate background e.g. engineering. Poor communication further complicated these issues, as unsynchronized inputs were provided to the architect from the different agencies and as personnel involved changed. The ICR mission found that on occasion, architects involved were not clear of the budgetary envelope when designing a building. Further, while the frequency of the Health Summit and Joint Review meetings was defined and the need for additional supervision was noted in the PAD, implementation was challenged by frequent turnover both on the part of the GOS and DPs. An independent evaluation of the SWAp which reviewed participation in missions described the monitoring process by DPs as hampered by “revolving door” staffing (Phillip Davies, May 2013).

Management of Communicable Diseases and Infectious Diseases; (iv) Improved Sexual and Reproductive Health; (v) Improved Maternal/Child Health; (vi) Improved Health Systems, Governance and Administration. A seventh area was later added; namely (vii) Improved Risk Management and Response to Disasters, Emergencies and Climate Change.

8

23. Program redevelopment in place of mid-term review: The midterm review was recast as a program redevelopment,10 which may have limited the opportunity to assess shortcomings of the SWAp in a candid manner and to course-correct. The redevelopment led to simplification in some processes (e.g. procurement) and provided additional clarification surrounding roles and responsibilities. In addition, the redevelopment resulted in the revision of the format of the POW, with activities grouped under the seven HSP Outcome areas which, in turn, could then be mapped onto the three SWAp components. While a mid-term review of the HSP was conducted, this only took place in 2013. An independent evaluation of the SWAp was also conducted in 2013, and it appears that recommendations made, such as to focus on specific priority projects (Philip Davies, p. 34) were taken into consideration, at least based on activities completed following the review. 2.3 Monitoring and Evaluation Design, Implementation and Utilization 24. Challenges in Project implementation were closely linked to weaknesses in monitoring and evaluation. Monitoring and evaluation of the Project faced challenges on two fronts: (i) development of M&E and HIS systems; and (ii) assessment of key indicators in the results framework over the course of the Project. These are further described below. 25. Design: The Results Framework of the PAD consisted of 25 indicators in total (Part A and Part B), with Part A consisting of a subset of overall HSP results indicators and Part B consisting of SWAp process indicators. Only a subset of 11 indicators (largely SWAp process indicators) were monitored as part of the results agreement, which may have resulted in a weaker emphasis on achievement of health outcomes than was desirable. An evaluation of the SWAp conducted in May 2013 noted that some indicators used were “poorly defined, lack targets or benchmarks and/or are likely to be difficult to collect and interpret” (Philip Davies). Further, while the use of results from sector performance monitoring was emphasized in the PDO, health outcomes indicators that were part of the results framework were those collected from large and infrequent surveys. Finally, even among the HSP indicators with targets, some targets may have been out of reach, such as a 25-50 percent reduction in diabetes prevalence. Underpinning these challenges was the lack of a robust HIS or M&E system that collected reliable data on a regular basis though support to the implementation of an M&E system was foreseen in the Project design under Component 3. 26. Several aspects of M&E specified in the PAD lacked clarity. In particular, the PAD notes that sector M&E reports would be discussed during the Health Summits, the Joint Review meetings and with MOF on a quarterly basis as part of the government’s budget monitoring process (p. 45). However, the PAD does not specify what constitutes a sector M&E report, and the ICR mission did not find evidence of discussions driven by sector M&E reports. Similarly, the PAD notes that the SWAp would develop a mechanism to encourage the use of the information collected in day-to-day decision-making within the different agencies and stakeholders, but does not provide further details on this proposed mechanism. 27. Implementation: Technical assistance for HIS fell through in the initial stages of the SWAp, and had knock-on adverse effects on data collection and analysis going forward. The

10 Neither the GoS Mid-Term Review nor the pool partners’ Mid-term Evaluation identified in the JPA took place. However, a Mid-Term Review of the HSP was conducted in 2013 by an external consultant (Joan McFarlane).

9

general need to improve M&E efforts was echoed over the course of the SWAp through multiple reports and documents (a total of four consultancy reports were developed in 2013/2014 focusing on HIS). An M&E manual focusing on Health Sector Performance was developed by the MOH in 2009, but did not provide baselines or targets for any of the indicators mentioned while the M&E framework only became available in 2011. Annex 1 of the M&E manual lists indicators used to monitor Health Sector Performance and makes it apparent that there was a need for improvement or development of institutional systems in order to assess the indicators. By September 2009, only three of the 25 indicators (Part A and Part B) had baseline data available. While it is not unusual for baseline data to be missing at the beginning of an operation, the lack of baseline data, targets, and definitions continued over the course of the Project. These reflected broader M&E weaknesses that occurred despite efforts such as the establishment of a Health Sector Information and Communication Technology (ICT) Taskforce in 2010 and a separate taskforce established through a cabinet directive in 2014. The final Aide Memoire of the Project (November 2015) notes that the SWAp should have focused more intently on HIS in the early years, including staff development in this area. M&E also presented challenges outside the GOS. The review conducted by DFAT and the Health Resource Facility (HRF) noted weaknesses in record-keeping among development partners, resulting in multiple, inconsistent and sometimes contradictory data being recorded. 28. Utilization: In the absence of a robust M&E system, utilization of data in decision-making does not appear to have routinely taken place. While results from large-scale surveys provided vital information on the health status of the country, such data was only available every few years, with a lack of data on sector performance over the course of the SWAp. Instead, it appears that the use of multiple forms and registries (some of which may have been outdated) continued over the course of the SWAp, with weak feedback loops and limited dissemination of data. An emphasis on achievement of results in the form of improved health outcomes, and a clear link between plans, expenditures and results was largely lacking. An assessment of M&E commissioned by the World Bank noted that the Health Sector Plan results framework was urgently in “need of critical review to ensure logical program management links and progression from inputs, to outputs to outcomes to impact (on health status)” (Nick Davis, 2013). While there were many useful outputs that were aligned with the priority areas of the HSP and that were driven by the SWAp modality, there is limited evidence that they led to improved outcomes, or to the achievement of the desired impact (Section 3.2). In retrospect, it may have been prudent to focus on fewer, regularly measureable indicators, and to emphasize the achievements of those outcomes over the course of the SWAp. 2.4 Safeguard and Fiduciary Compliance 29. Safeguards: At Project closing, the Overall Safeguard rating was Moderately Unsatisfactory. The only safeguard triggered by the Project was the Environmental Assessment (OP/BP/GP 4.01) as investments made through the SWAp were anticipated to increase health care waste. The implementation of a Health Care Waste Management (HCWM) Plan was included as a covenant in the Financing Agreement and as an intermediate outcome indicator on “improved medical waste management” in the Results Framework. A revised HCWM Plan was signed in 2013. In addition to IDA Safeguards Policies, the MOH undertook its own Environmental Impact Assessment which was submitted only after the SWAp extension was granted in 2013, and required revisions as it did not originally include a social impact assessment, focusing on physical and biological impacts only. This assessment focused on the impact of the three buildings at TTM Hospital, namely the PHC building, the Orthotic/Prosthetic workshop, and the Pharmaceutical

10

Warehouse. Regular supervision from the HCWM was provided over the course of the SWAp, but several issues remained unresolved by the end of the SWAp. First, there were insufficient attention to funding recurrent costs and maintenance to provide the needed inputs to improve HCWM. Funds were also not allocated for the replacement of incinerators due to depreciation. Second, segregation bins were found to be of insufficient quantity and quality while plastic bags were unavailable in sites visited. In general, staff continue to need further training surrounding the concept of proper waste segregation. While certain districts have implemented the recommendations surrounding HCWM, the ICR mission found that HCWM bins were accessible to the general population, unsecured, and mixed in terms of content, though the MOH has expressed commitment to resolving outstanding issues. 30. Fiduciary: The POW was frequently over-programmed and resulted in cost overruns despite activities being dropped. An Aide Memoire from November 2012 noted that several items had detailed designs and costings exceeding SWAp funds by US$9.4 million. This may have been reflective of a lack of knowledge surrounding the technical specifications necessary for an activity, resulting in an underestimate of costs (ISR #8, January 2013). At the same time, funds were not being disbursed as seen by high levels of unutilized funds. The 2013/2014 audit identified the purchase of faulty fixed assets using donor funds, though this was remedied by the MOH under the warranty for equipment. At the instigation of the DPs, a maintenance plan was developed for the sector, given the high investment in medical equipment and infrastructure made using SWAp funds. The Financial Management Implementation Review Report of November 2015 notes that it is unlikely that remaining works will be completed by June 2016, and while completion by September 2016 is likely, a more realistic expectation is a completion by December 2016. Financial covenants were complied with, and the Financial Management was considered Moderately Satisfactory at the time of Project closing. 31. Procurement: Procurement activities consumed the bulk of time in the early years of the SWAp, and encountered difficulties over the course of the Project. The possibility of procurement delays seriously undermining the SWAp was noted in the PAD (p. 9). The World Bank’s procurement procedures were used, and were consistent with government procedures. Despite the consistency, there was a perception that the use of two sets of procedures (GoS Tenders Board and WB), while preferable over the use of multiple donor-specific guidelines, resulted in delays for reviews and approvals, largely due to limited procurement capacity. Component heads were responsible for development of technical procurement documents, but lacked the capacity and experience to perform this function well. This weakness was seen in the inaccuracy of submitted proposals with regard to specifications and related cost estimates; professional review of specifications and solicitation of bids revealed that actual costs were substantially higher. As a result, activities meant to be funded under the SWAp had to be reduced, though it remains unclear as to how activities were subsequently dropped from the priority list. 32. As major infrastructure items (PHC Center, Malietoa Tanumafili II (MTII) Hospital renovations and Pharmaceutical Warehouse, and the Orthotics and Prosthetics Workshop) under the SWAp were coordinated by MOH and utilized by NHS, consultations surrounding design of the building included inputs from both agencies. Despite these consultations, there were reports of unsynchronized inputs from the two agencies, which subsequently resulted in delays and cost overruns. While major infrastructure items were approved as part of the POW of the first eighteen

11

months of the Project, at the time of Project closing (which included a two-year, no-cost extension) only the Orthotics and Prosthetics Workshop had been completed though the contract for the renovation (but not supervision of renovations) of the PHC Center had been signed. Procurement efforts are no longer being handled by an accredited specialist following the departure of the MOH Procurement Specialist in late 2014, though DFAT has been providing assistance with support from their Infrastructure Specialist. Renovation of the MTII Hospital Radiology Department remains incomplete and while the ICR mission was informed that tasks are on track and scheduled for completion by June 2016, it remains uncertain if completion by that date is possible. Further, there continues to be some concern regarding the quality of the infrastructure (see Annex 3). 2.5 Post-completion Operation/Next Phase 33. The World Bank exited the SWAp on December 18, 2015 and no follow-up operations in the health sector are planned at this time beyond technical assistance. The other DPs will continue to provide support to the MOH, and to oversee progress of the remaining infrastructure program which is currently planned for completion by June 2016 with the end warranty period in June 2017. Efforts such as the Health Program Advisory Committee (HPAC)11 are likely to continue going forward, but further fragmentation of the health sector (from two to three agencies) is a risk given plans for the establishment of the Health Promotion Foundation. Nonetheless, this risk may not materialize if a strong emphasis is placed on impact assessment and cost-effective interventions. The absence of robust monitoring and evaluation systems continues to remain a challenge, though WHO, ADB, and DFAT have expressed their commitment to provide support in the development of HIS. 34. Where infrastructure is concerned, the NHS has developed a Maintenance Plan but it is uncertain as to whether sufficient resources will be provided to maintenance efforts, particularly where preventive maintenance is concerned. From 2009/2010 to 2013/2014, approximately 0.24 percent (or SAT$500,000) of spending on infrastructure and equipment was spent on maintenance, half of which came from DPs (Philip Davies, 2013). While the 2014/2015 NHS budget allocation provides ~1 percent of its budget or SAT$746,200 for maintenance, a greater amount will need to be spent in this area going forward.

3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 35. Objectives: The SWAp focused on the Health Sector Program, which is aligned with the priorities and strategies articulated in the Health Sector Plan 2008-2018. The current Country Assistance Strategy (CAS, 2012-2016), which represents the first individual strategy for Samoa, emphasizes the importance of improving resilience going forward. The CAS notes that vital to efforts to improve resilience is strengthened planning services and improved public expenditure management, which are in line with a SWAp. Finally, the Government of Samoa’s 2012-2016 Strategy for the Development of Samoa (SDS) emphasizes strengthening economic resilience and encouraging inclusive growth through increasing investment in the productive sectors of the economy, and commits to continued investment in the social sectors, including in health as part of efforts to achieve a healthy Samoa.

11 Formerly known as the Health Program Steering Committee.

12

36. Design: The SWAp represented a key milestone for the GOS where the health sector was concerned. At the same time, other elements that may have contributed to the successful achievement of the Project were missing, such as an in-depth assessment of M&E capacity, and a clear, evidence-based, results chain. The absence of regularly updated epidemiological evidence made the identification of priority areas and successful interventions challenging, and consequently, the selection of activities under the POW may not have been as strategic and focused on the highest-impact interventions possible. Further, it is important to note that the use of an annually updated POW may not have been the most appropriate instrument for monitoring the sector and focusing on key policy issues as it lacked a holistic perspective and did not form part of the MTEF on a rolling basis as noted in the PAD (p. 44). 37. Implementation: The flexibility in Project design proved to be a strength when the country faced challenges such as the tsunami (2009), for which additional financing was provided. The SWAp has also been successful in getting different stakeholders across the health sector around the table as part of the implementation process, which is particularly important given the limited capacity of the MOH and NHS and the relatively heavy reliance on international development cooperation flows as seen in many small island states. While there was a shift in priorities that occurred following the redevelopment of the TTM Hospital, efforts continued to be made on health promotion and primordial prevention, which are important areas considering the burden of disease in the country. 38. In view of the information provided above, relevance is rated High for Objectives and Substantial for Design and Implementation. 3.2 Achievement of Project Development Objectives 39. The PDO was to “improve the Recipient's effectiveness in managing and implementing the Program based on the use of results from sector performance monitoring.” The complete list of indicators is provided in the Data Sheet while a list of activities and outputs completed under the project is provided in Annex 2; a partial description of indicators and activities is provided here to assess Project achievements against its PDO. 40. Achievement of the first sub-objective focused on improved effectiveness of management was considered Substantial largely based on Part B of the Results Framework (SWAp indicators). From the IOIs, it was inferred that aspects of management that the SWAp sought to improve was efficiency in (i) resource allocation (IOI #1 and IOI #2), use of human and physical resources; and (ii) governance (IOI #3-IOI #5), which were found to have demonstrated Modest and Substantial improvements over the course of the SWAp. (i) With regard to efficiency in resource allocation, the project improved alignment between

health sector finance and HSP priority areas to over 90 percent (KPI #1). Alignment with HSP priorities was also seen in Key Sector Partners’ corporate plans and government investments (IOI #2). HSP priority areas were those listed in the HSP, and were relatively broad. While health expenditure as a share of government expenditure increased over the course of the SWAp (IOI #5), it is unclear as to whether there was improved efficiency in government spending. A Public Expenditure Review (PER) focused on 2006-2012 notes that 80 percent of the increase in domestic resources in that period have gone toward increased personnel

13

spending. In particular, it emphasizes an increase in overtime payments and nursing staff was seen, though it appears the increase was largely among more experienced nurses despite the HSP noting gaps in lower levels (such as auxiliary nurses). It also notes increased funds for primary care and treatment services but a decline in resources for prevention. The PER further notes that increased spending on noncommunicable diseases (NCDs) is pushing out spending in other areas with greater benefit, namely primordial prevention. Implementation of policies and plans according to agreed priorities was seen through the development of an MTEF (IOI #1), but annual updates have not been performed since 2012. Also unclear is the extent to which funds were allocated in response to new information, if any, beyond that of large surveys or studies. However, responsiveness was seen with regard to natural disasters such as the tsunami and Cyclone Evan, where funds were provided to aid in the response thus ensuring minimal impact in terms of health consequences.

(ii) Greater efficiency in use of human and physical resources was measured over the course of the

SWAp based on the share of SWAp Program funds released according to agreed schedule (IOI #3) and the decline in reported drug stock-outs by facility (IOI #4). Some delays in disbursement of SWAp Program funds were observed, as well as cost overruns (see Section 3.3). While drug stock-outs do not appear to be a problem, stock-outs of reagents for laboratory testing appear to be a challenge. Despite these shortcomings, evidence (described below) suggest that overall, resources are being used more efficiently. In terms of human resources, the NHS completed a Human Resources for Health (HRH) Workforce Development Plan, highlighting gaps and needs for the coming years (e.g. mental health specialists, pathology, radiology), with some effort already made to address observed gaps (IOI #16). Through the SWAp, relevant staff were provided with opportunities to develop and retool their skills to support the country’s needs (see Annex 2), which allowed some tasks to be completed in-house at the MOH and NHS instead of being outsourced (e.g. graphic design). However, costs continue to be incurred due to a lack of General Practitioners (GPs) in the public sector, requiring contracting of private sector GPs to meet the needs of the public sector programs. Further, the PER notes from 2006-2012, the health sector has seen increases in average salaries (by 26 percent) and in employment (by 30 percent), raising questions about sustainability and affordability. Finally, there continues to be gaps in human resources for health in outlying areas. In terms of physical resources, the SWAp financed the purchase of biomedical equipment for the National Hospital,12 which was accompanied by a maintenance strategy developed by NHS and training of an in-country Biomedical Specialist, aimed at reducing the cost of overseas treatment, particularly for diagnosis. However, some weaknesses have been observed where the infrastructure investments of the SWAp are concerned, suggesting room for improvement where efficiency of physical resources is concerned (see Section 3.3). Going forward, commitment to improved efficiency has been demonstrated through the National Hospital in the reduced allocation for overseas treatment from 11 million Tala (2014/2015) to 4 million Tala (2.5 million Tala original allocation and 1.5 million Tala supplementary allocation)13 (2015/2016).

41. Improvements in governance was similarly seen in many areas:

12 DFAT financed USD$3 million dollars outside the SWAp for additional biomedical equipment. 13 The total budget allocation for the Health Sector in 2015/2016 is SAT$ 79.3 million.

14

(i) First, the SWAp improved multisectoral engagement where the achievement of health outcomes was concerned. Oversight of the SWAp was provided by the HPAC, which was chaired by the MOF, and included the private sector, nongovernmental organizations (NGOs), and development partners. The SWAp allowed for cross-sectoral collaboration which has been strengthened over the years with improved working relationships with the Ministries of Education (on nutrition and NCDs), Women, Social and Community Development (various aspects of gender-health and gender-financing), Agriculture (on improving nutrition and use of local healthy produce to promote wellbeing), Commerce and Trade (taxation on tobacco, sugary drinks), and the private sector (to promote health). In addition to the HPAC, the project saw more ownership from the Parliament through the Samoa Parliamentary Advocacy Group for Healthy Living (SPAGHL).

(ii) Where health sector governance was concerned, the relationship between MOH and NHS has

improved over the years with an increased awareness of the role each should play. There has been improvement in health sector management (IOI #8), though the NHS has been hampered by inadequate tools and inability to manage certain functions which remain under MOH decision making or management. The completion of the Demographic and Health Survey (DHS) (IOI #9) in 2009 and 2014 are major achievements made under the Project. In addition, the PER notes that while in 2006 almost the entire healthcare budget was classified as tertiary spending, this has changed over time as the government shifts to a community-level, primary-care based model though tertiary expenditure has been maintained in real terms and continues to constitute the largest component of health expenditure. A major weakness of the SWAp was the inability to develop an HIS system, further contributing to a lack of data where consistent sector performance monitoring, and relatedly, planning, are concerned. An evaluation in 2013 notes “no discernible use of data in planning or policy cycles” (Mark Spohr). These weaknesses have been described in Section 2.3. It is worth noting that there is some evidence that results of large studies (NHA, DHS, STEPS Noncommunicable Disease Risk Factor survey) were used in resource allocation and decision-making. For example, the STEPS 2013 survey highlighted the lack of progress on NCDs, thus providing an impetus for the implementation of WHO’s Package of Essential NCD interventions (PEN) which is currently being rolled out. Prior to that, the DHS 2009 highlighted the need to improve reproductive health, after which UNFPA was brought in to provide assistance. Recent increases in the birth rate (KPI #5) have also resulted in the involvement of MOF in the population agenda. While compliance among schools in areas such as nutrition and diet (which has shown improvement from 2012 to 2015) appear to have been closely monitored through the Annual School Monitoring Reports, evidence of the direct impact of these policies on the health of schoolchildren is limited. In general, monitoring efforts were irregular, and regular sector performance monitoring does not appear to have taken place over the course of the SWAp.

42. The second sub-objective focused on effectiveness of implementation was considered Modest. This reflected (i) how well the HSP was implemented based on the results of sector performance monitoring; and (ii) how well the HSP performed, which is particularly important given the second-tier PDO (as reflected in the PAD) and as reflected by Part A of the Results Framework (HSP indicators). Attribution to the SWAp where achievements of the HSP is concerned is challenging, as the Project did not exceed more than 14 percent of the GOS budget

15

over the period it was implemented, and a substantial investment was made outside the SWAp in the redevelopment of the TTM Hospital. Nonetheless, two key aspects are highlighted here: (i) Translation of plans and policies into action: The SWAp enabled the creation and the

institutionalization of the Policy Unit, which in turn developed a number of important policies and strategies including the National Food and Nutrition Policy, National Child and Adolescent Health Policy, NCD Policy, the Health Promotion Act 2014 and Food Act 2015. While the management-related outputs over the course of the SWAp were large and varied, there is limited evidence surrounding their utilization, particularly with regard to improvement in health outcomes. Assessment of management training funded under the SWAp revealed that 67 percent of MOH/NHS staff were not aware of policies in place, highlighting a clear need to improve dissemination and knowledge of these policies so that they can inform implementation of health service delivery. An exception was where tobacco-related policy was concerned, which included a clear results chain and resulted in improved outcomes (see KPI #8, IOI #17).

(ii) Achievements of HSP by priority area: Achievements of the HSP by priority area (described

as four “crucial areas of health challenges” and based on the Health Sector Situational Analysis conducted in May 2006) was mixed, with worsening outcomes particularly noticeable for NCDs. While the HSP lacked targets for many indicators, it is unlikely that worsening outcomes were desired. Achievements of the HSP based on a selected number of indicators are summarized below (Table 1).

Table 1. Achievements of the HSP HSP Area of Focus Status Control of NCDs KPI#3 Prevalence of diabetes Not achieved IOI#10 People aged 18 years and over overweight or obese Not achieved KPI#10 Percentage of people with high levels of physical activity Achieved Improved MCH KPI#4 Perinatal mortality rate Partially achieved IOI#11 Percentage of children under 1 year who received at least one dose of

measles vaccine Achieved IOI#15 Proportion of rheumatic heart disease patients complying with

treatment Achieved Universal access to reproductive health services KPI#5 Adolescent birth rate Not achieved IOI#14 Antenatal care coverage for at least one visit Achieved/ExceededControl of communicable diseases KPI#6 Incidence of water and foodborne infections Partially achieved Injury prevention and management KPI#7 Injuries in children (under 15 years old) Not achieved

43. Assessment of the drivers behind the lack of achievement of some areas of the HSP was difficult due to a lack of information. Primary care utilization by gender, age and domicile (IOI #13) remained unknown at the end of the Project, making it challenging to determine if the increase

16

in diabetes prevalence and overweight/obesity could be due to a change in the population accessing services. Similarly, data on the share of annual outpatient visits by the poorest quintile of population (KPI #2) is lacking. Evidence from the DHS from 2009 and 2014 provides comparable data, and proxy data suggests that improvements in accessibility were observed among the lower income quintiles (described in the Data Sheet under KPI #2), which may partly explain the increase in diabetes prevalence and overweight/obesity (KPI #3; IOI #10). This explanation is partial at best, and the picture that remains is one where health outcomes for several indicators worsened over time. 44. A summary of Efficacy ratings is provided in Table 2 below. Both parts of the PDO are weighed equally, and the overall rating obtained is Modest. Table 2. Summary of Efficacy Ratings14 PDO Rating Improved effectiveness of management - SubstantialEfficiency (i) Efficiency in resource allocation Substantial (ii) Efficiency in use of human and physical resources Modest Governance (iii) Multisectoral engagement High (iv) Health sector governance Modest Improved effectiveness of implementation - Modest (i) Translation of plans and policies into action Moderate (Substantial/Modest)(ii) Achievements of HSP by priority area Modest PDO Achievement Rating MODEST

3.3 Efficiency 45. The PAD does not include a full economic analysis of the Project, but instead highlights expected cost-effectiveness of interventions for preventing and treating diabetes and rheumatic fever. The PAD also highlights benefits associated with: (i) early diagnosis and treatment; (ii) productivity gains; (iii) health outcomes and quality of life; and (v) a reduction in risks and costs associated with disease outbreaks and other public health emergencies. The economic and financial analysis of the ICR (Annex 3) thus focuses on presenting the evidence surrounding these issues. With the exception of rheumatic heart disease, systematic improvements in early diagnosis and treatment were not observed over the course of the SWAp, in part due to the emergence of the TTM Hospital which skewed resources toward tertiary care. Where diabetes is concerned, evidence of productivity gains is weak, with increased prevalence and severity observed in recent years (Annex 3, Box 1). However, the efforts of the DFC at the Orthotics and Prosthetics Workshop has resulted in some productivity gains in a small share of the diabetes population. Health outcomes have demonstrated mixed results to date, and noncommunicable diseases remain an area of concern with limited evidence of early detection though there is some evidence of behavior change (Section 3.2). The response to natural disasters (tsunami and Cyclone Evan) can

14 Possible ratings are Negligible, Modest, Substantial, and High. An additional category of Moderate was added to represent achievement that was considered to fall between those two categories.

17

be said to be positive, as these disasters did not result in disease outbreaks despite widespread devastation. While a public health emergency was seen in the form of an H1N1 epidemic, this occurred early in the SWAp (2009). In addition, efficiency in resource allocation, and use of human and physical resources have been discussed previously, and were rated Substantial and Modest, respectively. 46. An assessment of outcomes in relation to cost is necessary in assessing Efficiency; these are detailed in Annex 3 and discussed briefly here. Major project expenditures were those of infrastructure and equipment, which saw procurement delays and exceeding cost estimates. Challenges in the form of cost overruns, delays and/or unsynchronized inputs, were observed where the Nurses Hostel, PHC Centre, Orthotics and Prosthetics Workshop, Pharmaceutical Warehouse, and renovations to the MTII Hospital in Savaii were concerned, with each major infrastructure investment associated with a distinct set of challenges. An example of challenges faced is provided in Annex 3, Box 2. Despite a two-year extension, there continues to be outstanding infrastructure investments. Based on the mixed results where health outcomes are concerned and the inefficiencies associated with the major investments, Efficiency is found to be Modest. 3.4 Justification of Overall Outcome Rating Rating: The overall rating of Moderately Unsatisfactory reflects the Substantial rating for Relevance and Modest ratings for Efficacy and Efficiency. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 47. A strength of the SWAp was its ability to leverage funding from other sources. The Orthotics and Prosthetics Workshop funded under the SWAp received further assistance from DFAT through Motivation Australia, and have managed to use their new location to also provide a Diabetic Foot Clinic (DFC). The Workshop has gone above and beyond in their role where DFC services are concerned, and have coordinated the provision of additional health services (such as eye appointments and dietician appointments) for diabetes patients. Further, the DFC have made concerted efforts to ensure appointments are made outside of school hours to ensure that schoolchildren (usually girls) do not miss school to accompany patients to their appointments. 48. The private sector has also become involved in efforts to improve health outcomes. An example of these efforts is the provision of exercise classes at several private companies in Samoa using guidelines provided by the GOS. The village community organizations have also participated in these efforts, with the total number of workplaces enrolled in the Healthy Workplace Program Monitoring at 55 and the total number of village community organizations at 213 as of 2012, the most recent year for which data is available. (b) Institutional Change/Strengthening 49. Several aspects of institutional strengthening were seen through the SWAp. The SWAp began soon after the MOH-NHS split took place, at a time when both entities were trying to come to terms with their new roles. The relationship between the two improved over the course of the SWAp, and evidence for the institutionalization of a sector-wide process was seen to have taken place in the form of improved dialogue and coordinated policy development efforts. These efforts

18

were facilitated through many avenues, such as through the HPAC, through the administration of surveys where collaboration with the Bureau of Statistics was seen, the occurrence of regular meetings between the NHS and MOH on NCDs and the collaboration of NHS and MOH on Village Health Fairs. In addition, the SWAp has succeeded in bringing together the various stakeholders in the health sector around the table, ranging from the private sector to civil society organizations. The value of these common platforms is recognized by the GOS, and it appears that they are committed to maintaining these notable accomplishments going forward. 50. The SWAp also strengthened the management of the sector through the development of many policies, protocols and frameworks and through capacity building (see Annex 2 and Section 3.2). Training was provided for 2125 people (IOI #19), and newly-acquired skills were utilized in many areas, such as where development of training materials for health promotion were concerned. These training activities also benefited other sectors, such as the Memorandum of Understanding between MOH and Ministry of Agriculture and Forestry (MAF) for the sharing of knowledge and skills in health, nutrition and food production activities. (c) Other Unintended Outcomes and Impacts (positive or negative) N/A 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops N/A

4. Assessment of Risk to Development Outcome Rating: Moderate 51. The risk to development outcome is considered Moderate. The Project strengthened overall capacity and resulted in improved multisectoral engagement and health sector coordination, but these achievements translated to limited improvements in health outcomes during the life of the SWAp. Further, it is uncertain as to how these improvements will take place over time. While the results of large scale surveys such as the DHS and STEPS and tools such as the MTEF and National Health Accounts (NHA) have been utilized in decision-making and resource allocation, the use of results from sector performance monitoring continue to be lacking. Finally, while the relationship between MOH and NHS has improved since the beginning of the SWAp, there continues to be fragmentation within the sector, and there is a risk of further fragmentation in the near future with the setup of the Health Promotion Foundation. The Bank has exited the SWAp and will remain engaged in a limited capacity through the provision of technical assistance. WHO, ADB, and DFAT will contribute to the improvement of HIS going forward, and other development partners (notably MFAT and DFAT) remain committed to overseeing completion of infrastructure investments made under the SWAp, and to improved health outcomes in the country.

5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 52. Quality at entry was adversely affected by weaknesses in M&E and in the results chain. SWAp consultations took place soon after the health sector reform that saw the split between MOH and NHS. The SWAp modality represented a new way of engagement for the country, and was

19

seen as an improvement over the previous Project approach. The preceding Project, the Samoa Health Sector Project (P064926) had closed in 2006 rated Moderately Satisfactory, and as part of implementation arrangements had provided in-country support for 1-2 years. To assist with implementation of a new modality, the PAD provided substantial detail regarding implementation arrangements (see Annex 13), and clearly stated the goals of the annual meetings (Health Summit and Joint Review) and the role of the Coordinating Development Partner. Similar clarity was provided in terms of procurement and financial management, though such clarity was lacking for M&E. Indeed, M&E design contained several weaknesses as described in Section 2.3. A results focus was also absent – a coherent results chain, appropriate indicators with accompanying definitions15, baseline, and targets were missing. While the areas of focus of the SWAp were relevant to the needs of the country, the background analysis conducted as part of Project preparation could also have been strengthened, as could several other aspects of Quality at Entry (Section 2.1). In summary, Bank performance in ensuring quality at entry was Moderately Unsatisfactory. (b) Quality of Supervision Rating: Moderately Unsatisfactory 53. Despite shortcomings in quality at entry, the Bank team provided close supervision over the course of the SWAp, though the decision to conduct a redevelopment in place of the MTR represented a major lost opportunity. The Task Team conducted monthly videoconferences and weekly audioconferences with the SWAp Coordination Unit (SCU) and relevant partners at the beginning of the Project. In response to slow progress at the beginning of the SWAp, an agreement was made for increased supervision, as detailed in the Aide Memoire from December 2011. This additional supervision came in the form of two operational support missions (in addition to the two/year already scheduled), and the provision of in-country presence for several months. There does not appear to have been discussions on the skill-mix needed (in terms of supervision or in-country presence) or the specific technical roles of the DPs, which, in retrospect, may have been particularly important as this was not a traditional SWAp (Section 2.1). There were a total of four Task Team Leaders over the course of the SWAp, and changes in staffing among the DPs were also seen. While there was always a harmonious relationship between the DPs, there were weaknesses in record-keeping. When coupled with the frequent changes in staffing, this may have meant that monitoring of the results framework was not done as closely as it could have been, and along with it, the opportunity to focus on outcomes instead of outputs was lost. 54. A decision was made to conduct a redevelopment16 of the SWAp in place of an MTR to readjust the SWAp given challenges faced related to implementation, with delays in procurement serving as the major issue. The redevelopment confirmed that the development objectives, performance indicators and implementation arrangements of the Program would remain unchanged. However, it resulted in the strengthening of the HPAC, expansion of the SCU to become a Sector Coordination Unit (renamed as the Health Sector Coordination, Resourcing, and Monitoring Division), modification of the PoW templates, and changes to procurement thresholds

15 The Final Aide Memoire (November 2015) notes that no additional support from the SWAp was provided to develop a data definition dictionary to provide clarification on indicators being collected for SWAp review. 16 Further details regarding the redevelopment are available in Annex III of the Follow-up Letter to the November 2010 Health Redevelopment Mission.

20

which addressed some challenges that the SWAp had encountered. Issues associated with quality at entry, such as the weaknesses in M&E and the weak results chain were not addressed. The decision to conduct a redevelopment in place of an MTR thus represented a further lost opportunity, as the MTR would likely have confronted the weaknesses to date and allowed for a restructuring to take place if needed. The Bank’s systems were used for monitoring of fiduciary and safeguard policies, and these were followed closely. In addition, close supervision and proactive management allowed for timely response through the SWAp following the tsunami (2009) and Cyclone Evan (2012). In view of the achievements and lost opportunities described above, the quality of supervision provided was considered Moderately Unsatisfactory. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory. This is based on ratings of Moderately Unsatisfactory for Quality at Entry, Quality of Supervision, and Overall Outcome. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 55. The SWAP saw strong country leadership and commitment as evidenced by chairing of the Health Summit and Joint Review meetings by the MOF ACEO for the Aid Coordination and Debt Management Division. The GOS provided all planned contributions to the financing envelope despite the occurrence of natural disasters. Evidence, where available, was taken into consideration when providing input into the design of the SWAp. In particular, the NHA 1999 had highlighted the lack of investment in health promotion and primordial prevention while the STEPS 2002 had highlighted the increasing burden of NCDs; these areas were subsequently identified as priority areas in the SWAp. When challenges in meeting deadlines occurred, the Prime Minister got personally involved in order to mandate action from MOH and NHS in order for SWAp deadlines to be met and outstanding issues to be resolved. In general, government performance was found to be Satisfactory. However, taking the rating of implementing agencies performance into consideration (as they constitute part of the GOS), government performance is rated Moderately Satisfactory. (b) Implementing Agency or Agencies Performance Rating: Moderately Unsatisfactory 56. The SWAp had two implementing agencies, namely the MOH and NHS. While the MOH was tasked with overall coordination and oversight of the Project and Program implementation, NHS was responsible for the implementation of Component 2. As previously noted, the MOH Act 2016 resulted in a split in functions between the MOH (policy, regulatory, oversight) and NHS (service delivery) immediately prior to the Project, which resulted in tensions, and ambiguity regarding roles and responsibilities at the beginning of the SWAp. MOH managed all financing under the SWAp, including financing allocated to the NHS for Component 2. The relationship between the two entities has improved over the course of the SWAp, though there continues to be room for further improvement and synchronization. One area of weakness was procurement, as detailed in Section 2.4. Challenges in procurement impacted efficiency of the Project, as described in Section 3.3. The impact of the SWAp on health outcomes remains mixed at project closing, and it is uncertain to what extent the Project allocated resources to highest-priority, highest-impact interventions or whether a specific mix of activities was selected in response to the epidemiological

21

burden of the country. Given weaknesses in procurement and efficiency, and the mixed health outcomes observed at closing, performance of the implementing agencies was found to be Moderately Unsatisfactory. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory. This is based on ratings of Moderately Satisfactory for Government Performance, Moderately Unsatisfactory for Implementing Agencies Performance, and Moderately Unsatisfactory for Overall Outcome.

6. Lessons Learned 57. The Project may have benefited from the use of incentives linked to achievement of outcomes in the form of results-based financing. Disbursement-linked indicators (DLIs) could potentially have been a useful approach in the context of this SWAp. The use of DLIs may also have incentivized improvements in M&E and HIS as progress would need to have been documented in order to access funds. DLIs would need to be developed appropriately given the presence of substantial non-SWAp funds in the health sector. 58. The use of a clear results chain linking activities to outcomes, a theory of change, or the MTEF may have allowed the GOS and DPs to be more selective about the activities funded under the SWAp and emphasized the importance of M&E where individual activities were concerned. Such an approach would also have driven a clear prioritization process for the POW. As part of efforts linking activities to outcomes, templates could also have been developed early on emphasizing measurement of impact of proposed activities and identification of the link from activity to outcome. 59. A phased approach such as an Adjustment Program Loan (APL) may have also been beneficial, where the first phase focused on the development of key policies, programs and systems, and the second phase sought to improve the utilization of these systems for the improvement of health outcomes. Such an instrument would have narrowed the focus of the Project and placed greater emphasis on achievement of results. An example of an APL-type operation is the Tanzania’s Health Sector Development Program Project (Phase I) and Second Health Sector Development Project (Phase II), which also employed a SWAp and aimed to support the Health Sector Development Program over the period 2000-2011. 60. Recurrent problems should be investigated in greater detail. Infrastructure activities faced many challenges, and a closer look may have identified the source of these problems early on, allowing them to be rectified as soon as possible and preventing recurrence of similar problems down the line. An alternative approach would have been to include a clear escalation pathway for resolution of issues during the SWAp. 61. The lack of an MTR represented a lost opportunity for candid assessment of the SWAp and to course-correct. It could also have provided the necessary evidence for the redevelopment of the SWAp, ensuring that the redevelopment exercise represented a comprehensive response to problems identified to date and possible restructuring if needed. Weaknesses in the quality of record-keeping among DPs and gaps in data where M&E efforts were concerned which occurred

22

despite a focus on M&E within Component 3 may have been noticed then and a more rigorous approach to data documentation and collection could have taken place.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies Comments from the Borrower/implementing agencies are included in Annex 7. (b) Cofinanciers Comments provided by the cofinanciers (DFAT and MFAT) are included in Annex 8. (c) Other partners and stakeholders No comments were received from other partners and stakeholders.

23

Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$ Million equivalent)

Components Appraisal – Confirmed

Cost Estimate17 (US$ millions)

Appraisal – Likely Cost Estimate (US$ millions)

Latest Cost Estimate (Dec 31, 2015) (US$ millions)

Component 1 2.15 Not Specified 2.50 Component 2 7.35 Not Specified 21.44 Component 3 1.90 Not Specified 7.48

Total Cost 11.4018 24.30 31.42 (b) Financing

Source of Funds Indicative Financing (US$)19

Committed Funds (US$)20

Amount Disbursed

(US$)20

Percentage Disbursed

Borrower (GOS) 1,500,000 1,828,156 1,828,156 100%World Bank 3,000,000 5,869,648 (1) 5,869,648 100%DFAT 7,200,000 15,600,000 15,600,000 100%MFAT 12,600,000 11,768,012 (2) 11,768,012 100%Interest Received 468,045 Total 24,300,000 35,065,816 35,533,861

(1) Based on the November 2015 Aide Memoire. Total World Bank credit funds allocated was SDR 3.9 million (US$6.0 million equivalent). Table shows disbursements/commitments in US$ equivalent (November 2015) for Credit 44320 of US$2,912,494, and Credit 47210 of US$2,957,154.

(2) MFAT funds allocated is NZD16.0 million, table shows US$ equivalent.

17 This amount included commitments from MFAT and DFAT for the first 2 years of Program implementation only (2008-2010). In addition, both MFAT and DFAT expressed that subject to approval they were likely to contribute additional funds for the subsequent 3 years (2010-2013) (MFAT - US$4.8 m equivalent and DFAT US$8.10 m equivalent). With the additional funding, the resource envelope was expected to be US$24.3 m (at current exchange rates) at the time of appraisal. 18 This includes taxes and duties estimated at US$0.8 million. 19 This amount as shown in the PAD. 20 This includes additional funding of US$3 million paid directly by DFAT for medical equipment.

24

Annex 2. Detailed Project Components and Outputs by HSP Outcome The Program Components21 and Outputs as listed in the PAD were as follows: Component 1: Health Promotion and Prevention Planned outputs under Component 1 were as follows:

Subcomponent (i) development and implementation of health promotion strategies including, inter alia, provision of Sub-Grants to support the implementation of health promotion activities consistent with priorities set out in the Program

Subcomponent (ii) supporting the prevention of non-communicable diseases through behavior change promotion and screening

Subcomponent (iii) supporting activities aimed at reducing anemia among children Subcomponent (iv) strengthening infectious disease surveillance, screening and

prophylaxis for post-streptococcal rheumatic heart disease and reducing the prevalence of vector-borne diseases such as dengue and filariasis

Subcomponent (v) implementation of sexual and reproductive health interventions Subcomponent (vi) carrying out an injury prevention program Subcomponent (vii) implementation of the Health Care Waste Management Plan and the

handling of bio-safety hazards. Component 2: Enhancement of Quality Health Care Service Delivery Planned outputs under Component 2 were as follows:

Subcomponent (i) dissemination and training of staff on clinical standards and protocols for disease management adapted to the Samoan context

Subcomponent (ii) upgrading the skills of health workers Subcomponent (iii) supporting improvements to reproductive, maternal and child health

services Subcomponent (iv) improvement in primary health care and community health outreach

through integrated health services Subcomponent (v) improvement in the forecasting of pharmaceutical requirements,

procurement, storage, distribution and use through adoption of an upgraded warehouse and inventory management system

Subcomponent (vi) improvement in physical disability services through activities and investments aimed at improving the quality of life of people with physical disabilities

Subcomponent (vii) supporting improvements to oral health and dentistry Subcomponent (viii) supporting improvements to blood safety collection and storage Subcomponent (ix) improvement in mental health services through

refurbishing/upgrading the mental health block of the hospital to ensure safety and security of mental health patients and staff [dropped given priority placed on the Orthotics and Prosthetics Workshop]

Subcomponent (x) carrying out of small infrastructural improvements and maintenance, and provision of medical equipment to health care facilities.

21 Indicative resource allocation for each component is provided in Annex 5 of the PAD and was subject to review at the Annual Health Summits.

25

Component 3: Strengthening Policy, Monitoring and Regulatory Oversight of the Health System The outputs were as follows:

Subcomponent (i) implementing a human resources development strategy for the health sector

Subcomponent (ii) supporting the monitoring and evaluation of the implementation of the Program and the sector at large through use of improved information systems

Subcomponent (iii) supporting the implementation of a public-private partnership framework for service delivery

Subcomponent (iv) establishing standards for quality assurance of pharmaceuticals and enhancement of the quality assurance and regulatory function for the sector

Subcomponent (v) strengthening the capacity of the SWAp Coordination Unit to coordinate and carry out its Program administration and monitoring responsibilities.

26

The SWAp financed the following activities (activities in bold were those financed as part of the post-tsunami response): Component 1: Health Promotion & Prevention

National Health Promotion Program Improve Typhoid Investigation National Campaign to Promote Fruit & Vegetables Reduce Prevalence of Dengue and Filariasis Cases Anemia Reduction in 0-5 Age Group Reducing Tobacco Smoking Prevalence Cervical (Pap Smear) Screening Program Strengthening Secondary Prevention - NKFS Strengthen Comprehensive Screening program Injury Prevention Program Population Risk Screening for Diabetes Infant and young child Feeding Program Health Care Waste Management Post Tsunami Anti-Alcohol Campaign

Component 2: Enhance Quality Health Care Service Delivery

Health Sector Warehouse for pharmaceutical & medical supplies Primary Health Care Services delivery for Apia Urban area Physical Disability Services improvement EPI catch up Campaign Improve Diagnosis and Management of Pediatric Cardiac Disease Imaging Equipment Refurbishment Program Development of BA Science Program Improving transportation systems for outreach Improve Mental Health Unit Service Delivery National Blood Transfusion Service Program Improve Health Care Equipment for Patient Care Develop capacity to implement evidence based primary prevention program in Oral

Health Extension of NKFS Dialysis Unit MTII Hospital Improvement of Infrastructure H1N1 Ventilators/supplies Procurement of Medical Equipment and Supplies for Health Sector Strengthening Secondary Prevention - NKFS Pandemic Disaster Reflection and ICHS Leadership Workshops

Component 3: Strengthen MOH Policy, Monitoring and Regulatory

Strengthening of ICT System Development of Protocols for health care services Human Resources for Health Sector Database

27

Credentialing System for Nurses & Midwives Annual Analysis of Health Sector Situation Responsive Human Resource Strategy for nurses & midwives Revenue Generation and Health Financing Standards for Quality Assurance of pharmaceutical supplies Enhancement of Quality Assurance Regulatory Function SWAp Unit Operational Costs Project Management Training Extension of Diabetic Clinic WHO Regional Committee Meeting MTEF/Governance Training

28

Final outputs are organized by HSP Building Block following Mid-term Redevelopment, and were as follows:

1. Governance and Leadership Strengthening Health Promotion Council Activities led by SPAGHL

TA to develop Regulations for MOH Act 2006

TA to develop Regulations for Health Professional Council

TA to review the Health Sector Plan

TA for Institutional Framework to enhance QA and monitoring

TA to undertake Marketing Strategy to enforce the WHO Code of Marketing breast milk

TA to develop the National Tobacco Control Policy

TA to develop the General Health Prevention Policy

TA to develop the Health Promotion Policy

TA to develop the Non-Communicable Disease Policy

TA to develop Priority Strategies for Anaemia

TA to review Primary Health Care Strategies

TA to undertake Food Safety and Food Preparation Audit

TA to conduct Cleanliness Audit of Toilets for WinLA Organisations

TA to develop the Health Sector History based on the Six Building Blocks of the Health System

Health Promoting School Standards developed and published

TA to review Food and Nutrition Policy

Develop standards and protocols for the Prevention and Control of Dengue and Filarisis

TA to conduct Food Premises Cleanliness Audit

TA to Develop the Cervical Screening Program – Policies, Protocols & Guidelines

Sanitation Standards developed and implemented

TA to develop Standards & Protocols for Vector Control Programs (Post Tsunami)

Support HPF Legislation – support Scoping Visit to VicHealth and consultations

Health Promoting School Standards published, stakeholder trainings conducted etc.

Annual Health Forums for two years

Exhibition for Health Sector Archives and exhibition implemented

Advocacy and Communal Engagement (50th Anniversary)

TA to improve Asset Management at NHS and to develop NHS Assets and Facility Management Plan

2. Health Service Delivery Support for the establishment of the Health Promotion Foundation

29

Supporting the HPF Manager position (ACEO level) which has already been established and implementing its core functions

Tobacco Control and Smoking Prevention Programs

Billboards (140) for tobacco control prevention program produced and installed

Pilot RHD screening: support for TA to undertaken screening and procurement of mobile RHD van

Improve collection and segregation of health care wastes – HWC

Outsourcing of Health Care Professional Services (support for the hiring of the consultant specialist, 3 GPs and a nurse specialist)

Monitoring of Water Treatment System for affected villages (Post Tsunami)

Reducing tobacco smoking prevalence amongst school children

Anaemia Reduction in the 0-5 age group national program – health sector

Reduce prevalence of Dengue and Filarisis cases in the health sector.

Improve Typhoid investigation program for sector partners

National Campaign to Promote Fruit and Vegetables for community based organizations

Injury Prevention Program with National Campaign, evaluation plan with sectors

Strengthening Secondary Prevention Program – NKFS and clinical partners as well as strengthening the comprehensive health screening program for health sector

Increasing child survival through infant and young child feeding – sector

Improvement of health care waste management (HCWM)

EPI Catch up Campaign – Savaii and health sector

Improve Diagnosis and management of Pediatric Cardiac Disease for NHS and Health Sector Partners

Post Tsunami – H1N1 Vaccination Campaign

Firm to design and supervise Construction of MOH Nurses Hostel and Renovations to the MOH Centre for Credentialing of Nurses, Midwifery and Allied Health Professionals

Firm to design and supervise the construction of the Primary Health Care Centre, Orthotics and Prosthetics Workshop, Pharmaceutical Warehouse and renovations to the MTII Hospital in Savaii

3. Human Resources for Health (HRH) / Human Workforce

Health Leadership and Management Development Program implemented for the health sector

Improving Nursing & Midwifery Workforce for the Revitalisation of PHC within and / or reflective of total health care context of Samoa. – UTS Contract.

TA to improve procurement processes at NHS and staff trained

IHR training for Health Quarantine staff

Workshops on clinical procedures conducted

HCW Generation Source established (surveillance) Training of Database

30

Undertake workshop on selected clinical procedures

TA to conduct ECG Clinical Training

Undertake workshop on selected clinical procedures ECG Clinical Training

Two weeks workshop for Improved Best Practice in Neonatal

TA to train Nursing Auxiliary

TA to undertake staff training on Clinical procedures ie ECG Operations, Reading and interpretation

Support and facilitate NHS workforce planning

CISCO local training for ICT staff

TA to undertake Renal Training for NKFS Nurses

NHS SWAp Manager for NHS Comp 2.

Component 1 & 3 Assistants (Principal Level)

National Procurement Specialist

Additional professional developments as listed in the training matrix

4. Health Information System

TA for HIS Policy & Planning (Phase 1)

Develop and enhance Health Sector HIS Surveillance

IT Network Connection to Nursing School Building

Geographical Information System / Geographical Positioning System (GIS/GPS)

Demographic and Health Survey 2009

STEPS Survey including mental health sub-survey

5. Health Financing TA to develop and review the Midterm Expenditure Framework

6. Medical Products and Technology

Mobile Clinic for RHD for screening. Portable mobile X-Ray units for Imaging Division X-Ray Mobile vans to be used at RDHs and in emergency situations Portable echo machine (Post Tsunami) Procurement of mammography machine for early detection of breast diseases and related

trainings. Portable echo cardiogram for RHD screening (Post tsunami). Static Clinics for Dental Units at the 3 Primary Schools: St Mary’s, Marist and Apia

constructed. Forklift for warehouse and Orthotics Buildings procured Improve pharmaceutical stock control storage (cooler and bar coding system) Design and supervision of Nursing Hostel and simulation laboratory / renovation of nursing

school building (T&G)

31

Construction of Nursing hostel and simulation laboratory / renovation of nursing school building

Procurement of Bio-medical equipment Consultations and operations related to Biomedical Equipment support was provided Mobility tools for physical disability unit Design and Supervision of the HNS PHC, Warehouse and Orthotic Buildings Construction of NHS, PHC Warehouse and Orthotic buildings Improve Pharmaceutical Stock control and storage (cooler and bar coding system) Design and supervision of MTII Hospital Construction of MTII Hospital facilities Hospital equipment procured for MTII Hospital and other RDH (Rural District Hospitals)

in Savaii Additional medical equipment for NHS New Hospital – Phase A Vehicle procured to improve emergency and relief management services for NHS Emergency standby generator for emergency support to MTII hospital purchased Operational Costs for the National Dental Epidemiology Survey

Vector Control Equipment: Antibody tests/stool jar for affected villages (Post Tsunami)

Food Safety Equipment for Environmental Health Inspection

Public Health Laboratory Equipment for Food and Water

Health Sector Warehouse for all Pharmaceutical Products

Establishment of National Blood Transfusion Service and implementation – NHS and Samoa Red Cross – this activity is half completed, supply of refrigerators for storage and promotion of outreach program to encourage blood donors conducted.

32

Annex 3. Economic and Financial Analysis 1. The PAD does not include a full economic analysis of the Project, but instead highlights the expected cost-effectiveness of interventions for preventing and treating diabetes and rheumatic fever. The PAD further emphasizes that the Project would result in several achievements as described in the Table below, though a lack of data prevents assessment of some areas.

Expected Result Actual Result 1. Earlier detection and

response to illness and reduced reliance on most costly, tertiary care services at late stage of illness

Evidence is lacking that early detection and response efforts have systematically increased over the course of the SWAp. A major reason for this was the emergence of the teaching hospital (TTM Hospital) which skewed resources toward tertiary care. While it is possible that it will allow for increased in-country treatment going forward, and relatedly, cost savings, evidence is unavailable at this time for a comprehensive assessment of cost-savings, and the limited evidence to date suggests increased costs in general, with lost opportunities for improved cost-effectiveness where diabetes is concerned (see Box 1). A notable exception is rheumatic heart disease (RHD), where increased screening and treatment efforts have been demonstrated over the life of the SWAp, and translated into reductions in incidence of acute rheumatic fever from 29 per 100,000 in 2011 to 7 per 100,000 by 2015 while compliance to treatment (IM Penicillin) among those with RHD has been maintained at 80% or higher since 2008. Other issues include unavailable results of analysis from specimens collected (e.g. the Men’s Health Fair), and a lack of availability of reagents for testing (e.g. for chlamydia) though syndromic management is conducted.

2. Improved customer satisfaction

Baseline information is not available for a comparison of trends over time. MOH reports that the proportion of clients satisfied with health services was 60% in 2012.

3. Resource savings (for households and government) due to improved resource allocation, gains in internal efficiency

Resource allocation was assessed under PDO Sub-objective #1, and was found to be Substantial. Improved health sector governance was assessed under PDO Sub-objective #2 and was noted to be Modest. It is unclear that there has been a reduced reliance on tertiary care services, suggesting a potential lost opportunity for resource savings. Further, new infrastructure investments for tertiary care services are expected to incur high recurrent costs while there continues to be a need to maintain district health services.

4. Reduction in risks and costs associated with disease outbreaks and public health emergencies

No disease outbreaks were observed in the period following the tsunami (2009) and Cyclone Evan (2012). An H1N1 outbreak occurred in 2009 but this was in the early stages of the SWAp. While immunization rates have generally improved, STIs

33

continue to remain an area of concern with chlamydia rates unchanged at 28% (NHS Lab, 2009 and 2015). Immunization rates (IOI #11) and typhoid rates (KPI #6) have demonstrated a decline over the course of the SWAp, though not much improvement has been seen in treatment of diarrhea (KPI #6). No new cases of HIV have been reported since 2009.

5. Reduction in time lost due to illness and improved productivity

Evidence on this front has been mixed, as illustrated using evidence from diabetes (see Box 1).

6. Improved health outcomes and quality of life

Health outcomes have demonstrated mixed results to date, and noncommunicable disease remain an area of concern with limited evidence of early detection though there is some evidence of behavior change (see Section 3.2).

34

Box 1. Diabetes Management in Samoa Data from the STEPS Survey suggests a large increase in diabetes in Samoa, from a prevalence of 21.5% (2002) to 45.8% (2013) among adults between 25 and 64 years old. Evidence from the National Kidney Foundation of Samoa (NKFS) suggests a substantial increase in late-stage treatment in recent years. The local patient population increased from 28 (2008) to 116 (2015) and the Dialysis Registry has increased from 50 (2010) to 495 (2015).22 The dropout rate is high at around 60% (2015), with only around 10% of those registered for treatment attending regular consultations for diabetes management. It is worth noting that the NKFS does not have a nutritionist on-site and thus consultations provided may be limited in scope. In addition to the increase in prevalence, the average age of the patient population is reportedly declining, suggesting additional costs and lost productivity. As of 2010/2011, the cost of dialysis was estimated at US$38,686 per patient per year (The Economic Costs of Noncommunicable Diseases in the Pacific Islands, World Bank, 2012). The mortality rate is high, with approximately 25% of patients dying in the most recent financial year. Further, while some funding for the NKFS is provided every year in the budget, it is unclear that spending is efficient at the cost of $38,686 per patient per year and a 25% mortality rate. In contrast, the Orthotics and Prosthetics Workshop (construction of which was funded by the SWAp) provides a diabetes foot clinic (DFC) through funding received by Motivation Australia. The Orthotics and Prosthetics Workshop staff have taken on a role as a one-stop center for their 42 clients to date (June 18 to December 1, 2015), and have made efforts to coordinate their visits to various specialists. Their dropout rate is less than 5% with an average of six follow-up appointments provided after the initial assessment, and evidence suggests approximately 80% of patients have been able to resume their normal duties upon receiving treatment.23 Staff who received training abroad in Orthotics and Prosthetics (funded by Motivation Australia, an NGO) have been in place since January 2016, but have not been able to utilize their newly-acquired skills due to a lack of equipment. Construction of the Workshop was completed in April 2015 (with the builders warranty set to expire in April 2016), but continues to lack Orthotic and Prosthetic-making equipment, which was originally supposed to be funded by the SWAp though this was canceled. Instead, funding for equipment will now be provided by Motivation Australia with substantial delay. Thus, while the services rendered in terms of DFC are cost-efficient, the lack of equipment has not allowed for maximum cost-efficiency of the workshop. Funding for the DFC operated through the Workshop is not provided through the annual budget despite evidence of cost-efficiency.

2. An assessment of the outcomes described above in relation to cost is necessary in assessing Efficiency. Major project expenditures were those of infrastructure and equipment, which saw substantial delays and frequent overruns of cost estimates. Despite a two-year extension for completion of infrastructure, it had yet to be completed by December 2015. The estimates for

22 2008 figures obtained from interviews conducted during the ICR mission. 23 Figures for the drop-out rate and resumption of normal duties obtained from interviews conducted during the ICR mission.

35

completion of infrastructure activities were underestimated in the POW, resulting in a major shortfall in October 2012. A comparison of estimated and actual costs shows that the resulting budget shortfall totaling SAT $15.5 million was due to quite substantial cost overruns, with the largest overrun of SAT $5.8 million for construction of PHC, Pharmaceutical Warehouse, and Orthotics Center. In the case of nonmedical equipment for the new hospital (TTM Hospital), no funds were originally allocated for this purpose, and an additional request of SAT $3.8 million was made (Aide Memoire, November 2012). 3. Each of the major procurement items, namely the Nurses Hostel, PHC Centre, Orthotics and Prosthetics Workshop, Pharmaceutical Warehouse, renovations to the MTII Hospital in Savaii, and the purchase of equipment for the new TTM Hospital experienced challenges. An illustrative example of challenges faced using the MTII Hospital is provided in Box 2 below. The experience described in Box 2 was not an isolated incident, and in addition to increased costs, likely resulted in lost opportunities. Box 2. MTII Hospital Renovations Improvements to the MTII Hospital were initially anticipated to be SAT$1.5 million, with actual costs originally estimated to be closer to SAT$4 million (Aide Memoire, November 2012). This discrepancy was likely the result of costing conducted by technical experts instead of those more familiar with construction, which appears to have occurred in other instances of infrastructure development over the course of the SWAp. The ICR mission found that there was a lack of clarity regarding the budget as consultations with MOH and NHS took place. MFAT confirmed their willingness to provide additional funds for the renovation and extension of the Hospital. Final costs at the end of the SWAp were SAT $5.8 million (MOH Accountant), reflecting further overruns from the originally estimated amount of SAT $4 million. The Hospital was originally supposed to house an X-ray machine from TTM Hospital in Apia. However, that machine broke down in March 2014. Several months later (in late 2014), the design works for the Hospital was approved and signed. A new machine was ordered by NHS soon after the machine broke down, though the MOH was only advised of this new machine in May 2015, at which point the construction work was halfway through. It is unclear when the MOH and architect were informed of the change in equipment size from the old machine to the new machine. Subsequently, the NHS requested use of the CT Scan room to house the machine while the X-ray room was being extended to accommodate the size of the new machine, which was much larger than the old machine. This necessitated a redesign, and translated to some variation to the original construction costs which was not approved by the Tenders Board. The NHS then requested the long-term (instead of temporary) use of the CT Scan room as it could accommodate the X-ray machine, but this request was denied on the grounds that the room was needed for the CT Scan machine. The CT Scan machine subsequently broke down. Following that (and approximately one week before completion of construction), the NHS requested reconfiguration to the original design, but this could not be done as the redesign had already started. The renovated building was handed over to the MOH in December 2015. At the time of the ICR mission and more than two years after the original end date for the SWAp, both the X-ray and

36

CT Scan spaces remain closed off and unutilized. While the new X-ray machine arrived some time ago, it remains unused. The mobile X-ray machine continues to be used in place of the new machine.

4. Financial sustainability also requires consideration of recurrent costs. These have been

described in Section 2.5 of the ICR. While financially sustainable with ~1% of the NHS budget for 2014/2015 going toward maintenance costs, it is thought that a greater amount will be necessary going forward, particularly given the additional infrastructure investments made in recent years.

37

Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members

Names Title Unit Responsibility/

Specialty Lending Muhammad Ali Pate Senior Public Health Specialist EASHD Team Leader Aparnaa Somanathan Senior Economist GHNDR Team Member

A. Juliana Williams Senior Program Assistant EASHD Senior Program Assistant

Reem Hafez Consultant GHNDR Team Member Betty Hanan Consultant GSP06 Team Member Ian P. Morris Consultant GHN02 Team Member Melinda Good Chief Counsel LEGES Legal Sheila Braka Musiime Counsel LEG Legal

David Michael Chandler Sr. Financial Management Specialist EAPDE Financial Management

Stephen Paul Hartung Financial Management Specialist GGODR Financial Management

Cristiano Costa e Silva Nunes Senior Procurement Specialist GGODR Procurement Evelyne Villatoro Senior Procurement Specialist GGODR Procurement

Supervision/ICR Eileen Brainne Sullivan Senior Operations Officer GHN02 Team Leader Eva Jarawan Lead Health Specialist EASHH Team Member Aparnaa Somanathan Senior Economist GHNDR Team Member Neesha Harnam Health Specialist GHN04 Team Member Nicholas Davis M&E Specialist EASHD M&E

A. Juliana Williams Senior Program Assistant EASHD Senior Program Assistant

Sabrina Terry Program Assistant GHN02 Program Assistant Cornelio Quintao De Carvalho Team Assistant EACDF Team Assistant Ian P. Morris Consultant GHN02 Team Member Betty Hanan Consultant GSP06 Team Member Bengt Jacobson Consultant EASHD Team Member Nicolas Drossas Consultant EASNS Team Member David Bruce Whitehead Financial Management Specialist GGO20 Financial ManagementStephen Paul Hartung Financial Management Specialist GGODR Team Member Cristiano Costa e Silva Nunes Senior Procurement Specialist GGODR Procurement Evelyn Villatoro Senior Procurement Specialist GGODR Procurement Jinan Shi Senior Procurement Specialist GGODR Procurement Eric Leonard Blackburn Procurement Specialist GGO08 Procurement Manuela Da Cruz Procurement Specialist GGO08 Procurement Joyce Miriam Denise Witana Procurement Specialist GGODR Procurement Jonas Garcia Bautista Safeguards Specialist GENDR Safeguards L. Panneer Selvam Safeguards Specialist GSU03 Safeguards Mark Hodges Safeguards Specialist IPN Safeguards

38

(b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)

No. of staff weeks USD Thousands (including travel and consultant costs)

Lending FY07 13.61 43.68 FY08 24.55 122.26

Total: 38.16 165.94 Supervision/ICR

FY09 5.79 128.41 FY10 1.96 136.51 FY11 6.24 87.83 FY12 5.09 81.49 FY13 18.47 137.95 FY14 10.91 58.26 FY15 13.68 73.93 FY16 15.54 82.59

Total: 77.68 786.97

39

Annex 5. Beneficiary Survey Results N/A

40

Annex 6. Stakeholder Workshop Report and Results N/A

41

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Comments from the Government of Samoa on the draft ICR are provided below. Datasheet Implementing Agencies: Refer to PAD 2008. MoF is the Executing Agency and MoH is the Responsible Agency. These should be reflected in the document. Cofinanciers and Other External Partners: The SWAp DPs were noted as Pool Partners because their funds were pooled into a Special Purposes Account (SPA) for the Health SWAp Program. That reference became irrelevant as the years progressed with the maturity of the sector-wide approach modality in the health sector. Results Framework Analysis: IOI #1: If the period of review is from 2008-2015 then the Procurement Plan for the Project was achieved because that Plan was updated throughout the years even with the last Technical World Bank mission in 2015. IOI #3: …and Government towards the end of the FY2015-2016. IOI #5: Government’s commitment to the health agenda and Health Sector Program should be acknowledged herein. IOI #20: The Health Sector Infrastructure Project is about 80% complete with the National Pharmaceutical Warehouse and PHC Centre to be completed by July 2016. Despite the delays, the completed infrastructure works (Nurses, Midwifery and Allied Credentialing Centre, Nurses Hostel, National Orthotics & Prosthetics Workshop and the Renovated & Extended MTII Hospital) continue to provide quality facilities for staff and visiting clients. The two medical equipment packages also provided under the Health SWAp Program, comprehensively equipped the new TTM Hospital on Upolu, the referral MTII Hospital in Savaii and Rural District Hospitals around the country. These achievements should be noted in this ICR for the value and difference made in service delivery given the new facilities and much needed medical equipment. The latter which was notably acknowledged by the NHS Biomedical Team in government’s ICR. Section 2.2 Implementation Para. 23: Strategies were grouped under the 7 HSP Outcome areas noted above so that they strategically respond to outcomes and related indicators. That was in conformance to government’s planning framework which was adopted across-government sectors. Section 2.3 Monitoring and Evaluation Design, Implementation, and Utilization Para. 26: The Sector M&E Reports are as follows: (1) HSP Annual Reviews; (2) Health Program Advisory Committee quarterly progress reports; (3) Bi-annual PMRs (Progress Management Reports) and updates of health related MDGs provided to MoF over the years. These sector M&E reports are discussed at various national (eg. Cabinet Development Committee meetings, Quarterly Sector Coordinators’ Forums) and sector forums and serve to provide sector direction and evidence based verifications.

42

Section 2.4 Safeguard and Fiduciary Compliance Para. 31: POWs were discussed and approved at the HAC meetings. All activities dropped from the POWs or priority list were well discussed with related sector agencies. Thus activities which were dropped were replaced by other priorities over time eg. some NHS activities were dropped given the priority given to the infrastructure works and medical equipment packages. The latter were much influenced by what the Govt of the PRC could support with the new teaching hospital. Following procedures, MoH was officially advised of such changes from the NHS Board or GM levels. Para. 32: Procurement activities for the Program continued to be undertaken by MoH through the Health Sector Coordination Division and DG’s Office with the Legal Consultant. Section 2.5 Post-completion Operation/Next Phase Para. 33: How is that? [in reference to further fragmentation of the health sector] Section 4. Assessment of Risk to Development Outcome Para. 51: The emphasis made on the relationship between MoH and NHS is overly stated when it’s more professional than anything else. This was also noted throughout the life of the SWAp, that the DPs emphasis on this relationship has done more harm than any good and that they also contributed to fragmented approaches and procedures because of the disrespect of government’s approved governance and communication frameworks. There are various documented evidence of such nature. Please explain the risk of further fragmentation with the establishment of the HPF? Section 5. Assessment of Bank and Borrower Performance Para. 53: Is this an incomplete sentence? [in reference to the sentence “There were a total of four Task Team Leaders over the course of the SWAp, and changes in staffing among the DPs were also seen.”] Para. 54: The HAC was established with the SWAp since 2008. It’s role was strengthened and re-defined as a policy advisory committee for the sector. Please refer to Redevelopment document provided for that historical achievement. The SCU was also re-defined with the establishment of the SWAp Program in 2007. By 2012 it was re-established and re-named as the Health Sector Coordination, Resourcing and Monitoring Division within MoH. It was the first sector coordination division acknowledged by PSC for all 14 sectors of government and that’s a sector achievement in itself. Para. 55: If this is related to the delays in infrastructure completion then it should be noted herein. Para. 56: Correction. MoH was the Responsible Agency for the Health SWAp. It’s implementation roles is in the management, monitoring, fiduciary, procurement and legal aspects of development assistance. IAs (Implementing agencies) were the service providers (eg. NHS, NKFS, NUS, etc) and sector partners like MWCSD, etc. Again the relationship between the two agencies is overly stated in this review when it should not be the case. It has thus missed the opportunities to note the various sector collaboration and synergies which has resulted in various sector achievements over the years of the SWAp Program.

43

Section 6. Lessons Learned Para. 57: This is part of the health reforms which was not fully realized during the term of the SWAp and is currently being worked out by MoH in the form of Purchaser-Provider arrangements. Annex 2. Detailed Project Components and Outputs by HSP Outcome Component 2: This was taken out of the SWAp Program given priority placed on O&P Workshop etc. Annex 3. Economic and Financial Analysis Box 2, Para. 1: To a large extend, the discrepancies came from initial unrealistic estimates made by the sector indicating lack of proactive analysis on the cost and functionalities of services needed to be renovated and extended. Additional requests for variations to design initially approved, also contributed to the estimated cost initially proposed for MTII for eg. Box 2, Para. 2: The chronological order of this event was provided to MoF and DPs and the mission should have had access to the documentation. The reference here is misleading and should be corrected. For the sake of this exercise, the design works for MTII was approved and signed upon by latter end of 2014. MoH was only advised of the new machine by May 2015. By then the construction works was half way through and contract variation was obvious by then, which NHS had officially noted with MoH of their being responsible for such.

44

The Borrower’s ICR is provided below.

Government of Samoa Summary Activity information

Activity Title Health SWAp Program

(i) Program Development Objective (PDO)

To improve the effectiveness of the Government of Samoa in managing and implementing the Health Sector Plan using results from sector performance monitoring.

(ii) Health Sector Program’s aim in the medium-term (FY2009-2013)

To improve access to, and utilization of effective, efficient and quality health services to improve the health status of the Samoan population.

Joint Partnership Arrangement (JPA)

Start and end date (original)

August 8, 2008 – December 31, 2013

Variation No. 1 December 16, 2013 – December 18, 2015

Total cost USD$33m or SAT$81.4m

Reporting period August 8, 2008 – June 2015

Completion report preparation

Prepared by: Ministry of Health

Others involved or consulted

Health Sector

Date report submitted :

45

Executive Summary Report 1. Background and context: The Health Sector Plan (2008-2018) provides the strategic directions for Samoa’s Health Sector. The Plan outlines the need to strengthen health systems to address existing and emerging challenges to the health of the population. The overarching goal (or mission) of the Plan is to regulate and provide quality, accountable, and sustainable health services through people working in partnership. The Plan identifies six priority areas and strategies: (i) health promotion and primordial prevention; (ii) quality health service delivery; (iii) governance, human resources for health and health systems; (iv) partnership commitment; (v) health financing; and (vi) donor assistance. It also emphasizes the need to improve the quality dimensions of health programs, including enhancing the quality of health staff and staff numbers to reduce currently evident capacity constraints. To monitor and evaluate the success of the Program in meeting its objectives, agreement was reached between the health sector and the development partners on: (a) a set of five strategic sector performance indicators; and (b) a set of intermediate indicators for each component of the Program. The five strategic sector performance indicators included the: (i) control of non-communicable diseases; (ii) child health, morbidity and mortality; (iii) reproductive and maternal health; (iv) control of communicable diseases; and (v) injury prevention. The full details of these program performance indicators are the core elements of the Health SWAP in Samoa. These are (i) clear sector policy and strategy as articulated in the Samoa Development Strategy, the Health Sector Plan 2008-2018, and corresponding 3-year health entity Corporate Plans; (ii) the health sector expenditure program(MTEF) with prioritized activities aligned with the HSP over the period FY2009-2013(government and development partner funds); (iii) a common performance management system for the sector with common-program indicators and mechanisms for donor coordination and partnership arrangements; (iv) processes for harmonization of budgeting, financial management, procurement, and reporting; and (v) a consultative mechanism for dialogue among public and private health sector providers, clients, and civil society led by Government.

Initially the Health SWAp was designed to have three broad components, which were directly aligned with the PDO. These were:

Component 1: Health Promotion and Prevention Component 2: Quality Health Care Service Delivery; and Component 3: Strengthen Policy and Regulatory Oversight of the Health System

By 2010 the Government of Samoa along with its Development Partners jointly agreed to consider a more outcome based approach to the Health SWAp. The redevelopment in essence allowed the Health Sector to rethink and readjust the design of the Program as a step forward in ensuring maximum synergies and participation is realized between related activities and programs that were driven to meet a common broad outcome. Sector ownership was key to allow the capacity within to lead, manage development and to utilize government systems which were in place, build on the partnerships developed over time to bring about mutual accountability and more effective

46

harmonization of analysis, funding and reporting; reduce the fragmentation of aid, increase value for money and to strengthen sector linkages. The transition was made effective and simplified due to the fact that prior to the Paris Declaration in 2005, Sāmoa had already begun a continuing program of reforms amongst which was the key objective to achieve development effectiveness. In fact the analysis of development assistance options in the Pacific by then, had indicated that Samoa was well placed to adopt a greater harmonization of donor policies and procedures. Government had therefore by the time the Health SWAp became effective had implemented and institutionalized two other sector SWAp Programs – Water and Education – and the re-design phase of the Health SWAp was to further strengthen aid management for better development results at national and sectorial levels. 2. Assessment against the Results Framework for Performance Monitoring:

Program performance in terms of health status, linked with broader national and international health development goals (e.g MDGs) The indicator results for the MDGs indicated that progress was made in reducing child mortality, increased vaccinations, reduced maternal mortality, increase births attended by skilled health personnel, increase ANC coverage, increase knowledge of HIV/AIDS, slight decrease in hypertension prevalence. Challenges included a reduced rate of contraceptive use, increased adolescent birth rate, and increase of diabetes. Refer to Appendix 1 for full results.

Intermediate health sector outcomes

The results framework was made up of 6 objectives; 1 SWAp institutional development objective and 5 intermediate objectives. They are defined as to overall “improve the effectiveness of the Government of Samoa in managing and implementing the HSP based on the use of results from sector performance monitoring”. This is supposed to reflect through: (1) Policies and plans implemented according to agreed priorities; (2) Greater efficiency in use of resources; (3) National ownership and commitment to health sector program; (4) Results of performance monitoring used in shaping the program implementation, and; (5) Improved health sector governance. Each of these objectives are measured by 2 indicators. Based on the indicator results for these 6 objectives, the table below summarizes the progress for the Health SWAp Program. The overall institutional development goal, Objective 3, and Objective 4 are areas where progress is a challenge. See Appendix 2 for a list of the SWAp results by indicator. Intermediate Indicator Results Grouped by Objective

Objective Baseline YR1 YR2 YR3 YR4 YR5

SWAp Inst. Dev. Objective

n/a n/a n/a n/a n/a n/a

Objective 1 Complete Complete Complete Complete Complete n/a

Objective 2 50% Complete Complete Complete Complete n/a

Objective 3 n/a 50% 50% 50% 50% 50%

Objective 4 n/a unchanged unchanged unchanged unchanged unchanged

Objective 5 n/a Complete Complete Complete Complete Complete (ongoing)

n/a= no data available to report

47

Monitoring performance of the health SWAp process itself • Outcome indicators were used to evaluate the performance of the Health SWAp in terms of

achieving priority objectives of the Health Sector Plan, however it did not consider the revised Sector priorities that were identified during the lifespan of the SWAp Program.

• Baselines were supposed to be established in the first year and performance reviewed on key indicators as part of the Health Summits. Some of the baselines are not applicable to its respected project outcome indicators.

3. Risk Management: Throughout the implementation of the activities under the three components, there have been challenging risks encountered in implementing of the SWAp Program as well as its outcomes. Below are some notable examples under the three components:

Table 1 Risk Management and Mitigation Measures

Risks Risk Rating

Mitigation and Adaptation Measures

Breakdown of sector partnerships derailing the Program’s implementation

M Broad consensus on core principles and Program design by all key partners.

Strong GoS Aid Coordination mechanism in MoF. Notable Sector Coordination platform managed by MoF

for the 14 Sectors which ensures quarterly. HPAC meetings regularized. Utilisation of Govt systems, management and

implementation. Continuous participatory engagement and consultations

especially with end users through design and construction meetings, public health sector forums and HPAC meetings, to ensure ownership.

Financial management risk with changes in health priorities, misuse of funds, use on ad-hoc activities that were not approved.

Capacity limitations delay Program implementation

S Mobilize existing latent in-country technical expertise. Support for capacity building at all levels of the health

system, including through technical assistance, pre – an in-service training, systems and procedures development. Efforts to retain build capacity in the Sector.

Procurement delays result in slowed Program implementation

H Use/retain built capacity for procurement. Procurement assessment and capacity action plan. Early procurement planning and preparation of

procurement documents for first 18 months of Program implementation.

Hiring of Procurement Specialist for NHS to expedite and facilitate the procuring of NHS activities.

Provided support for capacity building.

48

Conflict between health sector entities over evolving responsibilities and accountabilities within the sector entities maintain an open and participatory sector dialogue.

H The health sector reform is part of an overall public sector reform program with strong central agency and political support.

Formal arrangements between DPs and MOH and other health entities provide for open sectoral dialogue and sharing of information.

Government’s Official Communication Process reaffirmed and reinforced at all levels.

Weakened commitment from PSC to support health sector reforms especially in capacity building.

H High degree of commitment from PSC to approve sectorial organisational changes for eg. HSCRM Division by 2012 and positions.

MOH continues to dialogue with the PSC on critical positions requiring the strengthening of work undertaken by the Health Sector Program.

Prolonged feedback/endorsement on issues related to Program positions such as the Finance Manager, Projects.

4. Monitoring and Evaluation Matters / Issues: The assessment of the M&E Manual also carried out by the consultant Mr Phillip Davis by June 2013, noted that the M&E was identified as a priority from the beginning of the Health Sector Program. In the PAD, an M&E framework was suggested with two key measuring focal points: program outcome indicators to monitor health sector performance; and SWAp processes monitoring indicators to assess institutional developments outcomes, previously discussed. However, implementing M&E components in the monitoring framework of the Program were very challenging. There was no formal review of quality of Results Framework implementation, therefore there are no empirical data to measure the impact of implementation efforts. This section reflects on the quality and implementation of the results framework developed as per SWAP activities. First, the results framework for SWAp will be presented and summarized. Please refer to Appendix 2 for SWAp Monitoring Framework. Second, a brief summary of the outcome and indicator results for the Health Sector Monitoring and Evaluation (developed to supplement SWAp processes) will be presented for the context. Third the progress made in the implementation of health sector and SWAp monitoring and evaluation will be discussed. Finally, we will provide lessons learnt from implementation, challenges and recommendations. A. Component 1- Health Promotion and Prevention

Table 2 below summarizes the indicators as per SWAP processes for Component 1. Data for waste management in particular is not currently available. However, MOH is currently working with partners to solidify implementation of M&E procedures for this indicator.

49

Table 2. Component 1 Indicators

Indicator Baseline 2008-9 2009-10

2010-11

2011-12

2012-13

Freq. and Reporting

Instruments Responsibility for Collection

Project Outcome Indicators

Prevalence of Diabetes

22.30% 22.30% 49.70% Yr 2, Yr 5, annual cases

STEPS, Diabetes register

MOH, Diabetes Foundation

Adolescent Birthrate 15-19 years (annual)

42.6 birth per 1000 women aged 15-19 yrs.

44 per 1000 56 per 1000 5 years DHS 2009, 2014

SBS, DHS, MOH

Injuries in Children <15

237 216 366 293 267 282 Annual PATIS NHS, MOH

Intermediate Outcome Indicators

People 25-64 years overweight and obese

85.6% and 56%

85.6% and 56.0%

89.1% and 63.1%

Yr 2 and 5 STEPS MOH

Percentage of children under 1 yr. received at least one dose of measles vaccine

45% 55.70% 62% 67% 85% 98% Annual EPI/NHS, DHS

NHS, MOH

Improved medical waste management

Forthcoming April 30th

Annual Waste Management Audit Report

NHS, MOH

B. Component 2- Enhancement of Quality Health Care Service Delivery

Table 3 below summarizes the indicators as per SWAP processes for Component 2. It should be noted that data for primary care, antenatal care visits and rheumatic heart disease treatment compliance have years of missing data.

50

Table 3. Component 2 Indicators

Indicator Baseline 2008-9 2009-10 2010-11 2011-12

2012-13 Freq. and Reporting


Project Outcome Indicators

Perinatal Mortality (22 wks gestation to <7 days)

23.90% 7 per 1000 pregnancies of 7 mo. Or more

8 per 1000 pregnancies of 7 mo. Or more

Annual DHS 2009, 2014

SBS, DHS, MOH

Adolescent Birthrate 15-19 years (annual)

42.6 birth per 1000 women aged 15-19 yrs.

44 per 1000 56 per 1000 5 years DHS 2009, 2014

SBS, DHS, MOH


Primary Care Utilization (gender, age, domicile)

Forthcoming April 30th

Annual CHNIS, PATIS

NHS, MOH

Antenatal care coverage for at least 4 visits

61% (min. 3 visits)

58.40% 64% 73% CHNIS, PATIS

NHS, MOH

Proportion of Rheumatic Heart Disease patients complying with treatment 12/12 months

84% 86% Annual Rheumatic fever register

NHS

51

C. Component 3- Strengthening Policy, Monitoring and Regulatory Oversight of the Health System

Table 4. Indicators as per SWAP processes for Component 3

Indicator Baseline 2008-9 2009-10 2010-11 2011-12 2012-13 Freq. and Reporting



Staff mix and distribution according to national standards

Vacancies Data 2008

Registered Dental Practitioners – 17 Doctors Medical Practitioners – 103 Doctors Registered Pharmacists - 15 Source: Ministry of Health - Registrar 2015

Annual MOH

Evidence of performance monitoring leading to policy and regulatory action to improve health services

2008-2009: Interim database for HRH for the health sector in place as a starting point in identifying staff gaps by facility and qualification. Tobacco Control Policy 2010 developed to enforce the implementation of and public compliance to the Tobacco Control Act 2008. 2009-10: Tobacco regulations have allowed the Ministry to issue penalties for non-compliance. The development of the General Prevention Policy and Health Promotion Policy: Strengthening of ICT System trainings-led to the development of the health information systems and HIS policy 2010-2011: Develop capacity to implement evidence based primary prevention program in Oral Health (Fiji) 2011-2012: Improved Diagnosis and Management of Paediatric cardiac disease (New Zealand) 2012-2013: Imaging Equipment Refurbishment Program (Radiology)

Annual Review report and POW

MOH

demonstrated outcomes of training plan by component

n/a See Appendix

See Appendix

See Appendix

See Appendix

See Appendix

Annual Review report and POW

NHS and MOH

52

4 (a) Summary of Health Sector M&E Framework

A framework for monitoring sector performance was developed as part of the SWAP preparation. This framework contains seven (7) long-term health outcomes, each with a core set of indicators. Indicators for measuring performance are at 3 levels; national/sector level, institutional level (ministry level), and operational level. The manual provides a list of core health indicators identified by the Health Sector. A minimum set of core indicators have been used for reporting health performance on a 6 monthly, annual and 2-5 year basis.

These indicators and outcomes were operationalized in 2011 through the Monitoring & Evaluation Operational Manual 2011, which was a sector wide framework for monitoring performance. Additionally, this framework was incorporated in the Health Sector Plan for 2008-2018 under Key Strategy 3 which seeks to strengthen health information development and management through surveillance, research, and health intelligence. The 7 outcomes are as follows. Table 5 shows a brief summary of indicators for each outcome area of the results framework:

Table 5. Results Framework Summary

Outcome 1 Outcome 2 Outcome 3 Outcome 4 Outcome 5 Outcome 6 Outcome 7

Prevalence of Alcohol Drinkers

Prevalence of Diabetes

Prevalence of STI's

Life Expectancy at Birth

Maternal Mortality Rate

Waiting Time for Emergencies, Triaging & General Outpatients

Climate Change Indicators (in development)

Prevalence of Current Smokers

Prevalence of Hypertension

Prevalence of Notifiable and Vaccine Preventable Diseases

Total Fertility Rate

Percentage of Births Attended By Skilled Health Personnel

Health Facilities and Providers That Are Accredited and Certified

Compliance with Disaster and Emergency Response Plans & Legislation

Prevalence of People Who Are Physically Active

Prevalence of Overweight and Obesity

Incidence and Death Rate Assoc. with TB

Adolescent Birth Rate

Infant Mortality Rate

Health Facilities Compliance with Legislation, Policies, Protocols, and Standards

Percent of Health Professionals with Competencies to Respond to Disasters

Proportion of the Population Who Eat 5+ Servings of Fruit and Vegetables a Day

Prevalence of Rheumatic Heart Disease

Acute Respiratory Infections Among Children Aged <5 Years

Contraceptive Prevalence Rate

Under 5 Mortality

Proportion of Clients Satisfied With Health Services

Prevalence of Cancer

Prevalence of HIV

Percentage of Infants Exclusively Breastfed

Health Facilities Utilization Rates

53

Table 5. Results Framework Summary

Outcome 1 Outcome 2 Outcome 3 Outcome 4 Outcome 5 Outcome 6 Outcome 7

Injuries in Children <15

Prevalence of Cervical Cancer in Women Aged 20 Years and Over

Percent of Health Personnel Aged 55 Years and Over

Number of Attempted Suicides

Antenatal Coverage

Health Professional to Population Rates

Proportion of 1 Year Old Children Immunized Against Measles

Total Health Expenditure

Proportion of Fully Immunized Children Immunized

Government Expenditure on Health

5. Governance and Management Matters and Issues: 5.1 Time period in designing and planning of the program for implementation was one year and was too rushed for such a comprehensive Program. Some shortcomings in terms of the sector’s readiness and capacity of the SWAp Unit, may have been overlooked. 5.2 The establishment of the Health Sector Coordination, Resourcing and Monitoring Division (HSCRM) by 2012 has allowed the MOH to oversee most aspects of the Health Sector Program (which the Health SWAp is part of) which has reduced transaction costs for government. It has also permitted a more holistic outfit with most development programs being coordinated from the Division to ensure synergies and avoid duplication of efforts and waste of resources. 5.3 Adoption of the SWAp modality has led to improvements in GoS capacity to plan and manage complex health sector projects. This has provided the continuous support from Government Central Agencies MOF as the Executing Agency of the Program, the AG’s Office and the Development Partners – namely DFAT, NZAP and the World Bank - has been a success throughout the lifespan of the SWAp Program. The significant development assistance provided for the Health SWAp in fact gave oomph and substance to the implementation of the HSP. It is also noteworthy to mention that through the SWAp modality, GoS has had previous experience with SWAp (in water and sanitation and education sectors) and Ministry of Finance appears to have had good capacity to serve as executive agency. The use of a single system to record and report SWAp-related finances, while initially demanding, has been more efficient than complying with varying individual project requirements.

54

There have also been rigorous procurement processes combining those of the GoS Tenders Board and the World Bank have been applied throughout the life of the SWAp. While those processes have created challenges and brought about delays in implementation, it is reasonable to assume that value for money and capacity building ought to have been achieved in all significant purchases of goods and services. 6. Lessons Learnt:

6.1 SWAp modality was and continues to be effective in harmonizing development activities for the sector and harmonizing assistance from donors to effectively realizing key strategic areas of the HSP. There have been excellent opportunity for professional development in specific areas given the investments in new medical equipment (eg. for Biomedical Team) and health infrastructure (eg. PMT).

6.2 The HSCRM Division sees the need to continuously review the Master Work Plan to be inclusive of activities supported under the local budget and development assistance such as WHO, UNFPA, Global Fund, SPC, SWAp etc to ensure holistic planning and reporting. This process has strengthened sectoral and across sector approach and place emphasis on synchronizing indicators and liked-activities.

6.3 The Program provided an effective change from Outcome-based Program rather than Component-based focused Program as it was originally designed. The Master Work Plan that guided the implementation of the Program’s activities follows the 7 health outcomes in which directly links to the HSP 2008 – 2018. The establishment of “Outcome Based Groups” for each health outcome to meet on a regular basis to plan, monitor and coordinate the health sector response or Program Implementation Plan; is the way forward.

6.4 The whole M&E System including personnel and framework should have been in place first before the Program was implemented. That would have ensured that specialized M&E person(s) were hired to monitor the implementation of SWAp activities as part of the whole M&E Framework and synchronize all the information to ensure that it is in line with Program objectives. Understanding and awareness of health workers on Health Sector developments (e.g M&E Framework, MTEF, etc.)

7. Overall judgement and critical matters and issues: The SWAp Program has enabled the country and the government as a whole to overcome inefficiencies, lack of government ownership, and a number of other problems that were constraining the impact of international support to our country. This Program and the approach it has taken place for the Health Sector embraced many of the principles of harmonization and alignment that were later endorsed by the Paris Declaration on Aid Effectiveness in 2005 and subsequent international meetings. This Program has become a locally owned program for the Health Sector in a comprehensive and coordinated manner. In saying this, the Program has been able to move towards the use of country systems. Acknowledgement on the successes of the Health SWAp program at CDC level by the

55

Prime Minister and the Forum Combat Peer Review for Samoa 11 – 12 November 2013; have been some of the main highlights. In fact it has been acknowledged over its lifetime that the program provided the main modality in implementing the HSP (2008-2018). In other words, the Health SWAp Program provided oomph or substance that gave life to the HSP. In respect of programme outcomes of the Program, it is visible that the SWAp has delivered benefits in a number of important areas especially with the services, goods and civil works provided for the sector. Despite these, the Program appears to have fallen short of expectations in a number of program outcome areas. Some of the more significant examples are: Failure to develop and implement a comprehensive health information system and

currently no HIS Policy which has had ‘knock-on’ adverse impacts on the ability to acquire and analyse health sector activity data for M&E purposes.

Slow progress with the three key capital works projects – pharmaceutical warehouse; orthortics and prosthetics facility; renovation and extension of the MTII Hospital and the renovations of the Primary Health Care Centre.

Delays in establishing improved cervical and breast cancer screening programmes. Aside from that, fiscal analysis has confirmed that there has been fiscal space for the Program given its limited demands for incremental recurrent costs (ie. SWAp Unit Operational Costs), which are estimated at less than 1.5percent of the health budget during the five years of the Program. Sustainability of the Program is also being enhanced by improved composition of recurrent budget allocations, including through increased allocations for maintenance and critical operation costs for service delivery, particularly in rural areas and for rural outreach. The Government's strong ownership of the HSP and the policy and investment priorities embedded therein are critical to its overall sustainability. The participatory bottom up planning process has increased health stakeholder ownership of the Program. The technical soundness of policy and investment choices included in the MTEF affect the sustainability of overall health sector financing. Specifically the focus on both primary and secondary prevention to contain costs, develop staff capacities, address key skill constraints and strengthen core government systems as part of Program implementation will mean that the future costs of the health sector in dealing with disease and disability will be substantially reduced by the Program's investments. Those which should enhance sustainability of health sector financing. Consolidating and strengthening approaches for capacity development, including (i) rationalization of the selection and oversight of technical assistance (to ensure it contributes to building local capacity and training of local staff); and (ii) establishing innovative arrangements for capacity building, including use of national policy, planning, budgeting, and reporting procedures to the extent possible. All such developments which has ensured that the health sector strategically aligns its planning and progress reporting according to the Government of Samoa’s Strategic Planning and Policy Framework. 8. Final actual expenditure against budget: Total Budget = SAT$81.4 million Total Expenditure as at June 30, 2015: SAT$61.4 million

56

Total Balance (Works and Goods) = SAT$20 million

57

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Australia (DFAT) and New Zealand (MFAT) response on the Implementation Completion and Results Report (IDA-44320 & IDA-47210 for the Health Sector Management Program Support Project and Additional Financing We are pleased to have worked in partnership with the Government of Samoa and the World Bank to implement this project, commonly known as the Health Sector Wide Approach (SWAP). The World Bank has produced a comprehensive Independent Completion Report (ICR). We welcome the opportunity to comment. We note that the evaluation is an independent Bank process and that the overall rating of moderately unsatisfactory is based on the Bank’s review and rating criteria. We were pleased to attach Australia and New Zealand’s own independent consultant to participate in this evaluation and we appreciate the flexibility shown by the World Bank to accommodate this. New Zealand and Australia initially funded this initiative from 2008 to 2013, and then agreed to a no-cost extension until December 2016 to ensure the completion of key infrastructure. The World Bank discontinued its involvement in the Project on December 2015. We urge that upmost effort be taken to ensure that the remaining infrastructure is completed to high quality. We are pleased that the ICR highlights the strong ownership by the Government of Samoa of this project. This ownership is evident by the establishment and ongoing role of the Health Programme Advisory Committee which continues to work across the sector and the flexibility of the programme to respond to the natural disasters which occurred during the program. The SWAP will have a positive legacy that will include:

Improved policies and processes Improved infrastructure including a new Pharmaceutical Warehouse; Orthotics and

Prosthetics facility; renovated MTII hospital and Primary Health Care Centre; and The purchase and upgrade of equipment for the hospitals and outreach.

Australia and New Zealand continue to support the health sector through a range of activities, and both agencies are currently undertaking design of future assistance to support improved health outcomes in Samoa. The lessons learned from the Health SWAp are welcome and will be important in informing future engagement in the health sector by Australia and New Zealand. We note some of the key lessons for all parties highlighted in the ICR:

More attention was required to clarify the roles and responsibilities of each health sector agency at the outset of the program. Despite this being recognised as “high risk” prior to the program commencing, it was not adequately addressed by all parties at that time.

58

More attention was required to address recurrent costs and asset maintenance associated with new initiatives. For example the Health Care Waste Management system. We recognise that this issue will require close attention as new health infrastructure is completed and is no longer within their defects liability period.

Procurement challenges were encountered over the course of the Project. This resulted in delays and cost overruns and a requirement for ongoing capacity building.

The lack of appropriate and measurable indicators and the lack of early establishment of

baseline information impeded timely problem solving. This in turn made it difficult to draw on evidence to inform programming and resource allocation.

Despite the stated emphasis of the Health Sector Plan on promotion, prevention and primary care services, resource allocation to these areas did not match ambition.

The focus of the SWAP concentrated on the capital resources provided by development

partners and how these were being utilised rather than on the total quantum of effort and resources available to the sector. This is evident in that the Medium Term Expenditure Framework (MTEF) had not been updated since 2012. Recent efforts by the sector to map the full range of development assistance to health sector agencies and how this assistance supports the achievement of their plans is an indication that this issue has been recognised and is in the process of being addressed.

Looking ahead Australia and New Zealand note the importance of continuing to work in partnership with the Government of Samoa to achieve Samoa’s vision of a “healthier Samoa”. In doing so, we note the importance of cooperation, coordination and mutual accountability and the need for regular dialogue, including at a senior leadership level.

59

Annex 9. List of Supporting Documents

1. Project Appraisal Document, May 29, 2008

2. Financing Agreement (C4432-WS), July 23, 2008

3. Amendment to Financing Agreement, Credit 4432-WS, January 28, 2009

4. Amendment to Financing Agreements Credit 4432-WS and 4721-WS, December 3, 2013

5. Joint Partnership Agreement, July 23, 2008

6. Amendment to the Partnership Agreement, Credit 4721-WS and Credit 4432-WS, February 5, 2014

7. Implementation Status and Results Reports No. 1-15

8. Aide Memoires 2008-2015

9. The Economic Costs of Noncommunicable Diseases in the Pacific Islands: A Rapid Stocktake of the situation in Samoa, Tonga and Vanuatu, November 2012

10. Samoa Public Expenditure Review Notes - Taking stock of expenditure trends from FY06-FY12, March 2014

11. National Kidney Foundation Annual Report, 2013-2014

12. Evaluation of Samoa Health Sector Management Programme (Health SWAp), Philip Davies, May 2013

13. Summary of Recommendations from previous reports on Health Sector M&E and HIS in Samoa, Joan Marfarlane, January 2015

14. Assessment of Monitoring and Evaluation for the Samoa Health Sector Program, Nick Davis, June 2013

15. Health Sector Plan, 2008-2018, Mid-Term Review Report, Ministry of Health, 2013

60

Annex 10. List of Indicators and Arrangements for Results Monitoring (PAD)

PART A: Health Sector Program Indicators Program Development

Objective Impact Indicators Use of the information

The proposed health sector program’s aim in the medium-term (2009-2013) would be to improve access to, and utilization of effective, efficient and qualityhealth services to improve thehealth status of the Samoan population.

Control of non-communicable diseases Prevalence of diabetes Improved maternal and child health Perinatal mortality rate Universal access to reproductive health services Adolescent birth rate Control of communicable diseases Incidence of water- and food borne

infections Injury prevention and management24 Injuries in children (under 15 years)

Outcome indicators will be used toevaluate the performance of the health sector program in terms of achieving priority objectives of theHSP. Baselines will be established in thefirst year and performance reviewed on key indicators as part of the Health Summits.

Intermediate Results One per Component

Results Indicators for Each Component

Health Promotion and Primordial Prevention: Supporting the transformationof the health sector from a medical model towards wellness orientation and health promotion

Component One: People aged 18yrs and over

overweight or obese Percentage of children under 1 year

received at least one dose of measles vaccine

Improved medical waste management

These are proxy indicators for behavior change in the Samoan population in line with objectives of the HSP and the program.

Quality Health Services: Improving the quality of health services through strengthened human resources, standards, supplies, equipment and infrastructure

Component Two:

Primary care utilization by gender, age, domicile

Antenatal care coverage for at least one visit

Proportion of Rheumatic Heart Disease patients complying with treatment

These are proxy indicators for improvement in quality of health delivery

Improving Policy, Monitoring and Regulatory Oversight of the Health Sector: Support the MOH in its policy development, coordination and regulation of the sector.

Component Three :

Staff mix and distribution according to national standards

Evidence of performance monitoring leading to policy and regulatory action to improve health services

These are proxy indicators of improved health sector stewardship.

24 To include S. typhi

61

Program Development Objective

Impact Indicators Use of the information

Demonstrated outcomes of training plan by component

62

PART B: Health SWAp Process Monitoring Indicators

Expected Outcome Program Indicator Use Health SWAp development objective: To improve the effectiveness of the Government of Samoa in managing and implementing the HSP based on the use of results from sector performance monitoring

Percent health sector budgets and disbursements conform to policy objectives and HSP priority areas25

Share of annual outpatient visits by poorest quintile of population (indicator of equity of access – HIES)

Improved public expenditure management, allocating resources aligned to the priorities.

Intermediate objective 1: Policies and plans implemented according to agreed priorities

MTEF and related Procurement Plan updated and adjusted based on recommendations from sector reviews

Key Sector Partners’ corporate Plans and Government investments aligned with HSP priorities.

Improved execution of budget by programs.

Intermediate objective 2: Greater efficiency in use of resources

Percentage of SWAp Program funds released according to agreed schedule

Number of reported drug stock-outs by facility26

Reduced duplication and better deployment of limited resources

Intermediate objective 3: National ownership and commitment to health sector program

Health expenditure as percentage of govt. expenditure

Stakeholders participation in program planning and implementation reviews

National ownership of the development process.

Intermediate objective 4: Results of performance monitoring used in shaping the program implementation

Disaggregation of data by sex, age and domicile enhances planning for services

Linking program results with budget allocation in iterative process

Intermediate objective 5: Improved health sector governance

MOH Financial Audits submitted on time and action plan agreed for resolving outstanding issues.

DHS and other statistical reports completed within

Mitigate the risk of corruption. Improving accountability

25 Proportion of budget expenditure on public health, primary care, hospital services and administration.

26 Any drug out of stock for more than one week at any health facility.

63

Expected Outcome Program Indicator Use stated timeframe and made public

65

Arrangements for Results Monitoring

Target Values Intermediary SWAp Outcome Indicators

Baseline YR1 YR2 YR3 YR4 YR5

Percent health sector budgets and disbursements conform to policy objectives and HSP priority areas Share of annual outpatient visits by the poorest quintile of the population MTEF and related Procurement Plan updated and adjusted based on recommendations from sector reviews Key Sector Partners’ corporate plans and government investment aligned with HSP priorities

SWAp begins27 To be determined by HIES28

Complete Complete

70% Complete Complete

Complete Complete

90% Complete Complete

Biennial Biennial Annual Annual

27 SWAp will introduce mechanism for Government to better align its resources with priorities. This indicator could be tracked using proportion of budget expenditure by the various program activities.

28 HIES will be completed by end 2008. The indicator is intended to measure equity and may be revised after the first HIES. At the March 2009 Summit baseline and target will be established for the rest of the Program.

66



Percentage of SWAp Program funds released according to agreed schedule Number of reported drug stock-outs by facility Health allocation as percentage of govt. budget expenditure Stakeholders participation in program planning and implementation reviews Disaggregation of data by sex, age and domicile enhances planning for services MOH Financial Audits submitted on time and action plan agreed for resolving outstanding issues. DHS and other statistical reports completed within stated timeframe and made public

15% None None

70% 15% Full Partial Complete

100% <20% of baseline 15% Full Complete

100% <10% of baseline >15% Full Full Complete

100% <10% of baseline >15% Full Complete

Annual Annual Annual Biennial Annual Audits Annual

67



Complete June 30, 2009

Complete

Complete

Complete

68

Annex 11. Summary of Agreements Date Name/No Summary July 23, 2008 Financing Agreement (C4432-WS) Original financing agreement – SDR 1.9 million January 28, 2009 Amendment to Financing Agreement,

Credit 4432-WS Extends dates for development of TORs to conduct DHS from June 2009 to December 2009

April 13, 2010 Recommendations of Statutory Committee for Credit 4721-WS (Closing Package)

Provides additional financing in the amount of SDR 2 million (100% allocated to goods, works, subgrants, consultant services, training and interim operating costs)

May 20, 2010 Supplemental Letter ref. Performance for Credit 4721-WS (Closing Package)

Provides Results Framework for the Project, and notes that “A major contribution expected from the project is to further strengthen the mechanism of consultation among health sector partners to set priorities for resource allocation monitoring and making adjustments in the Program based on the results.”

December 3, 2013 Amendments to Financing Agreements, Cr. 4432-WS and 4721-WS

Extends closing date to Dec 18, 2015. Financing reallocated so that it finances goods, works, subgrants, consultant services, training and interim operating costs for the Project. In original FA, $1,286,500 had been unallocated.

February 5, 2014 Amendment to the Partnership Agreement Extends closing date to Dec 18, 2015

69

Annex 12. SWAp Governance and Administration The following table, developed from information contained in the Joint Partnership Arrangement between Pool Partnership and Samoa for C4432-WS29 (July 23, 2008) includes a summary of the governance and administration of the SWAp: Organization/Agency Role Ministry of Finance Functions as the Executing Agency. MOF along with the MOH will have responsibility

for management of the financial contributions of the DPs. Ministry of Health Overall responsibility for Program administration, financial management and

procurement. It will also have overall coordinating role for the implementation of the Program.

Health Program Steering Committee Policy oversight and oversee Program steering and implementation. The HPSC will comprise high level GoS officials (MOH, MOF, NHS, Ministry of Women Community and Social Development, and MESC), coordinating representatives from the DPs, a representative of other health institutions, a representative of the private sector, and a representative of NGOs. The HPSC will provide coordination among local stakeholders and external dialogue and coordination with DPs. The HPSC will meet on a quarterly basis or more frequently if the need arises. The SCU will be the Secretariat for the HPSC.

SWAp Coordination Unit in the MOH Overall coordinating role for the Program. The SCU, led by the SWAp Coordinator, will be responsible for day-to-day Program administration, procurement and financial management. Each activity will be managed by an Output Manager, consistent with GoS budget and accountability structure. The SCU is responsible for:

i. Procurement and financial management under the Program. ii. Coordinating the preparation of the Annual Program of Work (APW) and

budget forecasts. POWs will be prepared in cooperation with the Component Focal Persons and will follow the MTEF process.

29 The agreement was between GOS and AusAID, NZAID, IDA, UNFPA, UNICEF, and WHO.

70

iii. Coordinating the preparation of quarterly and annual Program Management Reports (PMRs). PMRs will be prepared in consultation with component focal persons. Each PMR will contain information in relation to procurement, financial management, a summary of implementation of activities (by component), identification of problems and/or issues, recommendations as to how they may be solved, and an update of the rolling annual procurement plan (APP).

iv. Coordinating in collaboration with MOH, MOF, and the HPSC, the organization of the annual Health Summits (in March) and Joint Reviews (in September).

v. Provide Secretariat support to the HPSC. vi. Prepare the Financial Monitoring Reports (FMRs) for the Program.

Coordinating Development Partners (CDPs), i.e. representatives of the Development Partners

Appointed to support coordination among the Development Partners and the GoS and act as a focal point of communication between all Partners. The CDP role will be shared between one bilateral development partner and one multilateral development partner at any one time and rotated every eighteen months.

The JPA also specified the focal person for each of the components:

Component 1 will be the ACEO for Health Promotion and Prevention in MOH Component 2 will be the General Manager of NHS Component 3 will be the ACEO for Strategic Development and Planning in the MOH

71