Document of The World Bank Report No: ICR00002051...SB$ 1.00 = US$ 0.14 US$ 1.00 = SB$ 6.76 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS AusAID Australian Agency

Document of The World Bank

Report No: ICR00002051

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H3600)

ON A

GRANT

IN THE AMOUNT OF SDR 1 MILLION (US$ 1.5 MILLION EQUIVALENT)

TO

SOLOMON ISLANDS

FOR A

HEALTH SECTOR SUPPORT PROGRAM

TECHNICAL ASSISTANCE PROJECT

June 29, 2012

Human Development Sector Unit East Asia & Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective June 29, 2012)

Currency Unit = Solomon Islands Dollars SB$ 1.00 = US$ 0.14 US$ 1.00 = SB$ 6.76

FISCAL YEAR

January 1 – December 31

ABBREVIATIONS AND ACRONYMS AusAID Australian Agency for International Development DHS Demographic and Health Survey DP Development Partner FM Financial Management HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome HMIS Health Management Information System HSSP Health Sector Support Program HSSP-TA Health Sector Support Program – Technical Assistance ICR Implementation Completion and Results Report IDA International Development Association IFR Interim Financial Report ISN Interim Strategy Note ISR Implementation Status and Results Report M&E Monitoring and Evaluation MDPAC Ministry of Development Planning and Aid Coordination MHMS Ministry of Health and Medical Services MOF&T Ministry of Finance and Treasury MTEF Medium Term Expenditure Framework NGO Non-government Organization NHSP National Health Strategic Plan PAD Project Appraisal Document PDO Project Development Objective PER Public Expenditure Review SIG Solomon Islands Government SPC Secretariat of the Pacific Community STI Sexually Transmitted Infection SWAp Sector Wide Approach TA Technical Assistance TAL Technical Assistance Loan TOR Terms of Reference UNFPA United Nations Population Fund UNICEF United Nations Child Fund WHO World Health Organization

Vice President: Pamela Cox Country Director: Ferid Belhaj Sector Manager: Toomas Palu Project Team Leader: Susan Ivatts ICR Team Leader: Susan Ivatts

Solomon Islands Health Sector Support Program Technical Assistance Project

Contents Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 3 3. Assessment of Outcomes ............................................................................................ 9 4. Assessment of Risk to Development Outcome ......................................................... 14 5. Assessment of Bank and Borrower Performance ..................................................... 15 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 19 Annex 1. Project Costs and Financing .......................................................................... 21 Annex 2. Outputs by Component ................................................................................. 22 Annex 3. Economic and Financial Analysis ................................................................. 28 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 29 Annex 5. Beneficiary Survey Results ........................................................................... 31 Annex 6. Stakeholder Workshop Report and Results ................................................... 32 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 33 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders ...................... 38 Annex 9. List of Supporting Documents ...................................................................... 42 MAP IBRD 35742 ........................................................................................................ 43

A. Basic Information

Country: Solomon Islands Project Name: SB Health Sector Support Project (TA)

Project ID: P097671 L/C/TF Number(s): IDA-H3600 ICR Date: 06/28/2012 ICR Type: Core ICR Lending Instrument: TAL Borrower: SOLOMON ISLANDS Original Total Commitment:

XDR 1.00M Disbursed Amount: XDR 0.82M

Revised Amount: XDR 0.82M Environmental Category: C Implementing Agencies: Ministry of Health and Medical Services Cofinanciers and Other External Partners: B. Key Dates

Process Date Process Original Date Revised / Actual Date(s) Concept Review: 05/09/2006 Effectiveness: 07/14/2008 07/14/2008 Appraisal: 11/01/2007 Restructuring(s): 08/26/2010 Approval: 03/20/2008 Mid-term Review: 09/02/2010 09/02/2010 Closing: 03/31/2011 12/31/2011 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory

Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies: Moderately Satisfactory

Overall Bank Performance: Moderately Satisfactory

Overall Borrower Performance: Moderately Satisfactory

C.3 Quality at Entry and Implementation Performance Indicators Implementation

Performance Indicators QAG Assessments

(if any) Rating

Potential Problem Project at any time (Yes/No):

Yes Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

Yes Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:

Satisfactory

D. Sector and Theme Codes

Original Actual Sector Code (as % of total Bank financing) Central government administration 63 63 Health 10 10 Sub-national government administration 27 27

Theme Code (as % of total Bank financing) Health system performance 100 100 E. Bank Staff

Positions At ICR At Approval Vice President: Pamela Cox James W. Adams Country Director: Ferid Belhaj Nigel Roberts Sector Manager: Toomas Palu Fadia M. Saadah Project Team Leader: Susan Lynette Ivatts Muhammad Ali Pate ICR Team Leader: Susan Lynette Ivatts ICR Primary Author: Ian P. Morris Michelle Lee F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The Project's development objective is to improve the institutional capacity of MHMS in the areas of public expenditure management and sector performance monitoring. Such capacity development would be critical to the overall success of the Program in improving health services delivery and outcomes. Revised Project Development Objectives (as approved by original approving authority)

(a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Completion of the first rolling medium term expenditure framework

Value quantitative or Qualitative)

None

Completion of first rolling medium term sector expenditure program linking resources with priorities

Final revised version of MTEF produced

Date achieved 02/01/2008 12/31/2009 09/01/2011 Comments (incl. % achievement)

A substantial draft of the MTEF was developed in 2010, and a final MTEF was produced in 2011.

Indicator 2 : Increased proportion of health sector expenditure on primary health services and provincial health programs

Value quantitative or Qualitative)

None

Health expenditure increased from 17% on primary health services and provincial health programs

The proportion of health sector expenditure on primary health services and provincial health programs has increased from 17% (estimated base at appraisal) to 29% for FY11 (target 22%).


Source: Final MTEF 2011

Indicator 3 : Completion of Health Facility Survey Value quantitative or Qualitative)

None Health Facility Survey completed Not Applicable


The MHMS decided not to go ahead with the health facility survey.

Indicator 4 : Joint Performance Reviews Conducted Annually Value quantitative or

Less substantial reviews were undertaken in 2007

Joint Performance Reviews

Joint Annual Performance

Qualitative) and 2008 conducted annually

Reviews are conducted each year by the MHMS


JAPRs are held annually with the participation of Provincial Health Directors, program managers, DPs, non-government organizations (NGOs), and faith-based organizations (FBOs).

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Percentage of planned activities that are completed increased from 60% to 75%

Value (quantitative or Qualitative)

None Planned activities increased from 60% to 75%

This indicator was inadequately specified and not monitored quantitatively


Provincial health staff have indicated improved satisfaction with planning within budget constraints and recognized some difficulties in fully implementing planned activities

Indicator 2 : Increased community participation in health services planning at provincial level


All health planning is conducted in a centralized and top-down manner

Increased community participation in health services planning at provincial level

Community involvement has increased


There has been a gradual increase in community involvement in provincial health service planning through the operational plan development process although capacity constraints at this level need to be appreciated

Indicator 3 : Health facility survey completed and analyzed Value (quantitative or Qualitative)

None Health facility survey completed Not progressed


The MHMS decided not to go ahead with the health facility survey

Indicator 4 : Budget planning, development and accounting processes enhanced and institutionalized Value (quantitative or Qualitative)

All financial systems are fragile. At the provincial level, the system is unable

Budget planning, development

Annual provincial and national operational plans

to separate expenditures from the hospital and primary health care programs.

and accounting processes enhanced and institutionalized. This indicator was added after the mid-term review to better reflect the specifics of the project

with SIG and DP budgets are now developed routinely and accounting of expenditures on MYOB now allows separation of hospital and primary health care at the provincial level


Indicator 5 : Comprehensive sector performance indicators and monitoring system developed


Health systems are very fragile, unsystematic and often subject to delays in processing

Comprehensive M&E framework developed

M&E framework was developed as part of the new NHSP 2011-2015 and identifies agreed priority indicators for each program. It details the baseline, target, source and frequency of data to be collected


Indicator 6 : MHMS executive making decisions based on M&E reports


The MHMS executive place minimal priority on the need for timely information to inform their strategic decisions.

M&E framework developed and used by MHMS for decision making

The MHMS are making decisions based on a range of assessments including the M&E framework


The MHMS executive has been increasingly making informed evidence-based decisions based on a range of assessments including the MTEF and data generated from program and provincial operational plans

Indicator 7 : Balanced scorecards for provinces and national programs Value (quantitative or Qualitative)

None Balanced scorecards for provinces and

Not progressed

national programs Date achieved 02/01/2008 12/31/2010 12/31/2011 Comments (incl. % achievement)

Balanced scorecards were not progressed because it was not deemed a priority under the new leadership in the MHMS

Indicator 8 : Provincial planning process including medium term plan and operational planning system adopted with strong M&E framework


The Ministry has a top-down annual operational planning system without strong and timely M&E framework

Enhanced role of provinces in planning and implementing programs with a strong M&E framework

Provincial health plans with clear targets and indicators have been developed with the support from the planning team but the medium term plan process still remains fragile


This indicator was added to the results framework during the mid-term review

Indicator 9 : Increased share of provincial health plans implemented


Provincial plan are top-down and often comprised of a series of activity lists without a strategic program focus or M&E framework

Provincial plans developed, implemented and monitored for progress against agreed targets

Significant progress has been made in developing provincial plans and budgets. However, quantitative monitoring was in-adequate. Some provinces have reported difficulties with implementing all planned activities


Progress reports to be submitted in 2nd quarter of 2012 will identify the challenges faced by the provincial teams.

Indicator 10 : Self-assessment by trained provincial health directors 3 months after completion of management training


Self-assessment of management skills prior to training

Senior management staff undertake training

Formal training was limited due to high workloads, frequent overseas travel, and a significant number of alternate sponsorships

became available. No formal self-assessments were made


G. Ratings of Project Performance in ISRs

No. Date ISR Archived DO IP Actual

Disbursements (USD millions)

1 10/24/2008 Moderately Satisfactory Moderately Satisfactory 0.00 2 06/30/2009 Moderately Satisfactory Moderately Satisfactory 0.00 3 05/22/2010 Satisfactory Moderately Satisfactory 0.00 4 05/11/2011 Satisfactory Moderately Satisfactory 0.82

5 03/11/2012 Moderately Satisfactory Moderately Unsatisfactory 1.04

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made DO IP

08/26/2010 N S MS 0.20

There was considerable delay in project implementation which the Ministry acknowledged was mostly due to lack of understanding of the project arrangements within the Ministry itself and to a high turnover of senior staff in 2009. Restructuring enabled the grant closing date to be extended from March 31, 2011 to December 31, 2011.

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. The project was developed as Solomon Islands emerged from three years of serious civil conflict and three years of “emergency recovery” to a medium-term development program. At the time of appraisal, the Solomon Islands Government (SIG) was paying increased attention to medium-term strategy and planning at both the national and sectoral levels. The Ministry of Health and Medical Services (MHMS) had prepared: (a) its strategy document – The National Health Strategic Plan 2006-2010 (NHSP) with eight priority areas1; and subsequently, (b) the Health Sector Support Program (HSSP)2 a more detailed program designed to operationalize the NHSP with support of Development Partners (DPs). 2. The initial intention for HSSP was to have a pooled funding arrangement under a sector wide approach (SWAp). Significant staff weeks were committed to shaping this design with SIG, AusAID and other DPs (refer Annex 4b). However, after two years of project preparation, agreement could not be reached on fiduciary assessments and related requirements so HSSP was designed as an over-arching program with complementary financing of a single sector program led by the Government and in coordination with several DPs. Challenges for getting the health sector back onto a sustainable medium-term development path included that DPs financed about 50 percent of both total and recurrent sector expenditures, and concerns about the technical quality of health services and capacity of the health system to deliver services effectively and efficiently. 3. Several aspects of HSSP required further development, specifically, the sector’s overall resource allocation picture was not clear. The initial partial Medium Term Expenditure Framework (MTEF) developed as part of preparation of the HSSP did not reflect the full sector envelope (government plus donors) and the government had yet to revise expenditure priorities to allocate health resources in alignment with its strategy. In addition, the possibilities for efficiency gains, equity concerns and the sustainability of sector financing had yet to be fully explored. There was an unclear division of responsibility between provinces and national programs. Finally, the monitoring and evaluation (M&E) framework needed to identify a core set of key performance indicators to measure the effectiveness, efficiency, equity, and sustainability dimensions of sector performance. 4. This context provided the rationale for Bank support through a Technical Assistance (TA) Grant to improve the capacity of the MHMS in the selected areas of public expenditure management, de-concentration of services to provincial managers, sector performance monitoring, and management training. It was agreed that such capacity development would be critical to the overall success of the HSSP and that

1The Solomon Islands National Health Strategic areas were: (i) people focus; (ii) public health programs; (iii) malaria; (iv) common childhood diseases; (v) non-communicable diseases; (vi) HIV/AIDS and STIs; (vii) family planning and reproductive health; and (viii) health systems strengthening. 2The HSSP focus areas were: (i) community focus; (ii) priority health programs; and (iii) health system strengthening.

2

assistance with the development of key building blocks for the medium-term development of the health sector was critical. Furthermore, the Bank’s sectoral expertise and experience in sector wide operations and fragile and post-conflict countries were considered of value by the client and the key DPs. 1.2 Original Project Development Objectives (PDO) and Key Indicators 5. The PDO was to improve the institutional capacity of MHMS in the areas of public expenditure management and sector performance monitoring. This was critical to the overall success of the HSSP in improving health services delivery and outcomes. The PDO indicators were: (i) completion of the first rolling MTEF; (ii) increase in the proportion of health sector budget expended at provincial level and on primary health care services; (iii) completion of a health facility survey and use of the results to improve health sector planning; and (iv) quality, participatory annual joint sector reviews conducted on schedule. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 6. No revisions were made to the PDO during the mid-term review. Two intermediate indicators were added to the results framework to better monitor PDO achievement. 1.4 Main Beneficiaries 7. The primary target groups were stakeholders at both the national and provincial level. Key beneficiaries at the national level were to be MHMS executive, finance and planning staff through enhancing their capacity in public expenditure management and monitoring of the health sector. Provincial level managers would be better equipped to plan ahead and to work within a more predictable resource envelope to more effectively allocate and manage resources to improve provincial level health outcomes. 8. Secondary beneficiaries, in the form of health service users, were expected to benefit through the positive impact of the project on the HSSP with its focus on: (i) communities; (ii) reorientation of public health resources away from the center to the provinces; and (iii) emphasis on improving the equity of health outcomes. 1.5 Original Components 9. Component 1: Public Expenditure Management (US$545,000) aimed to: (a) support strengthening of MHMS (national and provincial) health planning and budgeting systems and procedures; and (b) build on work done during preparation to ensure priorities were set within a known resource envelope and linked to expected results. The focus was on: (i) completing and updating the health sector MTEF annually; (ii) strengthening provincial planning and budgeting procedures and monitoring; (iii) improving understanding of the costs and efficiency of health facilities; and (iv) increasing community and household engagement in health care.

3

10. Component 2: Sector Performance Monitoring (US$382,000) aimed to support the MHMS to develop the M&E framework for the HSSP covering the key dimensions of sector performance (effectiveness, quality, efficiency, equity, and sustainability) and integrate M&E into sector management. The component aimed to enable the MHMS to better track achievement of NHSP objectives, and to provide feedback through the annual budget planning and review processes so adjustments in resource allocation were linked to priorities. This would enable improved sector policy and management decisions based on evidence. This information would facilitate the annual joint sector (HSSP) review process and meetings of MHMS and DPs. 11. Component 3: Training and Capacity Building (US$523,000) aimed to strengthen the management capacity of MHMS, including provincial managers to be more effective in work program planning and execution. 1.6 Revised Components 12. There were no formal revisions to the main Project components. 1.7 Other significant changes 13. Producing the new NHSP 2011-2015 (new NHSP) was a new activity which was achieved by combining existing elements from the project components. It was fully consistent with the objective of developing capacity to ensure expenditure priorities reflect desired efforts to improve health services delivery. 14. Closing Date Extensions. The closing date was formally extended for nineteen months in August 2010 from March 31, 2011 to December 31, 2011 primarily due to the significantly delayed start-up of the project. This enabled key project activities to be implemented and the PDO to be more fully achieved.

2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 15. The overall quality at entry is rated as Moderately Satisfactory based on the following assessment: (i) the soundness of the background analysis - initial project preparation activities

focused on trying to forge an AusAID-International Development Association (IDA) pooled financing model for sector support with all DPs supporting and signing a Joint Partnership Agreement with MHMS. The joint donors, particularly the Bank and AusAID, supported a strong preparation effort to develop the key building blocks for pooled financing under a SWAp3. While initial analysis had suggested the

3 Relevant project preparation activities included: (i) a health sector strategy note; (ii) a qualitative study of health care decision making in the Solomon islands; (iii) an initial but partial health MTEF; (iv) a health expenditure review; (v) a

4

key building blocks for a SWAp were present (i.e. sound sector priorities; SIG ownership; agreement on desirability of pooled financing; and initial relatively positive fiduciary and procurement assessments) experience gained during preparation indicated it was not possible to gain consensus within SIG or with DPs on how to proceed on all fronts. Specifically, project preparation had taken two years; leadership within the MHMS had changed and was less committed to pooled funding; the central agencies became concerned about their capacity to manage the financial/fiduciary aspects of the project; and there was inadequate DP support for procurement and financial management processes acceptable to the Bank. A pooled fund SWAp model for the project was finally rejected in favor of a TA Loan/Grant (TAL), where SIG and key DPs adopted a model of complementary (parallel) financing of a single sector program led by SIG. This was an appropriate decision and fully consistent with the lessons learnt from the previous Bank-supported SIG Health Sector Development Project and other sector projects throughout the region. The main constraints to improving the health sector lay in resource allocation, management capacity, and a lack of effective performance monitoring. The project was designed to specifically address these gaps in the existing HSSP;

(ii) the suitability of project design - was appropriate to meet sector needs as identified

during project preparation. The objectives and components of the project were clearly identified by MHMS and DPs as strategic building blocks for both health sector development and to enable a pooled SWAp to be developed in the future – a continuing objective of both SIG and DPs. Importantly, SIG remained committed to their leadership of one HSSP program supported by DPs and the project design supported this. The focus of the project was on the development of core government systems (particularly provincial budgeting, sector expenditure planning and budgeting, and an M&E framework) that would assist HSSP implementation, and make a full SWAp feasible. A SWAp, all parties agreed, held out the prospects of reducing sector transaction costs while still driving for immediate results, particularly reallocation of resources towards expressed priorities. The final design recognized that DPs needed to learn to work collectively in support of one program and MHMS would also need to lead a collective effort in support of the HSSP. The final TAL design significantly reduced the range of Bank safeguard policies involved compared to the initial approach (e.g. environmental and land acquisition/involuntary resettlement policies) and reduced the scale of financial expenditure and procurement oversight;

(iii) the adequacy of participatory approaches and government commitment- the

joint World Bank and AusAID preparation and design missions facilitated extensive consultations with ministers, senior SIG officials, national and provincial MHMS staff, key health sector stakeholders, and DPs. There was strong participation of MHMS program directors jointly with provincial directors and some of their staff. Church agencies providing health services and health-related Non-Government Organizations (NGOs) also participated in project planning sessions. These discussions confirmed ongoing support from SIG and key stakeholders for the HSSP

report on annual planning and budgeting system; (vi) a review of hospital services and their management; and (vii) a review of pharmaceutical services, procurement and distribution arrangements

5

framework. The MHMS increasingly recognized the need to work more closely with provinces and NGOs – a positive achievement of the preparation process; and

(iv) the approach to risk and mitigation - the overall project risk was rated as high given the country and sector contexts. The assessments were realistic ratings of the risks and the identified mitigation measures appropriate.

2.2 Implementation 16. Overall implementation progress is rated Moderately Satisfactory as SIG was able to make reasonable progress implementing most activities by the end of the Project (as outlined in more detail in Annex 2). This was despite major changes in the MHMS senior management team during the project lifecycle, and the ongoing capacity challenges of a small workforce in a resource constrained environment. 17. Project leadership and staffing had a significant impact on implementation. During the first year of the project senior MHMS management were in a state of uncertainty about their future and this reduced focus on the project in the initial start-up phase. In the second quarter of 2009, a new Health Minister, Permanent Secretary, and an almost completely new MHMS senior management ‘executive’ team were appointed. The new executive was initially unclear about the project, its objectives, and implementation arrangements and how this worked with the HSSP and a major new AusAID funded parallel malaria project. Once the arrangements were clarified the new MHMS executive endorsed the project and its objectives but also requested support for: (i) management mentoring; and (ii) developing the new NHSP 2011-2015. The Bank agreed to support this as it was fully consistent with project objectives. MHMS engaged a senior health policy advisor to support: (a) a participatory planning approach for the new NHSP; (b) development of an M&E framework for the sector; and (c) on-demand management mentoring advice to the executive. This, together with the work of the MHMS-appointed financial management specialist under the project, proved decisive and reenergized implementation. This work linked expenditure management and M&E to revised health sector priorities through a participatory process for strategic planning involving provincial and program managers, NGO partners and DPs. 18. Fiduciary capacity was the other major factor influencing project implementation; this is outlined in more detail in section 2.4.

19. Beyond immediate project fiduciary concerns, some technical aspects of provincial budget structures (chart of accounts) and processes, controlled by the central agencies were outside the control of MHMS, and proved more difficult than anticipated at appraisal. This made the separation of hospital and primary health care programs at the provincial level more difficult, adversely affecting some aspects of the desired budget reporting and provincial planning processes within MHMS. This remains a work in progress, but capacity developed under the project will enable MHMS to adjust to and reap the gains of these wider reforms when implemented. 20. The Mid Term Review reconfirmed the PDO and expanded the results framework to better monitor on-going activities. The MHMS executive indicated it had come to understand better the effective use of TA. TORs and Bank dialogue with the executive

6

emphasized the desirability of TA to undertake “process” rather than “task” consulting. Using this ‘learning by doing’ process approach enabled the staff of MHMS to undertake the work themselves within strategic frameworks/ government systems. The new NHSP support (including the M&E design) and financial management assistance used this approach and it is likely this will improve sustainability of the project post completion. Key staff sent on training, including degree level training for one officer, will also assist in this regard, although continued staff mobility within MHMS remains a problem. 2.3 Monitoring and Evaluation Design, Implementation and Utilization M&E Design 21. Project monitoring was designed to be undertaken within the M&E framework of the overall health sector which was to be developed further under the project. This was developed in draft only late in the project as part of the new NHSP and is now being refined after discussion at the September 2011 MHMS/DP sector meeting. Major efforts are now underway to reinvigorate the health management information system (HMIS) to supply the desired monitoring data. The MHMS decided not to seek support to design and implement a balanced score card for provincial and national health programs (a project activity) but rather to focus on the overall HSSP M&E framework. 22. Some project indicators relied on qualitative information which were not well specified and/or easily collected (e.g. increased community participation in planning at provincial levels, executive making decisions based on M&E reports, percent of provincial health plans implemented) and others turned out to be more difficult to measure because of chart of accounts issues (e.g. separation of hospitals from primary health care) which required considerable reworking of accounts which was achieved mid-project with considerable effort. M&E Implementation 23. The project envisioned that participatory joint annual reviews (with SIG and DPs) of the HSSP would take place to monitor sector performance. This was an agreed “building block” for the SWAp. To reduce transaction costs, it was agreed that MHMS would not prepare separate M&E systems or reports for individual DPs. The Joint Annual Performance Reviews would discuss the progress of the health system and health indicators and their quality. Notwithstanding the delays in finalization of the M&E framework, the existing (partial) framework was used for the ongoing sector monitoring. The biggest constraint was the availability of data from the HMIS which faced long standing technical constraints processing health facility data. This is now being fixed as proposed by the M&E framework and it is expected that real time data will become available in July 2012. A Demographic and Health Survey 2007 (DHS) and analytical work undertaken by the Bank on health financing options4 drawing on the Household Income and Expenditure Survey provided key sector performance benchmarks and helped frame dialogue on the new NHSP and M&E framework.

4 World Bank, (2010). Solomon Islands Health Financing Options

7

24. The project results framework did not work as originally planned. The agreed benchmarks for a number of project indicators were not able to be monitored as intended. Nevertheless, the MHMS executive did monitor key aspects of the project and MHMS staff willingly undertook dialogue with the World Bank team and other DPs on the sector’s progress, including that of the project. A key project M&E process adopted was by reporting through review missions and documentation within relevant aides memoire. Bank aides memoire became the “de facto” DP report on sector performance and Bank project progress, and was used as such by the MHMS executive and DPs. M&E Utilization 25. The M&E draft framework developed for the new NHSP was only developed in late 2011 just prior to project completion. Nevertheless, the MHMS undertook annual sector reviews based on available data from project inception. Presentations by program managers and provincial managers at the Joint Annual Performance Reviews indicated they had at times, significant high quality data not reflected in the formal HMIS data bases, that: (a) underpinned their decision making on service priorities; and (b) informed dialogue between MHMS and DPs. 26. Available HMIS data, the World Bank Health Financing Options Note, the DHS 2007, and a situational analysis prepared by MHMS and DP/MHMS dialogue underpinned the decision making on the new NHSP and many HSSP decisions. The voice given to provincial managers through their participation in these processes also led to demands for change of national program priorities. This process also made decision making more transparent and contestable. Provincial and national program managers believe it also positively influenced annual planning and budgeting processes, an intended outcome of the project. While it is hard to demonstrate the extent of all the program priority changes as yet, the proportion of expenditures on primary health services increased from 17 percent to 29 percent (target 22 percent) and the portion of provincial spending is planned to increase while national programs decrease. The new NHSP outlines both significant reforms on program priorities and of organizational and management priorities. Increased commitment by MHMS to evidence-informed policy making, reinforced and buttressed by a range of project activities and agreements, is a significant outcome of the project. The MHMS executive fully recognizes this is both a new approach and that the processes will need to continue to be reinforced by DPs under the HSSP. 2.4 Safeguard and Fiduciary Compliance Procurement 27. This was rated a “high risk” at appraisal and an action program was agreed, most of which were upfront actions (procurement plan, high levels of prior review, on-going procurement advisory support, and a procurement manual). Provision was also made for upfront and on-going procurement training. Nevertheless, significant procurement delays emerged from the outset of the project and overall procurement performance was rated as “moderately unsatisfactory”. 28. Limited experience and capacity in a very small team was the major constraint within the procurement unit (the two staff involved had changed since the previous Bank

8

project). Management understanding and direction on the procurement processes, particularly in the early years of implementation, was limited. Continued staff mobility within the procurement unit and within the executive exacerbated these capacity constraints. Although MHMS had agreed to recruit an international procurement advisor for the period of the project (financed by AusAID under HSSP), MHMS was not able to attract and retain a suitably qualified specialist for the required timeframe. When a procurement advisor was appointed nine months after effectiveness, there was a misunderstanding about the advisor’s role in assisting the MHMS with Bank-financed project procurement. This significantly constrained the ability of MHMS to complete on a timely basis the initial procurement actions, including for two key advisors. The Bank subsequently responded to a MHMS request for assistance by initiating more intensive: (a) implementation support, including for procurement; and (b) joint dialogue with DPs and MHMS on implementation and performance monitoring. The Bank also reached a new agreement with MHMS and AusAID on continued procurement advisory support, and one member of the executive was made responsible for procurement to prevent continued procurement delays. Nonetheless, procurement delays throughout the project were significant, resulting in the initial two major TA positions being filled 16 months after project effectiveness, and the first disbursement occurring 24 months after effectiveness (although some expenditure had been pre-financed by MHMS). Financial Management and Auditing 29. Financial management of project accounts was rated as “moderately unsatisfactory” or “unsatisfactory” throughout its life, with incomplete or typically misplaced filing of paperwork on project files. Again, limited capacity in a small team was the main constraint. Project accounts were audited by the Government’s Auditor General. The audit for the period ending 31 December 2010 was over nine months late and was qualified by exception; the qualification did not identify a material misstatement of the financial statements. The final audit is due in June 2012. Interim Financial Reports were submitted late and then often needed further work to satisfy World Bank financial management requirements. Withdrawal applications were often delayed even though expenditures had been incurred (the government pre-financed project expenditures). A key staff member took long-term study leave and this further delayed withdrawal applications at one point. MHMS appointed a part time financial management specialist to assist with broader sectoral budget and accounting work under the project, and this also helped to improve the project disbursement process and capacity through training and technical support. Financial management support from the Bank’s fiduciary implementation support team was limited, and not well co-ordinated with broader Bank implementation support missions. During the last quarter of the project, the MHMS hired a new experienced Finance Controller to buttress the MHMS finance team; this has been an important step to improving management capacity and oversight in this core area. 30. The project did not trigger any safeguards policies. 2.5 Post-completion Operation/Next Phase 31. The Bank, MHMS and AusAID together with other DPs are actively discussing an ongoing work program to support the health sector. There is a consensus and commitment to improving the implementation and results focus of the HSSP with an

9

emphasis on how best to implement the strategic directions of the new NHSP. MHMS has committed to a work program to further improve sector governance and organizational reform. This includes coordination of technical cooperation, finalization of a role delineation policy to define the range of services across the country, improvements to financial and procurement management, including at the provincial level, and strengthening sector M&E. 32. In the immediate future the World Bank team plans to support MHMS through a Bank executed Trust Fund arrangement for the Pacific. Initially, this will finance further health expenditure analysis and assist with updating the MTEF, including setting out and tracking provincial funding allocations and expenditure in line with the directions set out in the new NHSP. Other areas, including health workforce issues, costing and efficiency analysis of national and provincial hospitals, options for public-private partnerships, and a health services plan for Honiara are under discussion. There will also be continued support to improve budget planning, preparation and execution at provincial and national levels. The Bank is currently recruiting an experienced health economist to be based in the Bank’s Honiara office. Although the position will have regional responsibilities, this will be a significant additional commitment by the Bank to sustain its work on the health sector in the Solomon Islands. 33. These efforts are fully consistent with the proposed focus on service delivery under the new Country Partnership Strategy being developed by the WB and SIG. The Bank and other DPs are also assisting SIG with a financial management reform program. This will assist with improving the relationship between central agencies and sectors and potentially set the basis for a health sector focus under a possible next phase of budget support provided under a SIG/WB Development Policy Grant. There may also be scope with these reforms, to explore a future health specific grant using the WB ‘Programming for Results’ instrument focused on improving service delivery at the provincial level.

3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 34. The PDO: to improve the institutional capacity of MHMS in the areas of public expenditure management and sector performance monitoring continue to be relevant to both current sector and country priorities and remain consistent with the Bank’s Interim Strategy Note FY 2010-2012 (ISN) for Solomon Islands – a strategic pillar of which includes: “supporting improved public administration and management.” The new NHSP 2011-2015, developed with project support, reaffirms the need to focus on: (a) health priorities based on evidence; (b) further de-concentration of health program expenditures and management, including provincial planning and budgeting, to get resources to where they are needed most; (c) overall financial management; (d) M&E; and (e) continued updates of the MTEF to underpin improved focus on expenditure priorities and efficiencies. The MHMS Corporate Plan 2011-2015 reiterates these points. This policy focus is fully consistent with the PDO. Project components remained relevant to the needs of the MHMS and targeted clearly identified priorities in the overarching HSSP. Sector stakeholders supported HSSP objectives, particularly as refined and included in the new NHSP.

10

35. The project design, including its framing within the context of joint efforts to establish a health sector SWAp, remained consistent with Bank priorities through both the World Bank’s Regional Engagement Framework FY 2006-2009 (which highlighted the importance of improving the effectiveness of public expenditure in the social sectors through strategic partnerships with key DPs); and in the ISN, which recognizes that changes in administrative arrangements and management of public resources are essential to improved service delivery. DPs, as outlined in section 7b, also strongly supported the focus and continued relevance of the key objectives and activities of the project. 3.2 Achievement of Project Development Objectives ICR Rating: Achievement of PDO is Moderately Satisfactory.

36. Project Outcome Indicator 1: Completion of Medium Term Expenditure Framework. A substantial draft was completed in 2010 and a final initial MTEF was produced by MHMS in 2011. Although the MTEF was significantly delayed it is now completed and informing dialogue on: (i) strategic resource allocation priorities; and (ii) incremental recurrent costs of sector investments and hence the fiscal sustainability of the program. The first annual update is planned for third quarter 2012.The MHMS recognizes this is the only source of detailed “whole of sector” health expenditure and its financing as the development budget processes are weak and do not provide complete pictures of sector expenditures. Thus the MTEF also greatly assists policy dialogue and dramatically improves the transparency of health expenditures and their financing. The central agencies are now beginning to develop a “whole of government” MTEF process so there is increased reason to believe the MTEF process will become institutionalized. 37. Project Outcome Indicator 2: Increase from 17%, the proportion of health sector expenditure on primary health services and provincial health programs. The proportion of health sector expenditure on primary health services and provincial health programs has increased from 17% (estimated base at appraisal) to 29% for FY11 (target 22%). (Source: Final MTEF 2011) 38. Project Outcome Indicator 3: Completion of Health Facility Survey. This survey was not undertaken as planned (see Annex 2). The MHMS delayed its initiation due to: (i) concerns about sequencing of other work and their capacity constraints; (ii) overlap with another AusAID study of infrastructure needs; and (iii) proposals from UNICEF to undertake a costing study using the tool Marginal Budgeting for Bottlenecks (later rejected by MHMS). Finally, MHMS decided, as a result of policy dialogue, to proceed with hospital and facility costing/survey work in the last year of the project. However, delayed procurement actions meant this work was not commissioned on a timely basis. Confirming the on-going relevance of this activity the MHMS is pursing options with DPs to finance a similar survey(s) now (see 2.5 above). 39. Project Outcome Indicator 4: Joint Annual Performance Reviews. Joint Annual Sector Performance Reviews are now held annually with the participation of Provincial Health Directors, program managers, DPs, NGOs, and faith-based organizations. Although the quality of the review processes and data availability varied considerably over the project period, the trend was in the right direction with the

11

2011review enabling MHMS and DPs to jointly reflect on the gains of the health sector and to identify specific areas for improvement (also supported by the first MTEF). The operationalization of the HMIS will also greatly improve the quality of this process in the future.

40. Component 1: Public Expenditure Management. This component focused on improving sector resource allocation planning and expenditure management and efforts to de-concentrate to the provincial level consistent with sector priorities. The aim was for: (i) the percentage of planned activities completed increased from 60% to 75% - this indicator was inadequately specified and not monitored quantitatively. However provincial health staff have indicated improved satisfaction with planning within budget constraints and recognized some difficulties in fully implementing planned activities; (ii) increased community participation in provincial health services planning - the level of beneficiary/stakeholder participation in the development of both the new NHSP and of provincial operational plans has been significant. Efforts intended to extend this down to facility level has been problematic and remain a challenge – constrained by over stretched provincial managers. Nevertheless, the achievements on stakeholder participation should not be underestimated, including getting Provincial Premiers and community groups involved in planning and monitoring discussions; (iii) the health facility survey to be completed and analyzed - this did not proceed as discussed in paragraph 38 although alternative data sources (including the health financing note) provided some important insights; and (iv) (added at mid-term review) budget planning, development and accounting processes enhanced and institutionalized - annual provincial and national operational plans with SIG and DP budgets are now developed routinely, and accounting of expenditures on MYOB now allows separation of hospital and primary health care at the provincial level. 41. Component 2: Sector Performance Monitoring. This component focused on improving the sector performance monitoring system. To this end the component envisaged: (i) development of a comprehensive M&E framework - this was developed as part of the new NHSP and identifies agreed priority indicators for key programs and for organizational and management reforms and details the baseline, target, source and frequency of data to be collected. Financial management, macroeconomic and microeconomic indicators are detailed in operational plans and the MTEF; (ii) MHMS executive would make decisions based on M&E reports - the MHMS executive has been increasingly making informed evidence-based decisions based on a range of assessments including the MTEF and monitoring data generated from program and provincial operational plans. The new NHSP was based on quantitative and qualitative data generated from a situational assessment and from available M&E data. The evidence based process will be improved when the HMIS and the M&E framework is fully operational in 2012. Evaluative work, including the MTEF and the Bank generated Health Financing Options Paper, also informed decision making. The NHSP Planning and Evaluation Advisor provided strategic mentoring advice and guidance to the MHMS executive team on the critical importance of M&E; (iii) the use of balanced scorecards for provinces and national programs - these were not progressed as it was not deemed a priority under the new leadership of the MHMS; and (iv) (added at mid-term review) provincial planning process including medium term plan and operational planning system adopted with strong M&E framework - provincial health plans with clear targets

12

and indicators have been developed with support from the planning team but the medium term plan process still remains fragile. 42. Component 3: Training and Capacity Building. This component focused on improving the human capacity in the health sector. It was envisaged that this would be monitored by: (i) self-assessment by trained provincial health directors 3 months after completion of management training - formal training was limited due to high workloads, frequent overseas travel, and the availability of a significant number of alternate sponsorships. No formal self-assessments were made although those attending the Bank sponsored Health Systems Flagship course indicated it underpinned their sense of the usefulness of the MTEF (because it set out total expenditures and how they were financed), setting priorities, M&E and the importance of planning within a budget constraint; and (ii) the increased share of provincial health plans implemented - significant progress has been made in developing provincial plans and budgets, however quantitative monitoring was inadequate. Some provinces have reported difficulties with implementing all planned activities particularly when they were developed without reference to budget constraints. Progress reports to be submitted in second quarter of 2012 will better identify the challenges faced by the provincial teams. 43. While performance is mixed, outputs under these components have contributed towards achieving the PDO. The budgeting system, particularly at provincial level, is now more firmly established and disaggregated to show separate primary health care and hospital expenditures. Managers now plan within a relatively stable and known resource envelope and budget execution is monitored on a regular basis. On a number of occasions, project outputs helped MHMS to argue with some success an evidence-based case when SIG proposed budget reductions due to broader fiscal issues. MHMS has noted that it could not have made these achievements without the project. 3.3 Efficiency 44. The project appraisal did not undertake a cost-benefit or other formal analysis of cost-effectiveness. On this basis, the ICR has not undertaken a cost benefit analysis. Nevertheless, the appraisal did assert that the project would: (a) assist to improve allocative efficiency of health expenditure by aligning allocations to priority needs; (b) improve the technical efficiency of health expenditures by providing the tools and improved information; and (c) improve equity in resource use by providing information to form the basis of reallocating resources to improve regional equality. 45. The project has been partially successful in this (increased resources were allocated to primary health care and to provinces) and the new NHSP strongly commits the MHMS to improved allocative efficiency. The MTEF has demonstrated the scale of the challenge to redirect both domestic and DP sector financing towards agreed priorities. The MTEF, for the first time documents the “whole of sector” expenditure program and how it is financed. Sector analytical work undertaken by the Bank has shown that access to health is remarkably equal in Solomon Islands compared to most countries with a similar income. However, work at the provincial level shows that on a per capita basis, health resources are unequally distributed. Analytical work generated by the project was

13

used by MHMS to reallocate provincial grants on a more equitable basis. The challenge will be to reallocate staffing according to need. 46. The health facility work to estimate unit costs and service indicators would have added to the richness of the information available to inform decision making. However, this activity was not progressed as the new NHSP was given precedence by MHMS. It is clear that MHMS and all DPs now see the benefit of this costing information and some of this will be included under future support. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 47. The overall rating is based on the continued relevance of the objectives and the extent to which they were achieved. Project objectives remain highly relevant to the current strategic priorities of Solomon Islands and the World Bank. MHMS used the project to set a sound foundation for ongoing sector development through (a) the new NHSP which defines MHMS’ overall aims and M&E arrangements for the next five years; (b) completion of the first MTEF; and (c) strengthening of the provincial planning and budgeting. These elements have helped identify and improve allocative efficiency of health expenditures (in a situation where details of total “whole of sector” expenditures were not known) and increased the overall policy dialogue between SIG and DPs on core sector priority issues. 48. The specific outcomes and outputs of the project are documented in Annex 2. Despite delays the MTEF was finally completed and will be continued on a rolling basis. It will continue to support planning within realistic resource envelops at national and provincial level. The new NHSP with its strategic priorities and M&E framework has very strong ownership within MHMS and with SIG more generally. Provincial planning and budgeting capacity has been developed considerably – including finding a way to meaningfully disaggregate expenditures between primary health and hospital care at the provincial level. Provincial primary health care expenditures have reached 29 percent of expenditures up from 17 percent and over the target of 22 percent. Formal joint MHMS/DP annual performance reviews are now conducted routinely. The MHMS and DPs will, in the future be able to use the M&E framework, supported by the improved HMIS data, about to come on line. Considerable training, including planning workshops with a “learning by doing” approach as with the new NHSP and the development of budget and accounting systems were very successful. Planning and budget execution have been strengthened – albeit from a very low base. 49. Given the thin capacity in this small post conflict country these achievements are considerable. Nonetheless, it is recognized that more could have been done to prevent and manage delays. MHMS recognizes they need additional support moving forward and this will be provided through new World Bank and other DPs assistance.

14

3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 50. Bank analytical work showed access to health services by the poor was quite equitable compared to other low income country. Work under the project led to important adjustments on provincial equity grounds in the distribution of national grants to provinces as a result of work undertaken under the project. The new NHSP has a reasonably strong gender focus with its increased focus on mother and child health and family planning. Disaggregated data by sex is a key feature of the M&E framework and MHMS actively supports training opportunities for women. The process of producing the new NHSP involved provincial premiers, community and NGO stakeholders which was a noted improvement on previous efforts. Annual provincial planning is increasingly involving communities although this needs further strengthening – including additional support to overstretched provincial directors. (b) Institutional Change/Strengthening 51. Strengthening the capacity of the MHMS to effectively manage and monitor health expenditure and outcomes is the main aspect of the PDO. Important progress has been made on this but all parties recognize that the institutional development and strengthening has been moderately satisfactory as it takes a long time in fragile states with weak systems to develop capacity. It has to be recognized that the existing inadequate development budget processes mean sectors (and SIG) do not have an accurate picture of resources planned to be spent and how they are financed. The MTEF documented this for the first time. Using a “process” consultancy approach for the development of the new NHSP and with the financial management work in the second phase of this work program clearly helped build capacity through its “learning by doing” approach. As a result ownership of MHMS has clearly improved as a consequence. The new NHSP sets out a significant organizational and management reform agenda which is forming the basis for ongoing dialogue and support from DPs. (c) Other Unintended Outcomes and Impacts (positive or negative) 52. Not Applicable 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 53. Not Applicable

4. Assessment of Risk to Development Outcome Rating: Moderate 54. The risk that development outcomes will not be sustained and deepened in the future is assessed as moderate. There are real countervailing considerations in this assessment. On the negative side there are risks that: (a) leadership of the MHMS will change resulting in a reduced commitment to the project (particularly decentralization) objectives; (b) the strength of the national program directors to resist de-concentration of

15

health programs to provincial managers will prevail; and (c) that the broader government does not continue with its financial management reform agenda or does not maintain a focus on financial and planning reforms. 55. On the positive side there is considerable evidence that most of the reforms will be adequately sustained. First, the core of the reforms supported by the project was clearly identified in the new NHSP. Second, as discussed in section 2.5, the MHMS has requested ongoing support from the Bank (in cooperation/partnership with AusAID) to support the ongoing implementation and deepening of these reforms. MHMS and other DPs (including AusAID) are in the process of agreeing new financing support for the health sector and while these are not yet finalized the ongoing dialogue is premised on the basis that the key building blocks for further development of the HSSP will require on-going work in these areas, including decentralization of health programs (provincial planning, budgeting and accounting of funds for service delivery at the provincial level). Third, decentralization is a strong theme of the current government and in many ways health is showing the way forward for other sectors. Finally, the central agencies have recently decided: (a) to formally create and implement a chart of accounts which will allow for significant decentralization of the financial management of service delivery including health; and (b) to introduce a “whole of government” MTEF process. For these reasons, notwithstanding the challenges that remain, the risks to development outcomes in the future are assessed as moderate.

5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 56. The preparation was supported by a strong well qualified Bank team. A situational assessment of the health sector was prepared and this was supported by initial information from a health module in the Household Income and Expenditure Survey undertaken by SIG with support of the preparation team. The final project design, as appraised, was focused on developing strategic building blocks necessary to: (a) increasingly devolve management of health services delivery to provincial managers within the MHMS; and (b) implement a results focused program of support for the HSSP jointly with DPs and government/MHMS in the future. This was a sound approach with continued strategic and policy relevance. The final project outcomes suggest the institutional development targets set for the project were ambitious but focused on the right issues, and as discussed, require continued support. The alternate approach of going immediately to a pooled funding SWAp (considered during preparation) without adequate building blocks in place, and a strong consensus with DPs on the approach, clearly would not have worked. However the work done under the project now makes this more feasible. In this regard DPs are now working with MHMS to work through the compacts needed to enable increased flexibility in the manner DP funding and TA is used to support service delivery programs. The government’s continued constrained fiscal position makes the focus of financial management reform and the MTEF appropriate. The MTEF focus on: (i) highlighting the misalignment of expenditures between HSSP

16

objectives; (ii) gaps in support for priority programs (by DPs and government); and (iii) the incremental recurrent costs arising from policy and investment decisions, underline that the project focus was relevant technically, financially and economically. 57. The risk assessments for the project at appraisal were accurate and reflected the considerable issues expected to be faced during implementation. Weak fiduciary arrangements were identified up front and a wide range of risk mitigations measures were adopted, including upfront actions. Nevertheless, reliance on another DP to maintain a suitable procurement advisor to support the MHMS procurement unit which was assessed as weak proved problematic. Financial management capacity, particularly maintaining project accounts was also weak and the mitigation measures also underestimated the problems to be faced by the project. This, while very important, relates to project accounts only and should not be interpreted to mean that the strengthening of the financial management of the MHMS and of provincial divisions overall were not well designed and prepared. The M&E system for the project was reasonably designed but was seen as a small part of the overall efforts until its relevance was highlighted as part of the development of the new NHSP. (b) Quality of Supervision Rating: Moderately Satisfactory 58. The quantity and quality of Bank supervision increased over the life of the project. During the first year, the Bank did not provide the level of implementation support that is needed in a limited capacity environment such as the Solomon Islands. Infrequent initial supervision (both technical and fiduciary) meant that the full extent of capacity and implementation constraints, particularly of procurement, was only fully realized one or more years into project implementation. This was exacerbated by turn-over of almost the entire senior management team of MHMS responsible for Project activities. From early 2010 supervision intensity increased and additional consultant support was added to the team to work with MHMS management and the TA engaged under the project, particularly for the initial provincial budgeting and planning support and the MTEF work. Increased Bank presence in the Solomon Islands and ongoing communication with the MHMS executive and DPs has meant the Bank managed to stay more fully engaged and increasingly responsive to implementation constraints over the final two years of project implementation. By participating in all main sector meetings with MHMS and DPs(at least three missions per year), the Bank has been able to engage effectively in a more coordinated support to the sector assisted by a period of strong leadership of the MHMS. It should be noted here that Bank supervision had to judiciously mix focus on the overall HSSP (working with MHMS and other DPs on the building blocks of the HSSP- an intended outcome of the project) and the specific fiduciary issues and institutional developments which were needed to manage the more narrow implementation aspects of the Bank project. Bank aides memoire were seen as very important monitoring documents for the overall HSSP by both MHMS and DPs. 59. It is evident that, in the final analysis, there was inadequate FM and Procurement support by Bank staff. Regional procurement and FM staff saw this project as relatively small and it was only on rare occasions that their supervision missions were coordinated with the main implementation support missions. This was seen as less important for

17

procurement because there was a very high level of prior review. There were only two formal and one informal FM reviews, all from March 2011. Thus very little was done to establish the basic system for project accounts at the outset, and as there was little or no expenditure (in part caused by implementation delays and subsequently limited capacity to prepare withdrawal applications), FM did not give the project a high priority. The first withdrawal was only made in mid 2010. In the final analysis the delays were primarily due to capacity constraints. The core problems associated with procurement and FM derived from these capacity constraints not from any desire to avoid the fiduciary protocols of the Bank. The technical task team provided considerable implementation support during missions and through email/phone to help compensate for this constraint. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 60. The overall rating of Bank performance is moderately satisfactory reflecting the two dimensions: quality at entry, and supervision/implementation support. While initial technical and fiduciary implementation support was inadequate for the limited capacity environment, the Bank task team increased its engagement as implementation issues emerged. From late 2009 this involved at least three missions per year as part of broader sectoral meetings, plus substantial email, phone and video conference communications. Implementation support arrangements facilitated the transfer of global knowledge through policy dialogue on the HSSP, the development of the new NHSP, and the establishment of core building blocks for development of the HSSP. Overall, this made a significant contribution to the health sector policy and management focus. 5.2 Borrower Performance Government/Implementing Agency Performance (on basis that government and implementation agency are inseparable for practical purposes). Rating: Moderately Satisfactory 61. At the time of writing the final ISR (March 2012) the 2010 audit was overdue by nine months, the resubmission of a number of IFRs were outstanding, and only 66 percent of the grant had been disbursed. For this reason the rating for Implementation Progress in the final ISR was Moderately Unsatisfactory. However, by the end of the disbursement period (April 2012) final project expenditure was 82 percent. The outstanding IFRs have now been submitted and accepted, and the audit completed qualification by exception; the qualification did not identify a material misstatement of the financial statements. While compliance with these project specific fiduciary requirements was inadequate, it is recognized that the small fiduciary team within MHMS was stretched managing their own Government budget and other much larger DP funding arrangements. The World Bank reporting and management requirements were an additional layer of work for a comparatively small funding allocation. However, when looking at project performance more broadly, the MHMS maintained a significant commitment to the objectives of the Project and the HSSP, and made good overall progress with its key project outcomes: the new NHSP with its M&E framework, the MTEF, and the achievements on provincial planning and budgeting. The MHMS has also delivered on many of its agreements including holding regular HSSP meetings that have supported high level policy dialogue for improved management of the sector.

18

MHMS staff have always made themselves available to meet with Bank Staff and other DPs. Based on this assessment the overall borrower performance is rated Moderately Satisfactory. 6. Lessons Learned 62. The MHMS at the time of project preparation, was trying to build consensus on a medium term health program, having relatively successfully stabilized service provision following the “troubles.” There were many DP projects financing different programs and one large AUSAID project managed by a contractor. Virtually all DP projects were managed outside of the SIG budget framework and there was no standard system (budget framework) for MHMS (or SIG) documenting planned expenditures and their outcomes. MHMS recognized they had no system for oversight and that the executive had no system to manage and monitor implementation of the overall health programs. DPs agreed and also recognized that the transactions costs for DPs and SIG were very high under the current arrangements. Governments and DPs had a growing commitment in the Pacific, consistent with global trends, to the development of SWAps as a way to reduce sector transaction costs, improve focus on outcomes and to support one agreed national strategy for the health sector. At face value, it seemed a SWAp, could address the key issues facing Solomon Islands. To this end, a significant part of preparation activities for the project focused on establishing the building blocks for a health SWAp. A key lesson learned is that the transaction costs to establish the preconditions for a pooled SWAp are very difficult in a fragile and low capacity environment. It requires a strong commitment of DPs to work together with a collective understanding of what this means as a way of doing business, including government execution of programs, use of government systems, and appropriate fiduciary oversight. There was significant conflict between DPs on these issues during preparation. The MHMS, understandably, was unable to exercise strong leadership on these issues as it was unsure how to proceed. There were also weak relationships with the central agencies of Government and they did not fully appreciate the issues either. Thus, many preconditions to establish a SWAp were absent. An agreement on upfront criteria to assess readiness of government systems for pooled financing would have been helpful. 63. The project team took the right decision to shift to a TAL focused on the core building blocks for the HSSP – support for one sector program; enhanced DP coordination; and joint monitoring and strengthening of MHMS budget and planning systems. This perhaps took too long because of the collective regional commitment to a SWAp without being clear on what the up-front readiness conditions should be. 64. A key question is whether a small project with high transaction costs is worth it? The transaction costs to both SIG and the Bank were significant. The view of this ICR is that the project was high risk with potentially high benefits and the Bank team took the right decision to prepare a TAL. The Bank was able to add value across the core domains of the project– a view expressed by the client and DPs (see annexes 7 and 8). This impacted positively on the wider HSSP – thus the small Bank TA grant leveraged the whole of the health program through its work on planning and budgeting systems, the MTEF, and most importantly the new NHSP and M&E. DPs indicated that the project

19

and Bank team greatly assisted MHMS and DPs to stay focused on the “big picture” i.e. how resources were being deployed, key gaps, and how outcomes were evolving. 65. Another lesson is that enhanced supervision and implementation support is needed, particularly in fiduciary management for projects, in countries such as the Solomon Islands, no matter what the scale of the project. Procurement and particularly financial management support from specialized staff in the Bank was inadequate and typically not coordinated with review missions. The separation of team responsibilities within the Bank for different parts of the supervision process in the recent past has reduced the effectiveness of the Bank in supervision of projects. A key lesson is that the Bank needs to develop a better business model for undertaking fiduciary supervision in small countries where it cannot justify locating fiduciary specialists in-country. 66. Another lesson is that persistence on the need for core Bank requirements sometimes pays off. The executing agency often complained strongly about the Bank’s insistence, under the project, to develop a procurement plan and follow it. The MHMS now openly recognize that this is precisely what they need for their own government development budget and are now planning one for their 2013 development budget because of significant delays in 2012 (and earlier) due to a lack of planning. 67. Finally, a core lesson of this project has been that the focus on the development of systems is “in and of itself” a capacity development strategy. In the latter part of the project TORs for TA, support was defined as “process” consulting i.e., providing tools and decision making frameworks for MHMS staff to make strategic decisions collectively. For the development of the new NHSP and for the finance advisor the TA support was specified as process support. This meant that outcomes were controlled by MHMS staff with the consequence that ownership of decisions and positions taken were clearly owned – a situation that is not always evident with a “task” consulting approach. It also means that much larger numbers of MHMS staff are involved and understand objectives and processes – a significant achievement.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 68. The MHMS believe the development objectives remained relevant and focused on key aspects of the health systems development. The new executive grew to better understand the benefits of the project as it was implemented. While small, the project was an integral part of the overall support to the HSSP. The MHMS had lost all institutional memory of the past project and the design, without a project management unit, was thus left to the executive to manage. This led to an implementation hiatus when the executive was changed. As an inexperienced executive there was limited understanding of the Bank’s focus on the procurement plan. This is now much better understood by the MHMS (other DPs undertook most major procurements) and it plans to use such an approach for the development budget. 69. The MHMS believe the project outcomes were significant. Highlighted is the new NHSP with its key programmatic priorities and organizational and management

20

reforms required to achieve the programmatic objectives. The M&E framework is also now well owned and its benefits better understood. The focus on budget systems development has been beneficial in developing capacity and allowing better understanding of how resources are being used. The support to the provincial planning process has been very significant and will enable further de-concentration of service delivery. The MTEF for the first time brought together “whole of sector” analysis of expenditures and how they are financed. The new executive was initially not sure of its benefits but now see the importance of completing it annually. The new NHSP and the budget and planning processes have involved extensive stakeholder and provincial involvement in health planning, budgeting and policy dialogue. This is an important development. 70. The MHMS recognizes that staff turnover and limited initial understanding of the project by the new executive led to project delays. Project delays were the core issue due to capacity constraints - not intent. Limited experience with recipient execution meant that implementation capacity needed to be developed. The overall rating of implementation agency performance is understood. 71. The Bank was very supportive on implementation issues during support missions and contributed to policy dialogue. This kept the focus on the strategic issues – particularly those related to overall expenditures and expenditure priorities. Its sector work provided timely strategic advice not otherwise available. Banks aides memoires were important mechanisms for the overall monitoring of the HSSP and the project. The MHMS also believes the support on financial management of the project from the Bank was inadequate. 72. With respect to the future the MHMS believes the MTEF, provincial planning and budget system reforms and the M&E framework are now well established and will be continued as a result of project support. It is recognized that additional and continued support will be needed for public financial management to deepen the reforms already in place. The MHMS is also keen to explore efforts to build on project achievements and consider results based financing at the provincial level. (b) Co-financiers 73. The DPs have all recognized the positive contribution of the project to the HSSP and of the Bank’s strong contribution through implementation support missions to the overall policy dialogue on health sector issues. It is clear that the DP commitment to a HSSP has grown and that trust between all parties, including of MHMS with DPs, has grown significantly over the past 2-3 years – trust that was not always there previously (see Annex 8).

21

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent)

Components Appraisal Estimate (USD millions)

Actual/Latest Estimate (USD millions) a

Percentage of Appraisal

Public Expenditure Management 0.545 0.578 1.06 Sector Performance Monitoring 0.382 0.366 0.96 Training and Capacity Building 0.523 0.286 0.55

Total Baseline Cost 1.450 1.230 Physical Contingencies/Unallocated 0.050 0.000 Price Contingencies 0.000

Total Project Costs 1.500 1.230 Project Preparation Facility (PPF) 0.000 Front-end fee IBRD 0.000

Total Financing Required 1.500 1.230 82%

(b) Financing

Source of Funds Type of Cofinancing

Appraisal Estimate (USD millions)

Actual/Latest Estimate (USD millions)

Percentage of Appraisal

Borrower 0.00 0.00 0.00 IDA Grant 1.50 1.52 1.01

TOTAL 1.50 1.52 1.01 a The final disbursed amount totaled SDR 817,666.94 which represents a disbursed amount of 82% of the original grant amount of SDR 1,000,000.

22

Annex 2. Outputs by Component

PDO Project Outcome Indicators Comments

The project development objective is to improve the institutional capacity of MHMS in the areas of public expenditure management and sector performance monitoring.

(i) Completion of the first rolling medium term sector expenditure program linking resources with priorities. The criteria to indicate successful completion are: the activities are aligned with clear objectives; the activities are budgeted in a realistic manner; and The MTEF is indeed used and reflected in the budget discussion, as evidenced by the reallocation of resources towards priority areas from one year to the next.

The MTEFs were delayed: an initial draft was completed in 2010 and a final version in 2011. The draft MTEF in 2010 formed the basis for the definition of the overall resource envelope for the new NHSP and for the annual planning and budgeting process in FY10 and FY11. The final MTEF in 2011informed the FY12 operational plans and budget. Importantly the MTEF created and informed dialogue on (i) sector expenditure priorities and formed the basis for some change – government and DP financed; and (ii) highlighted major future significant and unplanned imposts on future recurrent budget arising from both DP financed project investments (e.g. a new hospital and refurbished malaria building) and of policy decisions (e.g. returning Cuban trained doctors). A strategic decision of the new NHSP, based on the MTEF work, was to decrease national program recurrent budget funding by 14% and increase provincial programs by 17% by 2015. MHMS wants to periodically update the MTEF to inform planning and budget processes and ensure improved transparency of resource allocations.

23

Further, MOF&T is now planning to introduce a rolling national MTEF for all sectors over the coming months as part of the budgeting processes.

(ii) Increased from 17% the proportion of health sector expenditures on primary health services and provincial health programs

The proportion of health sector expenditure on primary health services and provincial health programs has increased from 17% (estimated base at appraisal) to 29% for FY11 (target 22%). (Source Final 2011 MTEF)

(iii) Results of completed health facility survey used in health planning; inefficiencies arising from inappropriate mix of inputs would be identified and corrected by reallocation of inputs; variations in unit costs and occupancy rates used to identify inefficiencies in production and reallocate resources appropriately.

This survey was not undertaken as planned. Initial priority, given limited management capacity in MHMS was to undertake the MTEFT and provincial planning work. In the initial project hiatus AusAID went ahead with a facility survey focused primarily but not exclusively on infrastructure. This finally provided some important information for the provincial planning processes and the MTEF. UNICEF entered into discussions to undertake a costing study using the tool Marginal Budgeting for Bottlenecks. Finally, the MHMS also decided not to proceed with the study and to focus on completion of the new NHSP. MHMS decided to proceed with hospital and facility costing/survey work in the last year of the project. Sector dialogue generated a demand for this work. However, time constraints, arising from international travel of the executive, meant selection of consultants were not finalized and the work lapsed.

24

(iv) Participatory annual joint reviews are conducted on schedule and clearly identify the main issues and formulate actionable recommendations

Formal Joint Annual Performance Reviews were conducted in 2009, 2010, and 2011 and will continue annually – next one scheduled for June 2012. The MHMS executive and DPs continue to see value in this process and jointly agree the process is becoming more productive and focused, and should continue, as all parties learn to work in new ways. Decisions made are monitored and discussed in joint dialogue established scheduled for three times a year. Improved data from the HMIS should further enrich this process.

Intermediate Outcomes Intermediate Outcome Indicators Results

Component 1: