Implementation Completion and Results Report (ICR) Document - … · The World Bank Page 1 of 72 Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004274 IMPLEMENTATION

The World Bank

Page 1 of 72

Document of

The World Bank

FOR OFFICIAL USE ONLY

Report No: ICR00004274

IMPLEMENTATION COMPLETION AND RESULTS REPORT Loan No. 8038-ID

ON A

LOAN

IN THE AMOUNT OF US$65 MILLION

TO THE

REPUBLIC OF INDONESIA

FOR THE

STATISTICAL CAPACITY BUILDING-CHANGE AND REFORM FOR THE DEVELOPMENT OF

STATISTICS (STATCAP-CERDAS) ( P106384 )

November 29, 2017

Governance Global Practice

East Asia And Pacific Region

This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization.

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

The World Bank

Page 2 of 72

CURRENCY EQUIVALENTS

(Exchange Rate Effective Nov 1, 2017)

Currency Unit = Indonesian Rupiah (IDR)

IDR 13,580.00 = US$1

US$0.74 = IDR 10,000

FISCAL YEAR

July 1 - June 30

Regional Vice President: Victoria Kwakwa

Country Director: Rodrigo A. Chaves

Senior Global Practice Director: Deborah L. Wetzel

Practice Manager: Robert R. Taliercio

Task Team Leader(s): Noriko Toyoda, Erwin Ariadharma

ICR Main Contributor: Lina Lo

Page 3 of 72

ABBREVIATIONS AND ACRONYMS

ABS Australian Bureau of Statistics

AIPEG Australia Indonesia Partnership for Economic Governance

Bappenas Ministry of National Development Planning (Badan Perencanaan Pembangunan Nasional)

BPK Supreme Audit Institution (Badan Pemeriksa Keuangan)

BPKP Finance and Development Supervisory Agency (Badan Pengawasan Keuangan dan Pembangunan)

BPS National Statistics Agency or Statistics Indonesia (Badan Pusat Statistik)

BR Business Register

CAPI Computer-Assisted Personal Interview

CERDAS Change and Reform for the Development of Statistics

CSI Corporate Statistical Infrastructure

EA Enterprise Architecture

GDP Gross Domestic Product

GDRP Gross Domestic Regional Product

GFMRAP Government Financial Management and Revenue Administration Project

GSBPM Generic Statistical Business Process Model

HR Human Resources

HRMIS Human Resources Management Information System

IAS Internal Audit System

ICR Implementation Completion and Results Report

ICT Information and Communication Technology

IEG Independent Evaluation Group

ISR Implementation Status and Results Report

IT Information Technology

KemenPAN-RB Ministry for Administrative and Bureaucracy Reform (Kementerian Pendayagunaan Aparatur Negara dan Reformasi Birokrasi)

KPI Key Performance Indicator

LBU Large Business Unit

LKPP National Public Procurement Agency (Lembaga Kebijakan Pengadaan Barang Jasa Pemerintah)

M&E Monitoring and Evaluation

MOF Ministry of Finance

MOU Memorandum of Understanding

NSDS National Strategy for Development of Statistics

NSO National Statistics Office

PDO Project Development Objective

PHRD Policy and Human Resources Development

PINTAR Project for Indonesian Tax Administration Reform

PMO Project Management Office

PMU Project Management Unit

PPI Producer Price Index

REOI Request for Expression of Interest

RFP Request for Proposal

SBFA Statistical Business Framework and Architecture

SOP Standard Operating Procedure

STATCAP Statistical Capacity Building

SUSENAS National Social Economic Survey (Survei Sosial Ekonomi Nasional)

TA Technical Assistance

Page 4 of 72

TOR Terms of Reference

TTL Task Team Leader

UNECE United Nations Economic Commission for Europe

USDA-NASS United States Department of Agriculture-National Agricultural Statistics Service

Page 5 of 72

TABLE OF CONTENTS

DATA SHEET .......................................................................................................................... 6

ICR EXECUTIVE SUMMARY ................................................................................................... 11

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ..................................................... 14

A. CONTEXT AT APPRAISAL ................................................................................................................ 14

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE)......................................... 16

II. OUTCOME .................................................................................................................... 21

A. RELEVANCE OF PDOs ..................................................................................................................... 21

B. ACHIEVEMENT OF PDOs (EFFICACY) .............................................................................................. 22

C. EFFICIENCY ...................................................................................................................................... 24

D. JUSTIFICATION OF OVERALL OUTCOME RATING .......................................................................... 24

E. OTHER OUTCOMES AND IMPACTS (IF ANY) .................................................................................. 24

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 25

A. KEY FACTORS DURING PREPARATION .......................................................................................... 25

B. KEY FACTORS DURING IMPLEMENTATION ................................................................................... 26

C. MAIN EVENTS/FACTORS LEADING TO CANCELLATION ................................................................. 28

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 30

A. QUALITY OF MONITORING AND EVALUATION (M&E) ................................................................. 30

B. ENVIRONMENTAL SAFEGUARD, SOCIAL SAFEGUARD, AND FIDUCIARY COMPLIANCE ............... 31

C. BANK PERFORMANCE .................................................................................................................... 31

D. RISK TO DEVELOPMENT OUTCOME .............................................................................................. 33

V. LESSONS AND RECOMMENDATIONS ............................................................................. 33

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 35

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 49

ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 52

ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 53

ANNEX 5. BORROWER’S COMPLETION REPORT AND/OR COMMENTS .................................. 54

ANNEX 6. SUPPORTING DOCUMENTS .................................................................................. 66

ANNEX 7. RATINGS OF INDICATORS BY SUB-OBJECTIVES ...................................................... 67

The World Bank Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS) ( P106384 )

Page 6 of 72

DATA SHEET

BASIC INFORMATION

Product Information

Project ID Project Name

P106384

STATISTICAL CAPACITY BUILDING-CHANGE AND REFORM

FOR THE DEVELOPMENT OF STATISTICS (STATCAP-

CERDAS) ( P106384 )

Country Financing Instrument

Indonesia Specific Investment Loan

Original EA Category Revised EA Category

Not Required (C)

Organizations

Borrower Implementing Agency

Ministry of Finance BPS-Statistics indonesia

Project Development Objective (PDO) Original PDO

To increase the effectiveness and efficiency of the Indonesian National Statistics Office (BPS-Statistics Indonesia) to produce and disseminate reliable and timely statistics in accordance with international standards and best practices, and to be responsive to user needs

Page 7 of 72

FINANCING

Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$)

World Bank Financing TF-98283

350,000 216,438 216,438

IBRD-80380

65,000,000 8,455,448 8,455,448

Total 65,350,000 8,671,886 8,671,886

Non-World Bank Financing

Borrower 14,000,000 8,370,000 1,620,000

Total 14,000,000 8,370,000 1,620,000

Total Project Cost 79,350,000 17,041,886 10,291,886

KEY DATES

Approval Effectiveness MTR Review Original Closing Actual Closing

07-Apr-2011 18-Jul-2011 31-Dec-2016 31-May-2017

RESTRUCTURING AND/OR ADDITIONAL FINANCING

Date(s) Amount Disbursed (US$M) Key Revisions

15-Apr-2014 0 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Cancellation of Financing

KEY RATINGS

Outcome Bank Performance M&E Quality

Unsatisfactory Moderately Satisfactory Modest

RATINGS OF PROJECT PERFORMANCE IN ISRs

No. Date ISR Archived DO Rating IP Rating Actual

Disbursements (US$M)

01 02-Apr-2012 Moderately Satisfactory Moderately Satisfactory 0

Page 8 of 72

02 25-Dec-2012 Moderately

Unsatisfactory Moderately Unsatisfactory 0

03 06-Nov-2013 Unsatisfactory Unsatisfactory 0

04 26-Jun-2014 Moderately

Unsatisfactory Moderately Unsatisfactory 0


Unsatisfactory Moderately Unsatisfactory .30

06 25-Jun-2015 Moderately

Unsatisfactory Moderately Unsatisfactory 1.13

07 05-Jan-2016 Moderately




09 12-May-2017 Unsatisfactory Unsatisfactory 6.62

SECTORS AND THEMES

Sectors

Major Sector/Sector (%)

Public Administration 100

Central Government (Central Agencies) 100

Themes

Major Theme/ Theme (Level 2)/ Theme (Level 3) (%)

Page 9 of 72

Economic Policy 38

Fiscal Policy 19

Fiscal sustainability 5

Public Expenditure Policy 5

Tax policy 5

Subnational Fiscal Policies 4

Macro-financial policies 15

External Finance 5

Monetary and Credit Policies 5

Macroeconomic Resilience 5

Economic Growth and Planning 4

Migration, Remittances and Diaspora Engagement

4

Private Sector Development 18

Jobs 4

Youth Employment 4

Enterprise Development 14

MSME Development 14

Public Sector Management 43

Public Administration 14

Municipal Institution Building 14

Data Development and Capacity Building 29

Data production, accessibility and use 29

Human Development and Gender 4

Labor Market Policy and Programs 4

Labor Market Institutions 4

ADM STAFF

Role At Approval At ICR

Regional Vice President: James W. Adams Victoria Kwakwa

Country Director: Stefan G. Koeberle Rodrigo A. Chaves

Page 10 of 72

Senior Global Practice Director: Vikram Nehru Deborah L. Wetzel

Practice Manager: Shubham Chaudhuri Robert R. Taliercio

Task Team Leader(s): Vivi Alatas Noriko Toyoda, Erwin Ariadharma

ICR Contributing Author: Lina Lo

Page 11 of 72

ICR EXECUTIVE SUMMARY

The Indonesia Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS) Project was approved in April 2011 with the objective of increasing the effectiveness and efficiency of the Indonesian National Statistics Agency or Statistics Indonesia (Badan Pusat Statistik, BPS) to produce and disseminate quality statistics.

The Government requested a STATCAP loan to support the BPS reform agenda, to enable the organization to better respond to challenges arising from the changes in its external environment. The rapid decentralization process had created a great demand for disaggregated statistics while at the same time reducing the availability of subnational data through line ministries. This had a major impact on BPS. Other challenges for BPS included maintaining capacity to capture new forms of economic activity and continuing to improve cooperation with data producers, users, and respondents. Furthermore, the President had put Bureaucracy Reform as one of the highest priorities of his administration, calling for government agencies to provide services that were ‘better, faster, and cheaper’.

The theory of change underlying the STATCAP-CERDAS project was that improved people, process, and technology aspects within BPS would significantly increase the quality of the statistical value chain—from identifying user needs to producing and disseminating statistics—which in turn would contribute to strengthening evidence-based policy making and better governance. The original project cost was US$83 million, of which US$65 million was an IBRD loan and US$18 million was the Government’s counterpart budget. The project was to be completed within five years by the end of 2016.

The project became effective in July 2011, but the Chief Statistician who had initiated and championed the STATCAP-CERDAS preparation was promoted to another ministry within three months, and a new Chief was not appointed until February 2012. During this time, BPS started the procurement of the project management (Owner’s Agent) and statistical consultancy packages, but progress stalled in the transition. Implementation was additionally hampered by the loss of the Project Management Unit (PMU) structure due to regulatory changes in December 2011. Implementation continued to be delayed through 2012, and the project restructuring dialogue was formally initiated in January 2013.

STATCAP-CERDAS was restructured in April 2014 to align the scope with the new leadership’s direction, focusing on the improvements to the core statistical business processes and systems of BPS and reducing scope where progress had been made by BPS under its own financing or with other technical assistance (TA). The implementation of the back-office information technology (IT) systems (Human Resources Management Information System [HRMIS] and Internal Audit System [IAS]) was dropped, which contributed to the bulk of the partial cancellation amount. The financing amount was reduced accordingly to US$47 million of IBRD loan and US$14 million of counterpart budget, and the closing date was extended to November 30, 2018.

STATCAP-CERDAS gained some traction after restructuring, as key consultancies were contracted and all project components moved into implementation stage. The Statistical Transformation and Quality Improvement consultant and the IT Strategy and Design consultant developed inputs to the technical specifications for the design, development, and implementation of the Integrated Statistical Applications Systems package. The Human Resources (HR Management and Organization Design consultant also produced people and organizational alignment requirements based on the new business processes and functions. Implementation of the integrated statistical information systems was critical to providing

Page 12 of 72

common IT tools and applications to support statistical production and dissemination processes across BPS.

The Request for Expression of Interest (REOI) for the statistical IT systems package was launched in February 2016, and the technical evaluation was completed by the end of October 2016. However, due to the complexity and size of the package, BPS requested the Government’s internal audit agency (Finance and Development Supervisory Agency [Badan Pengawasan keuangan dan Pembangunan, BPKP]) to review the procurement process before submitting the Technical Evaluation Report to the World Bank. BPKP raised a number of findings, notably that the procurement committee had not obtained the prior approval of the cost estimates from the competent authority in the Government before initiating the selection process and that the short-listing process was not in compliance with the World Bank’s Consultant Guidelines, even though the World Bank had issued no objections after having prior reviewed the process, and in spite of constant efforts from the procurement team to address the concerns of the client while reassuring that process was in compliance with the Consultant Guidelines.

Due to further delays, the project was suspended in December 2016 with remedial actions to (a) award the contract for the Design and Implementation of Integrated Statistical Information Systems consultancy and (b) establish an appropriate counterpart project team for its implementation. BPS failed to complete the procurement, and the project was eventually cancelled in May 2017 with US$8.46 million, or 13 percent of the original loan amount, disbursed.

Throughout its lifetime, project implementation was affected by turnover in key counterparts at the leadership as well as operational levels and suffered from challenges in isolating staff from the ‘business-as-usual’ to work on the reform project. STATCAP-CERDAS was led by three Chief Statisticians throughout its life cycle. All three chiefs embraced the same ‘to be’ vision for BPS modernization but differed in views on the path toward the ‘to be’ state. The original scope of the STATCAP-CERDAS presented a comprehensive institutional transformation program, which touched all parts of the BPS organization and required a leader to drive this multipronged transformation across the rank and file of his/her organization. The PMU was also reconstituted four times, partly due to rigid constraints of the civil service regulation to establish a cohesive structural unit to manage the reform on a full-time basis.

Although the project was not successful in deploying the information system to produce and disseminate statistics, STATCAP-CERDAS provided BPS with the requisite tools to continue its modernization. The TA provided by the project supported the design of the BPS Statistical Business Framework and Architecture (SBFA) with standardized business processes and operating model, the detailed designs of various Corporate Statistical Infrastructure (CSIs), IT Strategy and Enterprise Architecture (EA), and macro-organization redesign. The TA provided by the World Bank through trust fund support as well as by development partners such as the Australian Bureau of Statistics (ABS) was aligned with the STATCAP-CERDAS framework, which contributed to improved results. At the time of the Implementation Completion and Results Report (ICR), BPS had secured state funding from the Ministry of Finance (MOF) over the next four years to continue its transformation program and implement the statistics information systems development.

The overall outcome rating for the project is Unsatisfactory, largely due to the project’s inability to complete implementation. The rating for relevance is High, with the ongoing need for the project remaining strong. The rating for efficacy is Modest, with some of the Project Development Objective (PDO) indicators improved, though most of the intermediate results indicators underperformed. The

Page 13 of 72

rating for efficiency is Negligible, due to considerable implementation delays and early cancellation. The World Bank performance overall is Moderately Satisfactory.

In this ICR, the team has presented lessons and recommendations related to planning for sustainability, championing reform, organizing for reform, planning for complex information and communication technology (ICT) procurement, reorienting auditors’ focus, and targeting niche markets, which may be applicable to other institutional reform initiatives.

Page 14 of 72

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES

A. CONTEXT AT APPRAISAL

Context

1. At the time of project preparation in 2011, Indonesia had emerged as a middle-income country with gross domestic product (GDP) per capita at US$2,230 and a resilient economy that weathered the 2008/09 global financial crisis better than its neighbors. Although the poverty rate had been decreasing steadily, the rate of poverty reduction had not kept pace with economic growth. The 2001 ‘big-bang’ decentralization, with most of the public service delivery responsibilities and corresponding planning and budgeting authorities devolved to the subnational levels of governments, increased the demand for relevant, accurate, timely, and disaggregated data.

2. Indonesia’s National Statistics Agency or Statistics Indonesia (Badan Pusat Statistik, BPS) is a central government agency, organized vertically with regional offices located in all provinces and districts across the archipelago. BPS is the coordinator of Indonesia’s National Statistics System and is responsible for producing a wide range of key social and economic statistics, including poverty, labor, price, trade, production, and national accounts data. Official statistics are a crucial element of informed, evidence-based decision making and monitoring of national development strategies. Government agencies from the Ministry of National Development Planning (Badan Perencanaan Pembangunan Nasional, Bappenas) to the Ministry of Finance (MOF) to line ministries depend on BPS statistics for planning and budgeting, targeting of policies and programs, and outcome monitoring and evaluation (M&E).

3. At appraisal, the leadership of BPS was keenly aware of the changes in its external environment and was committed to a comprehensive reform of BPS to deliver better results for the public. The rapid decentralization process in Indonesia that started in 2001 had created a great demand for subnational statistics while at the same time reducing the accessibility of subnational data through the line ministries. This had a major impact on BPS. Other challenges for BPS included maintaining capacity to capture new forms of economic activity, and continuing to improve cooperation with data producers, users, and respondents. Much of the BPS survey operations still relied on point-to-point, paper-based processes, and its technology infrastructure was outdated; so, there was significant room for efficiency gains through investments in process improvements and technology. There were also opportunities for greater integration of surveys and greater use of common tools and shared systems across the organization to improve the timeliness and efficiency of data collection and processing. Furthermore, the President had put Bureaucracy Reform as one of the highest priorities of his administration and called for government agencies to provide services that were ‘better, faster, and cheaper’.

4. The World Bank had provided various technical assistance (TA) to BPS, including support to develop its National Strategy for Development of Statistics (NSDS), which identified the need for BPS to respond to challenges of increasing demand for more data arising from rapid decentralization and the country’s changing socioeconomic environment. In light of escalating demands and limited resources, BPS requested the World Bank for Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS) financing to assist its efforts in improving the quality of major statistical collections and investing in core statistical infrastructure to shift the organization toward more corporately integrated and coherent operations. The Chief Statistician also used STATCAP-CERDAS as the framework for harmonizing development partner assistance, including its cooperation program with the Australian Bureau of Statistics (ABS), as outlined in the 2010–2015 Memorandum of Understanding (MOU) between the two National Statistics Offices (NSOs).

Page 15 of 72

5. STATCAP-CERDAS was aligned with the priorities of the World Bank’s Country Partnership Strategy 2009–2012 (Report No. 44845-IND) under the cross-cutting engagement to support central government institutions and systems. The project was to strengthen BPS’s institutional capacity do deliver better information and services to the Government and the public. The project also aimed to support the national development priorities of bureaucracy and governance reform through (a) improved statistics from BPS to support informed and evidence-based decision making across the Government and (b) improving the institutional performance and efficiency of BPS.

Theory of Change (Results Chain) 6. The theory of change underlying the STATCAP-CERDAS project was that improving the people, process, and technology aspects within BPS would significantly increase the quality of the statistical value chain—from identifying user needs to producing and disseminating statistics—which in turn would contribute to strengthening evidence-based policy making and better governance.

Figure 1. Theory of Change for STATCAP-CERDAS Project

Project Development Objectives (PDOs) 7. The STATCAP-CERDAS project was to increase the effectiveness and efficiency of the Indonesian National Statistics Office (BPS-Statistics Indonesia) to produce and disseminate reliable and timely statistics in accordance with international standards and best practices, and to be responsive to user needs.

Key Expected Outcomes and Outcome Indicators 8. The PDO contained several compound objectives and was unpacked into two sub-objectives for this Implementation Completion and Results Report (ICR) (see Figure 1):

(a) To increase the effectiveness of BPS in producing and disseminating quality statistics

(b) To increase the efficiency of BPS in producing and disseminating quality statistics

Page 16 of 72

9. Because STATCAP-CERDAS was to support fundamental reforms and change efforts to improve the quality of statistics, key PDO-level indicators were arranged along the quality dimensions of official statistics, as commonly defined by international organizations and NSOs. The indicators measured improvements in accuracy, relevance, timeliness, accessibility, coherence, and interpretability of key BPS statistics.

Components 10. STATCAP-CERDAS comprised five components as follows (see also Figure 1):

A. Improved Statistical Quality (US$32.8 million) to streamline and integrate its statistical business processes

B. Improved ICT Platform and Statistical Information Management Systems (US$30 million) to modernize its IT systems and infrastructure

C. Improved Human Resources Management and Development (US$9.3 million) to strengthen its human resources (HR) policies and practices from personnel administration toward people management

D. Improved Organizational Alignment (US$4.8 million) to restructure its organization in line with new business processes and functions

E. Project Management (U$6.1 million) to ensure integration across components and manage change across the organization

11. The original project cost was US$83 million, of which US$65 million was IBRD loan and US$18 million was the Government’s counterpart budget. The project was to be completed in five years by the end of 2016.

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE)

Revised PDOs and Outcome Targets 12. After two years of zero disbursement, a Level 2 restructuring of the project was completed in April 2014 to focus the project scope on improving BPS’s core statistical business and to account for improvements being implemented by BPS and through other TA. The loan amount was reduced from US$65 million to US$47 million, and the counterpart budget from US$18 million to US$14 million. The project closing date was extended to November 30, 2018.

13. The PDO remained the same after restructuring, while the project was streamlined to focus on improvements to BPS’s core business processes and systems for statistical production and dissemination. A large part of the reduction in the financing amount stemmed from the cancellation of design and implementation of two information systems—Human Resources Management Information System (HRMIS) and Internal Audit System (IAS)—which could be the next generation of reform initiatives for BPS.

Page 17 of 72

Revised PDO Indicators 14. The desired outcome remained as improving statistical quality. The PDO indicators were streamlined but continued to reflect the statistical quality dimensions—accuracy, relevance, timeliness, accessibility, coherence, and interpretability. The intermediate results indicators were also revised and streamlined to align with the scope adjustments and reductions under each component. Table 1 summarizes the original and revised indicators.

Table 1: Original and Revised Indicators

Indicators Action per Restructuring

PDO INDICATORS

PDO 1: Accuracy: Demonstrable improvement in the accuracy of key statistics as evidenced by

(a) A reduction in the statistical discrepancy between the quarterly and annual production and expenditure estimates of growth in the national accounts

Dropped (TA on National Accounts provided by ABS)

(b) An increase in the response rate of the Annual Survey of large and medium manufacturing businesses

Unchanged

(c) An increase in the user satisfaction with the accuracy of BPS statistics, including the extent of revisions

Revised An increase in the user satisfaction with the accuracy of BPS statistics

PDO 2: Relevance: Availability of important new statistics as evidenced by the establishment of

(a) The introduction of the 2008 version of the System of National Accounts

Dropped (TA on National Accounts provided by ABS)

(b) A national producer price index Unchanged

(c) Quarterly labor force statistics Dropped (BPS changed its internal target.)

PDO 3: Timeliness: Improvements in the timeliness of key statistics as evidenced by

(a) Reducing the time lag in the production of preliminary and final imports and exports data

Dropped (BPS already achieved timeliness improvement.)

(b) Reducing the time lag in the publication of final figures from the National Social Economic Survey (SUSENAS)

Dropped (BPS would continue to improve in this area on its own.)

(c) Reducing the time lag in the publication of annual labor force estimates

Unchanged

PDO 4: Accessibility: An improvement in the accessibility of BPS statistics as evidenced by

(a) A dynamic up-to-date website Unchanged

(b) An increase in the user satisfaction with the accessibility of BPS statistics

Unchanged

PDO 5: Coherence: An improvement in the coherence of BPS statistics

As evidenced by a reduction in the discrepancy between the sum of the Provincial estimates of GDRP and the GDP for Indonesia

Unchanged

PDO 6: Interpretability: An improvement in the interpretability of BPS statistics as evidenced by

(a) An increase in the statistics produced by BPS, especially widely used statistics (for example, national accounts, prices, labor, and SUSENAS) that have supporting meta data that describe the concepts, data sources, and statistical methods that have been used

Unchanged (slightly revised) An increase in the statistics produced by BPS that have supporting meta data

(b) The availability of seasonally adjusted monthly and quarterly statistics Dropped (BPS would continue to improve in this area on its own.)

Page 18 of 72

INTERMEDIATE RESULTS INDICATORS

Component A: Improved Statistical Quality

1. User groups and technical advisory groups are established Dropped (Dropped due to process-oriented nature of the indicator)

2. Successful implementation of a Quality Assurance Framework across BPS

Unchanged

3. The establishment of a Common Business Register (BR) and a Large Business Unit (LBU)

Unchanged

4. A significant upgrade in the source data used in the compilation of the National Accounts

Dropped (TA on National Accounts dropped from scope)

5. Availability of an integrated data and information service system supported by the data warehouse

Unchanged

Component B: Improved ICT Platform and Statistical Information Management Systems

6. Improvement in common ICT infrastructure and its performance Dropped (Streamlined with other ICT indicators)

7. ICT standardization (ICT governance, ICT architecture, ICT procurement, applications development methodology, version management of applications and software, systems testing, standard tools, and support)

Revised Improvement in ICT Governance and Management

8. Improved availability of suitably qualified ICT experts Dropped (Streamlined with other ICT indicators)

Added Development of application systems for statistical production and dissemination

Component C: Improved Human Resources Management and Development

9. A workforce that is aligned with BPS’s future strategic direction Revised A Human Resource that is aligned with BPS’s future strategic direction

10. The successful implementation of an integrated HRM System Dropped (Subcomponent dropped from scope)

11. An effective performance management system for BPS Revised An effective, integrated performance management system for BPS

Component D: Improved Organizational Alignment

12. Improved alignment of the BPS organization and the future work environment

Unchanged

13. BPS functions in accordance with agreed corporate values Dropped (Subcomponent dropped from scope)

14. Development of a new monitoring and inspection system Dropped (Subcomponent dropped from scope)

15. Effective responses to reviews, audits, and coherency studies Dropped (Subcomponent dropped from scope)

Component E: Project Management

16. Effective project management Unchanged

17. Procurement carried out in timely and transparent manner Dropped (Streamlined with other project management indicators)

Note: GDRP = Gross domestic regional product; ICT = Information and communication technology.

Page 19 of 72

Revised Components 15. Restructured project components continued to be aligned with the three pillars of institutional transformation—People (Components C and D), Process (Component A), and Technology (Component B), and were unchanged to deliver the desired outcome. However, subcomponents and activities were revised as follows:

A. Improved Statistical Quality (US$6.9 million). This component was reduced in scope to focus on TA for specific statistical product lines and on analysis and design of core statistical infrastructure to enable greater statistical integration. Scope revision took into account improvements made by BPS and through other complementary TA, for example, in National Accounts area.

B. Improved ICT Platform and Statistical Information Management Systems (US$29.06 million). This component remained largely unchanged to develop BPS’s information technology (IT) strategy, Enterprise Architecture (EA), information systems governance framework, standardization and centralization of IT operations and resources, and development of integrated statistical production and dissemination systems. Hardware component was decoupled from the statistical IT applications package to be procured through the Government’s counterpart budget.

C. Improved Human Resources Management and Development (US$2.3 million for Components C and D, plus US$4.37 million for training and capacity building). This component was reduced in scope to focus on improvements in HR management and development policies and practices, design performance management system, and strengthen BPS’s HR function to shift from personnel administration to people management. Financing for training and capacity building of BPS staff was increased substantially, while the subcomponent on upgrading the HRMIS was dropped from the project.

D. Improved Organizational Alignment (budget combined with Component C above). This component was reduced in scope to focus on the new organization design in alignment with new business processes and functions. The subcomponent on strengthening the internal audit capacity, including implementation of the IAS, was dropped from the project.

E. Project Management (US$4.05 million). A new structural unit to manage BPS transformation and serve as the Project Management Unit (PMU) for STATCAP-CERDAS was proposed to the Ministry for Administrative and Bureaucracy Reform (Kementerian Pendayagunaan Aparatur Negara dan Reformasi Birokrasi, KemenPAN-RB) in November 2013, and the new structure was established in July 2014. This component also financed the Owner’s Agent consultant to support project management, change management, and quality assurance.

Other Changes

16. Despite scope reduction through restructuring, STATCAP-CERDAS failed to sustain steady progress and the project was cancelled in May 2017 with US$8.46 million or 13 percent of the original loan amount disbursed.

Rationale for Changes and Their Implication on the Original Theory of Change 17. The restructuring in April 2014 was intended to align the project with the new organizational direction after project implementation stalled in the leadership transition. The first Chief Statistician was promoted to another agency within the first six months after loan signing, and the new statistician was not appointed until February 2012. During this time, BPS started the procurement of the project management (Owner’s Agent) and statistical consultancy packages, but progress stalled in the transition. Implementation was additionally hampered by the loss of the PMU structure due to regulatory changes in December 2011. Implementation continued to be delayed through 2012, and

Page 20 of 72

the project restructuring dialogue was formally initiated in January 2013. BPS initially proposed to reallocate the bulk of the resources to the capacity-building component, as well as reduce the loan amount substantially. Eventually, in discussion with Bappenas and the MOF, it was agreed to keep the PDO unchanged to support the BPS institutional reform, but to focus the scope on core statistical improvements and integration.

18. Project restructuring focused the project scope on improvements to the core business of BPS, that is, production and dissemination of statistics. The original scope of STATCAP-CERDAS presented a comprehensive institutional transformation to improve people, process, and technology elements, including the rollout of three information management systems—Integrated Statistical Applications Systems, HRMIS, and IAS—which would require the leader to drive this multipronged approach across the rank and file of his/her organization. It was ambitious considering the enormous change in skills, processes, and technologies required to properly operationalize these changes and systems across the country within the original project period of five years. Implementation of back-office information systems (HRMIS and IAS) were dropped at restructuring. Consistent with learning from previous World Bank operations, hardware components were also decoupled from the Integrated Statistical Applications Systems package to simplify the procurement. The World Bank financing was to be for the design, development, and implementation of the information system itself, and BPS would procure the requisite hardware through the Government’s counterpart budget.

19. Most of the dropped or revised indicators as a result of the restructuring were adapted to reflect progress made by BPS, as well as through complementary TA provided in part by the ABS. The changes, therefore, did not affect the original theory of change.

20. Although STATCAP-CERDAS gained traction after restructuring, the project was cancelled in May 2017, because BPS failed to meet the conditions for lifting the suspension, that is, (a) to sign the contract for the integrated statistical information systems package and (b) to establish an appropriate counterpart project team for its implementation. At the request of BPS, the Government’s internal audit agency (Finance and Development Supervisory Agency [Badan Pengawasan keuangan dan Pembangunan], BPKP) reviewed the procurement process and raised a number of findings, notably that BPS’s procurement committee had not obtained prior approval of the estimated cost from the competent authority in the Government before initiating the selection process. BPKP also made findings that the short-listing process carried out by BPS’s procurement committee was not in compliance with the World Bank’s Consultant Guidelines, even though the World Bank had issued no-objections after having prior reviewed and found the BPS procurement committee’s recommendations to be consistent with the principles of the World Bank’s Consultant Guidelines. Based on BPKP’s findings, BPS proposed to cancel the ongoing procurement process and restart a new process, though the World Bank provided further clarifications on the applicability and interpretation of the World Bank’s Consultant Guidelines. Given there were only 18 months left until the project closing date of November 2018, the World Bank proceeded to suspend loan disbursement in December 2016 with remedial actions intended to move the implementation forward. STATCAP-CERDAS was officially cancelled in May 2017, because conditions for lifting the suspension were not met.

Page 21 of 72

A. RELEVANCE OF PDOs

Assessment of Relevance of PDOs and Rating Relevance Rating of PDO: High 21. The project aimed to support BPS’s Strategic Plan for Change and Reform for the Development Statistics (CERDAS) and institutional transformation, and the PDO remains highly relevant. Official statistics are a crucial element for informed, evidence-based decision making and monitoring of national development strategies. Demand for better quality, more timely, and disaggregated statistics from BPS remains high and continues to increase, as other sources of data such as administrative data systems remain underdeveloped. Within BPS, timeliness and quality improvements are still hindered by lack of common IT tools, for example, a common data collection and processing platform could be used by all survey areas and enable electronic data collection from the field, and corporate data warehouses would consolidate data that currently reside in disparate databases to enable data management across the organization. As of the time of this ICR, BPS has secured state budget over the next four years to continue its reform agenda and implement the statistical production and dissemination information systems. The STATCAP-CERDAS is aligned with the current Country Partnership Framework, contributing to the objective of improving social protection programs with higher quality statistics toward targeting the poor and vulnerable, and World Bank operations would have benefited from better BPS data.

22. The original design of the STATCAP-CERDAS was ambitious in rolling out three information systems—Statistical Production and Dissemination Applications System, HRMIS, and IAS—within the original project period of five years, given the average time to complete a World Bank project to implement enterprise-wide information system is 7.9 years, with the longest taking over 13 years. Through restructuring, the project scope was realistically adjusted to focus on the design and implementation of the Statistical Applications System to enable the efficient production and dissemination of statistics. At restructuring, efforts were made to strengthen implementation arrangements by establishing a structural unit for statistical transformation—to also serve as STATCAP-CERDAS PMU—and reconstituting the external advisory board.

II. OUTCOME

Page 22 of 72

B. ACHIEVEMENT OF PDOs (EFFICACY)

Efficacy of PDO: Modest 23. The STATCAP-CERDAS project was designed to comprehensively transform the way BPS conducted its business by standardizing and integrating its processes and leveraging IT to improve the quality of statistics. By project cancellation, most of the PDO indicators—accuracy, relevance, timeliness, accessibility, coherence, and interpretability—had improved. STATCAP-CERDAS also provided the framework for complementary TA notably by the ABS, as well as by the World Bank, which contributed to the statistical quality improvements, despite the project implementation delays and cancellation. Some changes in methodologies, such as the user satisfaction surveys, may affect the direct comparability of achieved results with baseline.

24. The project Results Framework with PDO indicators and intermediate outcomes indicators was reconstructed to strengthen the results chain that supports the project’s theory of change. Achievements of the project by effectiveness and efficiency are collected in annex 7 and summarized in Table 2.

Table 2: Summary of Achievements by Sub-objectives

Indicator Rating according to IEG Guidelines

High Substantial Modest Negligible

1. Increased effectiveness of BPS in producing and disseminating quality statistics

PDO INDICATORS FOR SUB-OBJECTIVE 1

Total for PDO Sub-objective 1 1 2 1 1

Percentage of PDO Indicators for Sub-objective 1 20 40 20 20

INTERMEDIATE RESULTS INDICATORS FOR SUB-OBJECTIVE 1

Total for Intermediate Results for Sub-objective 1 0 1 4 0

Percentage of Intermediate Results Indicators for Sub-objective 1

0 20 80 0

2. Increased efficiency of BPS in producing and disseminating quality statistics


Total for PDO Sub-objective 2 2 0 1 0

Percentage of PDO Indicators for Sub-objective 2 67 0 33 0


Total for Intermediate Results Indicators for Sub-objective 2 0 0 4 0

Percentage of Intermediate Results Indicators for Sub-objective 2

0 0 100 0

Note: IEG = Independent Evaluation Group.

Assessment of Achievement of Each Objective/Outcome

(a) Sub-objective 1: Increased effectiveness of BPS in producing and disseminating quality statistics

Rating: Modest

25. Most of the design of the building blocks to transform BPS and increase the quality of Indonesia’s statistics were achieved as reflected by the status of the intermediate indicators. Designs of the BPS Statistical Business Framework and Architecture (SBFA); Corporate Statistical Infrastructure (CSIs), such as quality governance structure and statistical business register; and BPS IT Strategy, EA, core managerial competencies, and BPS-wide strategy map and associated key performance indicators (KPIs) were completed, with some endorsed (SBFA and EA) by the BPS

Page 23 of 72

executives. These design documents are the basis for the next phase of BPS’s transformation journey to be implemented. The following are the achievements of the four PDO-level indicators that contribute to increased effectiveness:

• Accuracy. Statistics accuracy was improved with response rate to large and medium manufacturing businesses increased from 60 percent to 69.54 percent (target at 74 percent) at project cancellation, and user satisfaction increased from 43 percent to 91.73 percent, exceeding the target of 75 percent. It should be noted, however, that the methodology for User Satisfaction Surveys conducted by BPS changed over the years, which may affect the direct comparability of the achieved level with the baseline.

• Relevance. New statistics for national Producer Price Index (PPI) were published, with additional service sectors (including accommodation, food and beverage, passenger transportation, electricity and gas supply, and water supply), and studies on further sectors (including freight transportation, telecommunications, financial services, education, health, and waste management) have been conducted. For provincial PPI, producer survey data collection has been expanded to all provinces in 2016 and has been prepared for three sectors in five provinces. All these together move BPS a step toward improving availability and coverage of national and subnational PPI but short of the original target of covering all provinces and most large districts.

• Coherence. The discrepancy between the sum of the provincial estimates of GDRP and the GDP for Indonesia has increased from 0.08 percent to 2.03 percent. BPS adopted methodological changes to eliminate the discrepancy (0 percent) in 2010, but the value has been increasing slightly since then.

• Interpretability. At project appraisal, there was no fully documented metadata. At project cancellation, BPS has established an online Microdata Catalog System, containing metadata information for 310 data sets. Increased availability and accessibility of metadata have contributed to improved interpretability of BPS statistics.

(b) Sub-objective 2: Increased efficiency of BPS in producing and disseminating quality statistics

Rating: Modest 26. With the procurement of the statistical IT systems stopped (equal to more than 50 percent of the restructured loan amount), and none of the other contributing factors advanced to planned targets, achievement of this sub-objective is Modest.

• Timeliness. The time lag in the publication of annual labor force estimates improved from 150 days to 90 days, short of the target of 60 days. Further reduction in time lag will likely be difficult to achieve without the implementation of Computer-Assisted Personal Interview (CAPI) or electronic data collection.

• Accessibility. A dynamic up-to-date website with BPS statistics was improved with access increased from 51,000 to 127,147 unique visitors per month (monthly average over the year). There was also an increase from 61 percent to 83.47 percent (target was 80 percent) in user satisfaction on the accessibility of BPS statistics.

Page 24 of 72

Justification of Overall Efficacy Rating 27. On balance, the overall achievement of objectives given above is assessed as Modest.

28. By project cancellation, BPS had the requisite tools to continue its modernization agenda. Despite the failed implementation of the integrated statistical information systems, which could have moved BPS a major step forward toward improving its operational efficiency and statistical quality, the STATCAP consultancies supported to produce key deliverables aimed to lay the foundation, including the standardized business processes based on the Generic Statistical Business Process Model (GSBPM) of the United Nations Economic Commission for Europe (UNECE), EA, and detailed design documents for CSIs, to implement the IT systems for the statistical production and dissemination and rollout the reform.

C. EFFICIENCY

Assessment of Efficiency and Rating Efficiency Rating: Negligible

29. With US$8.46 million, or 13 percent, of the original loan amount disbursed, with modest outcomes overall, the efficiency rating is assessed as Negligible. This also takes into consideration modest to substantial advances in some of the PDO-level indicators and zero disbursement over the initial three-year period.

D. JUSTIFICATION OF OVERALL OUTCOME RATING

Overall Outcome Rating: Unsatisfactory

30. With zero disbursement pre-restructuring, this ICR does not adopt the split rating approach.

31. The justifications for ratings of relevance, efficacy, and efficiency are elaborated above. Rating for relevance is High, efficacy is Modest, and efficiency is Negligible. Taken together, the overall outcome rating is Unsatisfactory based on IEG guidelines.

Table 3. Overall Outcome Rating

Relevance Efficacy

Efficiency Overall Outcome Effectiveness Efficiency

High Modest Modest

Negligible Unsatisfactory Modest

E. OTHER OUTCOMES AND IMPACTS (IF ANY)

Institutional Strengthening

32. As stated earlier, the project supported the design of the building blocks to transform BPS and to increase the quality of Indonesia’s statistics, which has contributed to strengthening BPS’s institutional capacity to continue its transformation.

Page 25 of 72

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME

A. KEY FACTORS DURING PREPARATION

33. STATCAP-CERDAS was designed to support BPS to implement its Strategic Plan for Change and Reform for the Development of Statistics (CERDAS), which also supported the implementation of the BPS Bureaucracy Reform initiative. The CERDAS plan was developed by BPS through extensive internal and external stakeholder consultations and was part of its medium-term strategy (Rencana Strategis/Renstra) that set out BPS’s improvement priorities to be implemented over a five-year period. Based on the CERDAS plan, BPS completed a BPS Modernization Proposal in 2010 with detailed reviews of the existing states of statistical methodologies, ICT, and HR. Because the ambitious reform would touch all parts of the BPS organization, the broad-based consultative approach was intended to foster shared vision and ownership within BPS—among the leadership and the rank and file of BPS, both in the headquarters and regional offices.

34. The technical design of STATCAP-CERDAS also involved BPS’s development partners, such as the ABS, Australia Indonesia Partnership for Economic Governance (AIPEG), Statistics Netherlands, United States Department of Agriculture-National Agricultural Statistics Services (USDA-NASS), and Statistics Korea. In particular, the ABS and BPS have maintained a long-term partnership for the last 20 years that includes peer-to-peer learning and technical advisory support, and its 2010–2015 MOU for cooperation was closely aligned to STATCAP-CERDAS to provide complementary TA during project implementation.

35. Policy and Human Resources Development (PHRD) project preparation grant was mobilized to support the detailed analysis and development of implementation requirements, but this also resulted in lengthy project preparation period. In addition, it was expected that the procurement documents for the initial year would be finalized with the support of the preparation consultant before effectiveness, but the products delivered by the consultant were deemed not ‘fit for purpose’ and updated by BPS. By project effectiveness, it had been almost three years since the conception of the project at BPS, and the project champion and initiator, the first Chief Statistician, was promoted to a new post in another ministry. Consequently, this resulted in the loss of reform momentum, because the leadership change occurred shortly after loan effectiveness and time elapsed in the transition.

36. Overall risks at preparation and during implementation were both rated as Medium Driven by Impact. The main risks identified were related to the implementing agency’s capacity and governance, including low capacity to manage complex ICT procurement and contracts and low capacity to manage culture change from ‘silo’ to ‘corporate’ culture. While the appropriate risks were identified, implementation agency capacity risks were underestimated and were indeed rated High during implementation. Turnover at the top significantly affected the implementation progress, and the change management interventions, as designed, were not able to ensure the stability of the reform momentum through these transitions.

37. At appraisal, BPS demonstrated a readiness to implement its reform agenda and modernization program. Notably, during project preparation, the CERDAS plan had been developed through extensive consultation process involving BPS leaders and the rank-and-file staff in its headquarters and regional offices, as well as external data users; STATCAP-CERDAS was aligned with the broader Bureaucratic Reform design; a detailed requirements analysis was conducted using the project preparation grant, as well as World Bank and donor TA support; and a PMU had been established with full-time staff. However, some weaknesses remained, including the temporary structure of the PMU and lackluster performance of the preparation consultant resulting in incomplete bidding documents by project effectiveness.

Page 26 of 72

B. KEY FACTORS DURING IMPLEMENTATION

Factors Subject to Government Control 38. Turnover in key counterparts affected project implementation progress despite the extensive consultation process during project preparation. The first Chief Statistician led the design and signing of the project, steered the project to effectiveness less than three months after loan signing, and was then promoted to become the Vice Minister for Agriculture. Subsequent leadership changes and transitions across the board at Echelon 1 and 2 levels resulted in implementation delays. Even though officials promoted up the rank had been involved in the preparation of the STATCAP-CERDAS, project implementation progress was sluggish. Shortly after effectiveness, BPS launched the procurement of the Owner’s Agent and statistical consultants, but both processes stalled after the short-listing stage. By the time of restructuring, the overall performance rating was Unsatisfactory with all project implementation activities, including procurement processes, put on hold, while project restructuring discussions took place between the World Bank and BPS.

39. Major implementation delays followed each change at the top, despite conscious engagement to involve other Echelon 1 officials and middle management in the reform dialogue and decision-making process. The project governance and management framework did not work optimally during STATCAP-CERDAS implementation. While the internal modernization plan was signed off by unit leads, readiness for reform was mostly lacking in the top management in the headquarters. Two key structures were designed at the pinnacle of the project governance framework, namely, the Steering Committee and the External Advisory Board. The Steering Committee was chaired by the Chief Statistician and comprised all eight Echelon 1 officials of BPS, to provide strategic guidance and decisions. The External Advisory Board was designed to provide a forum for key stakeholders of BPS to become involved in the reform process and guide the progress of STATCAP-CERDAS and was chaired by the Bappenas Deputy Minister, with members from ministries, the central bank, academia, and industry (Indonesian Chamber of Commerce [Kamar Dagang dan Industri Indonesia]). These bodies did not function optimally and often could not meet regularly as planned, due to busy schedules of their members.

40. A lack of a stable PMU and dedicated reform teams for implementing STATCAP-CERDAS also contributed to the slow progress. During the project lifetime, the PMU was reconstituted four times. Given its criticality, submission of the proposal to establish the PMU as a permanent unit—Directorate of Statistical Integration and Transformation—was included as a covenant in the loan agreement. BPS submitted the proposal to KemenPAN-RB but was not successful in negotiating a new structural unit until July 2014. Even then, a new Echelon 2 unit could not be added due to restrictions under the prevailing civil service regulations, so three existing Echelon 3 units were vacated to manage the three transformation areas: statistics, IT, and HR/organization. Only the statistical transformation unit staff reported directly to the Project Management Office (PMO) head. In addition, other technical staff were appointed to serve on project working groups and task forces with specific responsibilities, for example, finalizing the SBFA and working with consultant experts to develop the detailed designs for CSIs. However, these arrangements were not effective as staff also had their ‘business-as-usual’ tasks in practice, and working relationships between BPS staff and consultants were not optimized.

41. STATCAP-CERDAS did not provide sufficiently visible changes and tangible results for the rest of the organization early on, because most of the major process and organizational shifts could not be made until the IT tools and systems were in place first. The World Bank’s complementary TA activities also endeavored to address this challenge through providing some concrete demonstration projects (for example, metadata standardization and CAPI pilot), but they were not sufficient. BPS also did not utilize effectively the US$14 million counterpart fund to support activities to increase implementation capacity or to achieve quick wins. This lack of ‘quick wins’ contributed to waning

Page 27 of 72

ownership of the reform, despite the change management efforts through the network of change agents throughout the organization.

42. The project gained some traction after restructuring. Four key consultancy contracts were awarded after restructuring, and all project components moved into the implementation stage. With the Owner’s Agent on board in December 2014, the Chief Statistician led the relaunching of STATCAP-CERDAS in January 2015 to demonstrate the leadership commitment to BPS staff and external stakeholders. The Bappenas Minister expressed high-level support. Project activities started with the support of TA consultants, including change management and communications activities in headquarters and regional offices; the development of the SBFA with standardized business processes and specification of common statistical infrastructure; and the development of IT strategy, EA, and standard operating procedures (SOPs) for IT services.

43. It took one and a half years after project relaunching for BPS to finalize the technical requirements for the integrated statistical information systems development package, the core of the project to deliver the tools and systems required to support the business process and organizational changes. Delays in the selection of the Statistical consultant and its subsequent slow work progress to prepare technical requirements for the information systems package affected the timeline of the procurement. The preparation of the Terms of Reference (TOR) for the systems package required inputs from both the Statistical and IT Design consultants and was not finalized until July 2016. While the TOR was being finalized, BPS initiated the selection process by issuing the Request for Expressions of Interest (REOI) in February 2016 and managed to proceed with the procurement relatively quickly to completing the technical evaluation in less than four weeks (which is a remarkable performance for such a large-value, complex IT contract) but failed to submit the technical evaluation report by the end of October 2016 for the World Bank’s prior review. This made it impossible to maintain the contract implementation start date in January 2017 and project completion by November 2018 unattainable.

Factors Subject to World Bank Control 44. There were five task team leaders (TTLs) during the six-year project cycle of STATCAP-CERDAS. However, one key team member, who was one of the co-TTLs at project cancellation, remained with the project since its inception, maintaining project continuity and preserving institutional memory. All the TTLs were based in-country and provided close supervision and implementation support. There was conscious effort to incorporate lessons learned from other public sector reform projects in the portfolio, such as the Government Financial Management and Revenue Administration Project (GFMRAP) and the Project for Indonesian Tax Administration Reform (PINTAR). However, STATCAP-CERDAS encountered similar obstacles that could not be overcome.

45. A total of nine Implementation Status and Results Reports were filed during the life of STATCAP-CERDAS averaging almost twice a year, especially after restructuring. The task team had been proactive throughout the project life cycle to document progress and raise issues and challenges on time for management attention. Support missions were fielded to ensure timely resolution of problems encountered, and trust fund resources enabled the provision of complementary TA. However, the task team had not been agile in navigating the multiple leadership changes at BPS, resulting in delays in project implementation at each of these transitions. STATCAP-CERDAS remained a problem project for a prolonged period.

46. To improve BPS implementation capacity and accelerate implementation progress, the World Bank team provided enhanced implementation support throughout the project, from facilitating BPS in the preparation of TOR to selection processes including contract management of various consultancies. The World Bank team also provided procurement training to BPS teams, including training to the Procurement Committees for each selection process. During the selection process for the statistical IT package, when BPS was faced with the challenge of drawing the short

Page 28 of 72

list of qualified and capable firms for the complex information system package, the World Bank team provided comprehensive guidance and proposed solutions to BPS and proactively worked with BPS and its consultants to conduct the procurement process in an expedient manner.

47. The World Bank could have considered other remedial actions under the prevailing circumstances, given that the grounds for BPS’s proposed cancellation of the procurement process of the integrated statistical information systems package were inconsistent with the World Bank’s Consultant Guidelines. The World Bank, however, decided not to pursue any further remedial action related to the procurement process, considering that the project in any case was heading for cancellation due to BPS’s inability to meet the conditions for lifting suspension.

Factors Outside the Control of the Government 48. Indonesia enjoyed relatively stable macroeconomics, and there was no conflict nor natural disaster during the life of the STATCAP-CERDAS project.

C. MAIN EVENTS/FACTORS LEADING TO CANCELLATION 49. After restructuring in April 2014, three consultancies provided key inputs to prepare for the design and procurement of the integrated statistical information systems package, which accounted for over 50 percent of the restructured loan amount. These were the Owner’s Agent, Statistical Transformation and Quality Improvement TA consultant, and IT Strategy and Design TA consultant. With respect to the information systems package, the Statistical and IT consultants were to provide inputs on the technical requirements for the TOR, while the Owner’s Agent consultant was to support the preparation of the Request for Proposal (RFP) document. BPS eventually engaged an individual procurement consultant to support the systems procurement, because the Owner’s Agent failed to complement its technical specialists with a procurement specialist.

50. The World Bank also conducted implementation support missions in May, August, and September 2016, specifically to support the acceleration of the information systems procurement and keep the project on critical path. These missions provided both technical support for the BPS IT team and training for the procurement and evaluation committees. Training was also conducted in stages to focus on the particular phase of the procurement process, that is, overview training at the beginning of the procurement and one focused on technical evaluation before the RFP closing date.

51. BPS initiated the selection process for the information systems package by issuing the REOI in February 2016, while there were delays in preparing the TOR as described in paragraph 43. After determining only two firms to be qualified for short-listing, BPS proposed to readvertise the REOI to solicit wider interest from qualified firms, and the REOI was readvertised in April 2016, this time also providing more details of the assignment. By June 2016, BPS prepared an initial short list of firms, but upon further clarification of the evaluation criteria, most of those firms were found to be unqualified for this large and complex assignment. BPS sought the World Bank’s guidance as advertising the REOI for the third time would have resulted in further delay at this critical stage, while providing no assurance of a different outcome. The World Bank team provided comprehensive guidance and solution to BPS, and in August 2016, BPS completed the revised short list of six qualified international IT firms, which was provided for no objection by the World Bank. BPS proceeded with issuing the RFP in August 2016, and proposals from three firms were received in early October 2016. BPS completed the technical evaluation by late October 2016, within four weeks of receipt of proposals, which was a noteworthy performance for an evaluation of such complex nature. However, the Technical Evaluation Report was never sent to the World Bank, pending BPS management approval. The new Chief Statistician requested the Government’s internal audit agency (BPKP) to review the selection process carried out by BPS so far.

Page 29 of 72

52. Around this time, BPS’s aversion to risks around procurement appeared to be heightened, especially given the high complexity and large value of the package. BPS cited concerns over findings by the Supreme Audit Agency (Badan Pemeriksa Keuangan/BPK) for project audit covering 2014–2015, which considered previous selection processes to have procedural deficiencies, even though the World Bank had prior reviewed and issued no objections. Around this time, there was also an ongoing court case involving a BPS official over procurement unrelated to STATCAP-CERDAS. In addition to the new Chief Statistician taking office in September 2016, new officials were appointed to key senior and line manager positions responsible for IT (Deputy Chief Statistician and Director for IT), which also contributed to the break in institutional knowledge along the chain of command regarding the technical discussions and decisions taken with respect to the systems procurement before their appointments.

53. In October 2016, BPKP provided its observations on the procurement process, which included (a) issues on procedural compliance that the procurement committee had not obtained prior administrative approval of the cost estimate from the competent authority in the Government before initiating the selection process and the short-listing process was not in compliance with the World Bank’s Consultant Guidelines and (b) issues on technical aspects such as adequacy of the cost estimate, sufficiency of time provided for proposal submission and for contract execution, and readiness of the TOR. Furthermore, even though the World Bank had prior reviewed and provided no objection to BPS’s revised short list of six firms, BPKP provided its own interpretation that the short-listing process was not in compliance with the World Bank’s Consultant Guidelines. Specifically, BPKP questioned the inclusion of a third firm from the same country (instead of the maximum two from any one country) to meet the short list requirement of six firms, but in BPKP’s opinion the short list should have been reduced to five firms while the sixth and only remaining qualified firm should have been excluded from the short list. While the World Bank team clarified the above aspects, including explanation of the principles and intent of the World Bank’s Consultant Guidelines, and proposed possible actions to address the technical issues raised in the BPKP review, BPS proposed to cancel the selection process and to start a new selection process.

54. The World Bank did not agree with BPS’s proposal to restart the procurement process. After two months of impasse, the World Bank suspended the project in December 2016 with the following conditions for lifting the suspension: (a) signing the contract for the information systems package and (b) establishment of appropriate counterpart project team for its implementation. During the suspension period, BPS proposed either to relaunch the integrated statistical information system procurement process including updating the TOR and cost estimates and extending the project for an additional two years to close at the end of 2020 or to partially cancel the amount allocated to the package and extend the project for 2.5 years to allow for a 50-month in-house development of an integrated statistical information system financed through loan reallocations. The World Bank did not agree with either of the proposed options, although the team did explore other options with BPS, such as partial cancellation to allow the completion of selected activities under the HR and organization components.

55. Ministry of National Development Planning (Badan Perencanaan Pembangunan Nasional/Bappenas) and the MOF also tried to convene meetings with the National Public Procurement Agency (Lembaga Kebijakan Pengadaan Barang Jasa Pemeintah/LKPP) and BPKP in an effort to resolve the impasse but were unsuccessful in reaching an agreed way forward. Eventually, with no action from BPS to meet the conditions for lifting the suspension for three months, the World Bank initiated the project cancellation process. STATCAP-CERDAS was formally closed on May 31, 2017.

Page 30 of 72

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME

A. QUALITY OF MONITORING AND EVALUATION (M&E)

M&E Rating: Modest

M&E Design

56. The M&E design did not capture the full picture of the STATCAP-CERDAS implementation progress. The core of the M&E design was the Results Framework with original indicators and baselines collected in annex 1 of the Project Appraisal Document. Achievements at project cancellation were captured in annexes 1 and 7 of this ICR, as well as summarized in Table 2. The PDO was complex, compounded with multiple sub-objectives that could be unpacked in various permutations. This was further complicated by the achievements of the PDO-level outcome indicators, with some exceeding the targets, and the mostly underachieved intermediate results indicators. Of the 17 indicators, 8 were at the PDO level. These PDO indicators provided snapshot readings that did not necessarily capture sustained impacts of the project, compared to those for project-specific intermediate results. The measurements chosen for the PDO indicators were also influenced by improvement efforts made by BPS, as well as through TA from the World Bank and other development partners, notably the ABS. As noted earlier, this was especially the case because the STATCAP-CERDAS framework was initially used to harmonize external technical support. Therefore, some PDO indicators showed statistical quality improvements, despite the implementation lag and low disbursement of STATCAP-CERDAS project.

M&E Implementation 57. BPS had conducted surveys and studies to qualify and quantify achievements against the indicators in the Results Framework. All data were collected by the PMU. However, there was no sufficiently robust verification and validation process for the status reported by the subject matter areas. Changes in measurement methodology for user satisfaction surveys, which were not under the World Bank’s control, could mean the achieved values may not be directly comparable with the baseline. BPS also made changes to the methodologies for GDP and GDRP compilation, which resulted in corrections to the baseline value for discrepancy.

M&E Utilization 58. The World Bank was proactive in monitoring implementation progress, and midcourse adjustments were introduced based mainly on the day-to-day interactions between members of the World Bank’s task team and their respective BPS counterparts.

Justification of Overall Rating of Quality of M&E 59. Given the significant shortcomings in the M&E system, the overall rating of the quality of M&E is Modest.

Page 31 of 72

B. ENVIRONMENTAL SAFEGUARD, SOCIAL SAFEGUARD, AND FIDUCIARY COMPLIANCE

60. STATCAP-CERDAS was rated as an environmental category C with no social or environmental safeguards policies triggered.

61. The STATCAP-CERDAS project aimed to improve the statistics essential to support evidence-based decision making in determining government social policies and expenditures for the poor and vulnerable in such areas as health, education, social assistance, and protection. Improved gender-disaggregated statistics would also have supported the Government’s efforts for gender mainstreaming in planning and budgeting, as well as M&E of policy and program impacts by gender.

62. BPS was the implementing agency for STATCAP-CERDAS. Both project procurement and financial management capacity assessments were completed at appraisal. The project complied with fiduciary covenants during implementation with internal control arrangements in place.

C. BANK PERFORMANCE

Overall Rating: Moderately Satisfactory

Quality at Entry

63. STATCAP-CERDAS was a carefully designed project that took into consideration international practices and was fully aligned with BPS’s internal multiyear plan and reform priorities. Following lessons learned from global STATCAP experiences, the team worked with the client to start by taking stock of its statistical activities and to prioritize the actions to strengthen statistical capacity in the form of its NSDS and/or its statistical master plan. At appraisal, BPS had developed a strategic plan to improve product lines as well as ICT and HR areas. All Echelon 1 and 2 officials of BPS signed off on the strategic plan to signal their commitment. Even though a full-time PMU structure was established during the preparation phase, it was housed in a temporary structure that subsequently had to be dissolved within six months of effectiveness due to changes in bureaucracy reform rules.

64. The design of STATCAP-CERDAS also considered experiences of donor assistance to BPS but overestimated implementation capacity. Past donor assistance to BPS, including the World Bank’s, focused primarily on specific technical issues without dealing with organizational issues. Although this approach was successful to a degree, it limited sustained impacts. To avoid this outcome, STATCAP-CERDAS was designed focusing on structural reform that included elements such as ICT governance, HR policies, performance management, and organizational culture. However, the inclusion of three significant information systems implementations in the original scope was too ambitious in light of the limited implementation capacity and may have been better if they had been sequenced.

65. STATCAP-CERDAS was designed in consultation with other development partners to ensure synergies. The preparation of the project involved development partners, such as the ABS, AIPEG, Statistics Netherlands, USDA-NASS, and Statistics Korea. In addition, the MOU between the ABS and BPS for 2010–2015 on technical cooperation between the two organizations was deliberately aligned to the STATCAP-CERDAS objectives and areas of support.

66. STATCAP-CERDAS adopted international good practice for institutional reform projects by embedding change management from the design stage. The project was also designed through a broad-based consultative process with both internal and external stakeholders of BPS, intended to foster shared vision and ownership at the inception. Realizing at the outset that institutional reform was a long-term objective, built into the project were opportunities for short-term successes essential for gaining credibility and maintaining momentum of the reform

Page 32 of 72

processes. In addition, STATCAP-CERDAS was designed for BPS to support the achievement of the Government’s ongoing bureaucratic reform initiative. Accordingly, training and development of new skills to adopt the new business processes and use of new ICT applications were factored into the project components. However, the project did not safeguard staff turnover common to multiyear projects. As noted above, with the leadership changes at inopportune junctures of project implementation, STATCAP-CERDAS ultimately suffered from lack of organizational ownership resulting in implementation delays.

67. The project frontloaded detailed analysis and technical requirement specifications during project preparation, but the long preparation period resulted in loss of reform momentum. The long preparation period allowed the project preparation consultant to conduct detailed analysis and prepare the bidding documents ahead of project effectiveness. The use of the PHRD project preparation grant also prolonged the preparation period, because the recipient-executed procurement process for the project preparation consultant was lengthy, and the contract was extended several times to complete the deliverables. By project effectiveness, it had been almost three years since the conception of the project at BPS, and the initial project champion and initiator, the first Chief Statistician, was promoted to a new post in another ministry.

68. Given that official statistics and statistical information systems are highly specialized technical domains, the World Bank could have been more proactive in identifying and providing information on consultants successfully used in other similar World Bank projects—both during project preparation and supervision phases. Based on the lesson learned from the weak performance of the project preparation consultant, the statistical and IT systems consultancies were split into two packages. However, this was insufficient to attract qualified consultants, because BPS received weak responses for procurement of key consultancy packages under STATCAP-CERDAS. Both the statistical quality TA and statistical IT consultancy packages required two rounds of REOIs to constitute a short list of qualified consultants, and only two proposals were received out of four short-listed firms for the former, and three proposals out of six short-listed for the latter. Many NSOs, even those in advanced economies, have traditionally developed their own systems, so there are limited commercial-off-the-shelf software specific to statistical infrastructure and few experienced contractors.

Quality at Supervision 69. STATCAP-CERDAS was well resourced with country-based TTLs throughout its life, supplemented by subject matter expertise as needed. Subject matter expertise included statistics, IT, institutional/organizational reform, and change management. The task team also proactively provided advice and organized training to strengthen implementation capacity, particularly on project management, procurement, and financial management. TTLs also proactively engaged relevant stakeholders from other government institutions, such as Bappenas, the MOF, and KemenPAN-RB, in efforts to keep them informed and obtain their support.

70. Trust fund resources enabled the provision of complementary TA activities to provide small but concrete demonstrations of statistical integration or technology implementation in practice. These included support for: (a) developing the BPS Microdata Catalog System to standardize documentation and publication of survey microdata metadata; (b) piloting digital data collection methods through the use of CAPI platform, with Survey Solutions software developed by the World Bank; and (c) facilitating peer learning from similar modernization experiences, such as the Turkish Statistics Office. These contributed to some modest improvements but were ultimately not sufficient to sustain reform momentum.

71. The World Bank was proactive in trying to find solutions to address implementation issues, in particular those affecting procurement and financial management processes. Throughout the project, the World Bank team provided proactive procurement and financial management support to the implementation teams. BPS successfully

Page 33 of 72

completed procurement of four consultancy packages, and even the procurement of the highly complex information systems package was conducted remarkably quickly until the technical evaluation stage. The task team also provided close support to improve FM capacity. Finalization of clear guidelines on payment verification process, clarity of roles within the PMU, and close support from the task team helped the project to improve the timeliness of payment toward the end of the project period. In addition to the extensive hands-on procurement training provided by the World Bank to BPS before and during the selection process of the statistical information systems package, the World Bank team also provided procurement training to more than 200 auditors from BPKP as part of the capacity building of the auditors for their better understanding of the World Bank’s Procurement and Consultants Guidelines. However, the BPKP team, which reviewed the ongoing selection process at the request of BPS’s leadership, was different from the ones that had been provided procurement training by the World Bank.

72. Given similar experience with the auditors’ (BPKP or BPK) role in other projects and consequently failed procurements, the Government authorities and the World Bank could have done more to anticipate potential issues and provide clarity for implementation agencies. Particularly for complex information systems development, providing procurement training to implementing agencies is necessary but may not be sufficient, as also noted in the Note on Cancelled Operations (NCO) for PINTAR.

Justification of Overall Rating of Bank Performance 73. Based on the proactive implementation support and TA, as described earlier, the World Bank Performance overall is rated Moderately Satisfactory.

D. RISK TO DEVELOPMENT OUTCOME 74. Even though the project was not successful in deploying the IT system for the production and dissemination of statistics, it successfully financed key deliverables that lay the foundations for the development of the IT system with government funding. Among others, the four consultancies financed by the project supported BPS to produce the Statistical Business Framework and Architecture (SBFA) with new business processes, the IT strategy, the EA, and the detailed design documents of CSIs. As of the time of the ICR, BPS had secured state funding from the MOF over the next four years to continue its transformation program and implement the statistics information systems development.

V. LESSONS AND RECOMMENDATIONS

75. Sequencing long-term reform. Recognizing that public sector reform projects are long-term games, ensuring a phased approach could deliver visible incremental results throughout the project to sustain reform momentum. In this case, a series of short-term goals or quick-wins should be included in the project design, rather than rely solely on a linear, sequential ‘waterfall’ approach, which may only produce visible results toward the end of a long reform process. The phased approach will demonstrate incremental results and benefits and serve to build implementation confidence early on in the project. When working with complex reform processes, teams should also speed up project preparation to avoid losing reform momentum early on. The new Multiphase Programmatic Approach may be a suitable instrument to support this type of initiative, because it aims to provide more opportunities for adaptation than a single long/large operation, as well as greater continuity than a series of stand-alone operations.

76. Sustaining reform ownership. Committed and visible leadership is key to successful reform, and teams should expect and plan for changes in government leadership in long-term reform processes. When supporting long-term reforms, changes in government leadership can hamper the project outcomes by delaying implementation and undermining ownership in the transition. To manage this risk, it is advisable to both (a) build project ownership by engaging and involving staff at all levels of the organization and (b) identify and engage external champions that can

Page 34 of 72

increase mandate and support for reform to drive implementation forward. Embedding change management is critical in institutional transformation initiatives to ease angst during reform implementation and foster buy-in and ownership at all levels. In addition to strong champions within the organization, systematically identifying and engaging the key stakeholders outside the organization who will benefit from and have strong demand for successful transformation will be necessary to gain and sustain political support for the reform continuity.

77. Organizing reform delivery. To effectively deliver institutional transformation, dedicated and empowered teams are required to be isolated from the ‘business-as-usual’. Managing institutional transformation programs requires full-time, dedicated teams similar to ‘delivery units’, which report directly to the organization’s leadership and have the ability to coordinate across the whole organization. Agencies should have the flexibility to design their organization to meet this requirement. Then, appropriate performance incentives should be put in place to attract and secure the best qualified staff. These could range from career or capacity-building opportunities to financial incentives to recognize and reward performance of staff working on reform projects.

78. Planning for complex ICT procurement. Ensure that the prerequisite preparation requirements and enabling conditions are in place before starting the procurement process. ICT transformation is often a key component of public sector reform projects, but procurement of complex ICT such as the development of complex software systems has a high failure rate. Lessons and recommendations from previous operations include an observation from PINTAR that ICT procurement is different requiring both specialized knowledge in ICT and competency in procurement. For projects with ICT-enabled management changes, the World Bank team should give greater attention to ensuring technical and institutional readiness by the project implementing agency. Considering many agencies do not possess sufficient capacity in managing the technical and commercial aspects of large and complex IT system procurement, the World Bank should also explore with the implementing agency on the possibility of drawing the required technical and procurement expertise from specialized departments to supplement the procurement committee, in addition to the external consultant support. In case such expertise is not yet well developed, the Government can also consider developing this function centrally within the competent technical agency, with a view to providing TA to any agency planning procurement and development of complex business or back-office systems.

79. Reorienting audit focus on project performance. The World Bank should work with the Government through training to reorient the auditors’ focus from strict procedural compliance toward achieving project performance and development outcomes. The focus of auditors on strict procedural compliance has undermined project outcomes of multiple World Bank operations by delaying or blocking complex, especially ICT-related, procurement processes. GFMRAP recommended that the World Bank should work proactively with the Government and the auditors through training to help understand the full scope of the World Bank’s Procurement Guidelines to avoid differing interpretations. The World Bank should also explore whether the provisions of loan agreement can be strengthened to make it explicit that the World Bank’s decision on the interpretation of the World Bank’s Procurement/Consultant Guidelines will be final and binding.

80. Targeting niche markets. Capacity-building projects for specialized agencies, such as NSOs, face common challenges such as weak responses to REOIs and weak consultants, caused by insufficiently developed markets for boutique technical skills. Implementing agencies should conduct active scans of relevant markets to identify the best consultants, individuals, and firms and proactively inform them of business opportunities under the project. World Bank supervision teams can also help by providing information on consultants successfully used in other similar World Bank projects.

.


Page 35 of 72

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS

A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Increased effectiveness of the Indonesian National Statistics Office (BPS-Statistics Indonesia) to produce and disseminate quality statistics

Indicator Name Unit of Measure Baseline Original Target Formally Revised

Target

Actual Achieved at Completion

Accuracy. Demonstrable improvement in the accuracy of key statistics as evidenced by: (a) an increase in the response rate of the Annual Survey of large and medium manufacturing businesses

Percentage 60.00 83.00 74.00 69.54

07-Apr-2011 31-Dec-2014 31-Dec-2018 31-Dec-2016

Comments (achievements against targets):


Target


Accuracy. Demonstrable improvement in the accuracy of key statistics, as evidenced by: (b) an increase in the user satisfaction with the accuracy

Percentage 43.00 75.00 75.00 91.73

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016


Page 36 of 72

of BPS statistics



Target


Relevance. Availability of important new statistics as evidenced by establishment of a national producer price index

Text PPI available for agricultural sector, excluding forestry sub-sector

PPI available for all provinces and most large districts

PPI available for all provinces and districts

National PPI published for the following additional service sectors: Accommodations, Food and Beverage, Passenger Transportation, Electricity and Gas Supply, and Water Supply. Study conducted for the following additional sectors: Freight Transportation, Telecommunications, Financial Services, Education, Health and Waste Management. Preparation for Provincial PPI has been conducted with producer prices survey data collection for


Page 37 of 72

goods and services sectors expanded to all provinces in 2016, and piloting of Provincial PPI for 5 provinces is in progress.

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016

Comments (achievements against targets): National PPI sector coverage has expanded, and data collection has expanded to all provinces to enable Provincial PPI calculation.


Target


Coherence. An improvement in the coherence of BPS statistics as evidenced by a reduction in the discrepancy between the sum of the Provincial estimates of GDRP and the GDP for Indonesia

Percentage 0.08 3.00 3.00 2.03

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016

Comments (achievements against targets): Baseline was revised from 27% by BPS, as discrepancy was made 0 (zero) in 2010


Target


Interpretability. An Text No statistics with fully Metadata available for Metadata available for 310 datasets with


Page 38 of 72

improvement in the interpretability of BPS statistics as evidenced by an increase in the statistics produced by BPS that have supporting meta data

documented metadata

National Accounts National Accounts metadata information published in BPS Microdata Catalog System

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016

Comments (achievements against targets): BPS makes metadata accessible through the Microdata Catalog System (MCS) and Statistics Referral System (SiRuSa)

Objective/Outcome: Increased efficiency of the Indonesian National Statistics Office (BPS-Statistics Indonesia) in producing and disseminating quality statistics


Target


Timeliness. Improvements in the timeliness of key statistics as evidenced by reducing the time lag in the publication of annual labor force estimates

Number 150.00 60.00 60.00 90.00

07-Apr-2011 31-Dec-2015 31-Dec-2016 31-Dec-2016

Comments (achievements against targets): Timelag reduced to 90 days in line with timeline for GDP publication


Target


Accessibility. An improvement in the accessibility of BPS statistics as evidenced by: (a) a

Number 51000.00 91800.00 91800.00 127147.00

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016


Page 39 of 72

dynamic up-to-date web site



Target


Accessibility. An improvement in the accessibility of BPS statistics as evidenced by: (b) an increase in the user satisfaction with the accessibility of BPS statistics

Percentage 61.00 80.00 80.00 83.47

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016


A.2 Intermediate Results Indicators

Component: Component A: Improved Statistical Quality


Target


The successful implementation of a Quality Assurance Framework across BPS

Text No corporate quality assurance policy or program

Quality assurance program established and effective

Quality assurance program established and effective

Quality governance structure (unit) has been discussed and may be established in 2017; study of quality gates conducted for


Page 40 of 72

Large and Medium Manufacturing Industries Statistics.

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016



Target


The establishment of a Common Business Register (BR) and a Large Business Unit (LBU)

Text No BR or LBU available Review of effectiveness of BR maintenance strategy

Review of effectiveness of BR maintenance strategy

Detailed Design Document for the Statistical Business Register is in finalization phase.

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016



Target


Availability of an integrated data and information service system supported by the data warehouse

Text No integrated data & information system supported by data warehouse available

Integrated data & information system supported by data warehouse available

Integrated data & information system supported by data warehouse available

Statistical Business Framework and Architecture (SBFA), IT Strategy and Enterprise Architecture (EA) have


Page 41 of 72

been finalized and endorsed by BPS Executives. Business and functional requirements for Corporate Statistical Infrastructure (CSIs) developed and detailed specifications documents are in finalization phase.

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016


Component: Component B: Improved ICT Platform and Statistical Information Management Systems


Target


Improvement in ICT Governance and Management

Text ICT Strategy in BPS Strategic Plan 2009-2014; Existing ICT functions in the Regulation of the Head of BPS No.1/2009 on BPS Organization; Initial business process mapping and analysis

All standards available and routinely used as part of "business as usual"

Improvement of ICT Strategy, and ICT Governance & Management Framework based on the initial experience and feedback from internal stakeholders

BPS IT Strategy, IT Governance and Management Framework / Policy, IT Service Management (Policies, Standards and Procedures), and IT Transformation Roadmap have been finalized and endorsed


Page 42 of 72

in BPS Analysis Document 2010-2011

by BPS Executives.

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016



Target


Development of application systems for statistical production and dissemination

Text Systems for common statistical infrastructure do not exist; Statistical processing systems developed by SIS and subject matter areas available

N/A - not an indicator originally

Application systems for all statistical infrastructure available and in use

Business and functional requirements for Corporate Statistical Infrastructure (CSIs) developed based on agreed BPS Statistical Business Framework and Architecture (SBFA). Detailed specifications documents for CSIs in finalization phase.

07-Apr-2011 31-Dec-2016 30-Nov-2018 31-Dec-2016


Component: Component C: Improved Human Resources Management and Development


Page 43 of 72


Target


A Human Resource that is aligned with BPS's future strategic direction

Text No HR Strategic Plan is available; No Technical and Non-Technical Competency Framework and Dictionary are available

A workforce that is aligned with BPS's strategic direction

Improvement of BPS HR Strategic Plan based on the initial experience and feedback from internal stakeholders; Implementati on of improved HR Strategic Plan

Review of current BPS capability, tools, and regulations (high level) towards the new business process (SBFA) completed, including core & managerial competencies, workload analysis, regulation review, training & development strategy and readiness for new business process. Draft required core & managerial competencies for future BPS are completed, but not yet endorsed by the management.

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016


Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at


Page 44 of 72

Target Completion

An effective, integrated performance management system for BPS

Text A performance system that is not in line with BPS's future strategic direction and does not provide sufficient incentive for improved performance

An effective performance management system for BPS

An effective Performance Management System for BPS

BPS-wide Strategy Map completed. Key Performance Indicators using Balanced Scorecard for Echelon I and II are completed, but not yet endorsed by the management.

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016


Component: Component D: Improved Organizational Alignment


Target


Improved alignment of the BPS organization and the future work environment

Text BPS organization alignment is not yet optimal

An organization design that is well aligned with the BPS's strategic direction

Improvement of the organizational design based on the initial experience and feedback from internal stakeholders

Organization Structure at macro level (echelon I and II) based on GSBPM is completed, but not yet endorsed by the management.

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016


Page 45 of 72


Component: Component E: Project Management


Target


Effective project management

Text No corporately agreed project management system

Review of the application of the BPS Project Management System

Review of the application of the BPS Project Management System

227 BPS managers have been trained in project management

07-Apr-2011 31-Dec-2016 31-Dec-2018 31-Dec-2016


B. ORGANIZATION OF THE ASSESSMENT OF THE PDO

Objective/Outcome 1. Increased effectiveness of BPS in producing and disseminating quality statistics

Outcome Indicators

1. Accuracy. Demonstrable improvement in the accuracy of key statistics as evidenced by (a) an increase in the response rate of the Annual Survey of large and medium manufacturing businesses 2. Accuracy. Demonstrable improvement in the accuracy of key statistics, as evidenced by (b) an increase in the user satisfaction with the accuracy of BPS statistics 3. Relevance. Availability of important new statistics as evidenced by establishment of a national producer price index 4. Coherence. An improvement in the coherence of BPS statistics as evidenced by a reduction in the discrepancy between the sum of the Provincial estimates of GDRP and the GDP for Indonesia


Page 46 of 72

5. Interpretability. An improvement in the interpretability of BPS statistics as evidenced by an increase in the statistics produced by BPS that have supporting meta data

Intermediate Results Indicators

1. The successful implementation of a Quality Assurance Framework across BPS 2. The establishment of a Common Business Register (BR) and a Large Business Unit (LBU) 3. Availability of an integrated data and information service system supported by the data warehouse 4. A Human Resource that is aligned with BPS's future strategic direction 5. An effective, integrated performance management system for BPS

Key Outputs by Component (linked to the achievement of the Objective/Outcome 1)

Component A 1. BPS Statistical Business Framework and Architecture (SBFA) and Implementation Roadmap 2. Detailed Design Documents for Corporate Statistical Infrastructure (CSIs) (for example, data collection, processing, analysis and dissemination systems; metadata management system; frame and register systems; database and data warehouse) 3. Methodological advice on sampling methodologies for continuous household surveys and for pilot of Integrated Horticulture and Estate Crops Survey 4. Establishment of BPS Microdata Catalog System for documentation and publication of survey metadata (http://microdata.bps.go.id) 5. Study visits to NSOs on institutional transformation and statistical integration (Australia, New Zealand, Canada, Turkey, Italy, UNECE) 6. Technical training (for example, Short course on Sampling Methodologies for Survey Statisticians in University of Michigan, Agriculture statistics training by USDA-NASS, Peer learning workshop on GSBPM implementation by the Turkish Statistical Institute) Component C 7. Gap Analysis Report on Performance Management, Individual and Team Capability, Reward and Recognition, HR Functions, and Training and Development 8. BSC-based Performance Management System for Corporate (until Echelon II Level) 9. Job Family Design 10. Core and Managerial Competency Model 11. Individual Performance Management Policy, covering Performance Planning, Implementation and Evaluation 12. Training and Development Strategy and Plan 13. Technical Training (for example, Training Needs Analysis, Assessors, Balanced Scorecard)

http://microdata.bps.go.id/


Page 47 of 72

Change Management 14. Project Launch Workshop in January 2015 15. Change Visioning Workshop in February 2015 16. Planning of Change Champion Kick-Off in March 2015 17. Change Management Report 2015 and 2016 18. Change Progress Online Census (CPOC) - Report 2015 and 2016 19. Change Management Program and Work Plan (Quarterly - 2015 and 2016) 20. Induction workshops for BPS staff joining STATCAP-CERDAS PMO, Working Groups and Task Forces in 2015 21. Change Agent Network and Training with identified Change Champions established in HQ and regional offices in June 2015 22. Training on Change Management in HQ and regional offices (2015–2016) 23. BPS Leaders Session in March 2016 24. Change Management Plan and Strategy in September 2016

Objective/Outcome 2. Increased efficiency of BPS in producing and disseminating quality statistics

Outcome Indicators

1. Timeliness. Improvements in the timeliness of key statistics as evidenced by reducing the time lag in the publication of annual labor force estimates 2. Accessibility. An improvement in the accessibility of BPS statistics as evidenced by (a) a dynamic up-to-date website 3. Accessibility. An improvement in the accessibility of BPS statistics as evidenced by (b) an increase in the user satisfaction with the accessibility of BPS statistics

Intermediate Results Indicators

1. Improvement in ICT Governance and Management 2. Development of application systems for statistical production and dissemination 3. Improved alignment of the BPS organization and the future work environment 4. Effective project management

Key Outputs by Component (linked to the achievement of the Objective/Outcome 2)

Component A 1. SAKERNAS CAPI pilot using Survey Solutions; and workshop on lessons learned from 2012–2016 CAPI pilots in BPS 2. Technical training (for example, CAPI-Survey Solutions training) Component B 3. BPS IT Strategy 4. BPS Enterprise Architecture


Page 48 of 72

5. IT Governance and Services Management—Policies and SOPs 6. BPS IT Transformation Roadmap 7. Technical Training (for example, Enterprise Architecture [TOGAF, RSA]; IT Governance [COBIT]; IT Service management [ITIL]; IT Project Management and Software Development [Agile]; Data Management) Component D 8. Gap Analysis Report on Organization and Job Design 9. High-Level Work Load Analysis 10. High-Level Organization Design (including Regional Offices) Component E 11. Establishment of project website as socialization platform (http://statcapcerdas.bps.go.id) 12. Development of Collaboration Tool Platform 13. Training for PMO and project staff (for example, Training in Organizational Project Management and PMO for Public Organizations, Rapid Results Initiatives Coaching Workshop, Project Management and MS Project, Monitoring and Evaluation, Risk management, Business English Courses, Procurement, Financial Management)

http://statcapcerdas.bps.go.id/


Page 49 of 72

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION

A. TASK TEAM MEMBERS

Name Role

Preparation

Vivi Alatas TTL

Budi Permana Procurement Specialist

Christina I. Donna Financial Management Specialist

Melinda Good Country Lawyer

Vivianti Rambe Environmental Safeguards Specialist

Amien Sunaryadi Governance and Anticorruption Specialist

Dayu Amurwanti Governance and Anticorruption Specialist

Shubham Chaudhuri Sector Manager

William Wallace Team Member

Yoichiro Ishihara Team Member

Haeduck Lee Team Member

Noriko Toyoda Team Member

Roksana Khan Team Member

Erwin Ariadharma Team Member

Supervision/ICR

Noriko Toyoda, Erwin Ariadharma TTLs

C. Bernard Myers TTL

Theo David Thomas TTL

Achmad Zacky Wasaraka Procurement Specialist

Budi Permana Procurement Specialist

I Gusti Ngurah Wijaya Kusuma Financial Management Specialist

Christina Donna Financial Management Specialist

Ahsan Ali Team Member

Yash Gupta Team Member


Page 50 of 72

Tina George Team Member

William S. Hebb Team Member

Agus Suherman Team Member

Lina Lo ICR Author

Nicholas Menzies ICR Team Member

Carlos C. D. Ferreira ICR Team Member

B. STAFF TIME AND COST

Stage of Project Cycle Staff Time and Cost

No. of staff weeks US$ (including travel and consultant costs)

Preparation

FY07 1.600 8,722.29

FY08 13.850 114,105.12

FY09 50.200 205,614.11

FY10 13.462 230,706.82

FY11 10.328 166,952.05

FY12 0 43,608.95

FY13 6.371 36,284.43

FY14 .404 2,305.58

FY15 0 3,329.75

FY17 0 1.14

Total 96.22 811,630.24

Supervision/ICR

FY12 6.950 36,267.38

FY13 3.250 48,655.42

FY14 22.175 123,794.78

FY15 37.355 113,970.47

FY16 39.878 127,363.49

FY17 21.462 85,883.64


Page 51 of 72

FY18 6.435 38,782.40

Total 137.51 574,717.58


Page 52 of 72

ANNEX 3. PROJECT COST BY COMPONENT

Components Amount at Approval

(US$, millions) Actual at Project Closing

(US$, millions) Percentage of Approval

(%)

Component A: Improved Statistical Quality 26.28 2.86 10.9

Component B: Improved ICT Platform and Statistical Information Management Systems

24.00 0.63 2.6

Component C: Improved Human Resources Management and Development (including BPS Staff Training and Capacity Building) + Component D: Improved Organizational Alignment

11.03 1.28 11.6

Component E: Project Management 3.69 3.69 100.0

Total 65.00 8.46 125.10


Page 53 of 72

ANNEX 4. EFFICIENCY ANALYSIS

N/A


Page 54 of 72

ANNEX 5. BORROWER’S COMPLETION REPORT AND/OR COMMENTS


Page 55 of 72


Page 56 of 72


Page 57 of 72


Page 58 of 72


Page 59 of 72


Page 60 of 72


Page 61 of 72


Page 62 of 72


Page 63 of 72


Page 64 of 72


Page 65 of 72


Page 66 of 72

ANNEX 6. SUPPORTING DOCUMENTS

• Project Appraisal Document, Report No: 60319-ID, dated March 15, 2011

• Project Paper, Report No: 87045 v1, dated April 8, 2014

• Implementation Status and Results Reports and Implementation Support Mission Aide Memoires - April 2012 through May 2017

• Borrower’s Completion Report and Results Framework, dated October 27, 2017


Page 67 of 72

ANNEX 7. RATINGS OF INDICATORS BY SUB-OBJECTIVES

Indicator Target (Nov 30, 2018)/Baseline (April 11, 2011)

Achieved (May 31, 2017a)

Rating according to IEG Guidelines


1. Increased effectiveness of BPS in producing and disseminating quality statistics


Accuracy: Demonstrable improvement in the accuracy of key statistics

1. An increase in the response rate of the Annual Survey of large and medium manufacturing businesses

Target: 74% Baseline: 60%

69.54% X

2. An increase in the user satisfaction with the accuracy of BPS statistics


91.73% X

Relevance: Availability of important new statistics

3. Availability of important new statistics as evidenced by establishment of a national producer price index (PPI)

Target: PPI available for all provinces and most large districts Baseline: PPI available for agricultural sector, excluding forestry subsector

National PPI published for the following additional service sectors: Accommodation, Food and Beverage, Passenger Transportation, Electricity and Gas Supply, and Water Supply. Study conducted for the following additional service sectors: Freight Transportation, Telecommunications, Finance, Education, Health, and Waste Management. Producer prices survey data collection expanded to all provinces since January 2016, and piloting of provincial PPI for 5 provinces in progress.

X

Coherence

4. An improvement in the coherence of BPS statistics as evidence by a reduction in the

Target: 3% Baseline: 0.08% ( from original

2.03% X


Page 68 of 72





discrepancy between the sum of the Provincial estimates of GDRP and the GDP for Indonesia

baseline of 27% as corrected by BPS)

Interpretability

5. An improvement in the interpretability of BPS statistics as evidenced by an increase in the statistics produced by BPS that have supporting meta data

Target: Metadata available for National Accounts Baseline: No statistics with fully documented metadata

BPS Microdata Catalog System established and 310 data sets with metadata information published.

X

Total for PDO Sub-objective 1: n.a. n.a. 1 2 1 1

Percentage of PDO Indicators for Sub-objective 1:

n.a. n.a. 20 40 20 20


Improved Statistical Quality

6. Successful implementation of a Quality Assurance Framework across BPS

Target: Quality assurance program established and effective Baseline: No corporate quality assurance policy or program

Quality governance structure developed, has been discussed, and may be established in 2017; study of quality gates conducted for Large and Medium Manufacturing Industries Statistics.

X

7. The establishment of a Common Business Register (BR) and a Large Business Unit (LBU)

Target: Review of effectiveness of BR maintenance strategy Baseline: No BR or LBU available

Detailed design document for the Statistical Business Register has been finalized.

X

8. Availability of an integrated data and information service system supported by the data warehouse

Target: Available Baseline: No integrated data and information system supported by data warehouse available

EA and SBFA have been finalized and agreed by BPS executives. Detailed design documents for the working survey databases, input, and dissemination data warehouses are in finalization phase.

X


Page 69 of 72





Improved Human Resource Management and Development

9. A Human Resource that is aligned with BPS’s future strategic direction

Target: Improvement of BPS HR Strategic Plan based on the initial experience and feedback from internal stakeholders; implementation of improved HR Strategic Plan; Baseline: No HR Strategic Plan is available; No Technical and Non-Technical Competency Framework and Dictionary are available.

Review of current BPS capability, tools, and regulations (high level) toward the new business process (SBFA) completed, including core and managerial competencies, workload analysis, regulation review, training and development strategy, and readiness for new business process. Draft required core and managerial competencies for future BPS completed (not yet endorsed by BPS executives).

X

10. An effective, integrated performance management system for BPS

Target: An effective performance management system for BPS; Baseline: A performance system that is not in line with BPS’s future strategic direction and does not provide sufficient incentive for improved performance

BPS-wide Strategy Map completed. KPIs using Balanced Scorecard for Echelon 1 and 2 are completed (not yet endorsed by BPS executives).

X

Total for Intermediate Results for Sub-objective 1:

n.a. n.a. 0 1 4 0

Percentage of Intermediate Results Indicators for Sub-

objective 1: n.a. n.a. 0 20 80 0


Page 70 of 72





2. Increased efficiency of BPS in producing and disseminating quality statistics


Timeliness

11. Improvements in the timeliness of key statistics as evidenced by reducing the time lag in the publication annual labor force estimates

Target: 60 days Baseline: 150 days

90 days X

Accessibility

12. An improvement in the accessibility of BPS statistics as evidenced by a dynamic up-to-date website

Target: 91,800 unique visitors per month Baseline: 51,000 unique visitors per month

127,147 unique visitors per month

X

13. An improvement in the accessibility of BPS statistics as evidenced by an increase in the user satisfaction with the accessibility of BPS statistics


83.47% X

Total for PDO Sub-objective 2: n.a. n.a. 2 0 1 0

Percentage of PDO Indicators for Sub-objective 2:

n.a. n.a. 67 0 33 0


Improved ICT Platform and Statistical Information Management Systems

14. Improvement in ICT Governance and Management

Target: Improvement of ICT Strategy, and ICT Governance and Management Framework based on the initial experience and feedback from internal; stakeholders Baseline: ICT Strategy in BPS Strategic Plan 2009–2014; Existing ICT functions in the Regulation of the

IT Strategy, EA, and IT Transformation Roadmap have been finalized and agree by BPS executives. SOPs for IT services management have been prepared.

X


Page 71 of 72





Head of BPS No. 1/2009 on BPS Organization; Initial business process mapping and analysis in BPS Analysis Document 2010–2011

15. Development of application systems for statistical production and dissemination

Target: Application systems for all statistical infrastructure available and in use; Baseline: Systems for common statistical infrastructure do not exist; Statistical processing systems developed by SOP and subject matter areas available.

Business and functional requirements for CSIs developed based on agreed BPS SBFA. Detailed specifications documents for CSIs in finalization phase

X

Improved Organizational Alignment

16. Improved alignment of the BPS organization and the future work environment

Target: Improvement of the organizational design based on the initial experience and feedback from internal stakeholders; Baseline: BPS organization alignment is not yet optimal.

Macro-level organization structure (Echelon 1 and 2) based on GSBPM is completed (not yet endorsed by BPS executives).

X

Project Management

17. Effective project management

Target: Review of the application of the BPS project management system

227 BPS managers have been trained in project management.

X


Page 72 of 72





Baseline: No corporately agreed project management system

Total for Intermediate Results Indicators for Sub-objective 2:

n.a. n.a. 0 0 4 0

Percentage of Intermediate Results Indicators for Sub-

objective 2: n.a. n.a. 0 0 100 0

Note: a. Data from BPS.