Document of The World Bank · 2016. 7. 14. · Component 1: d) O&M arrangements are in place and/ or functioning for more than 85% ... (PNPM Generasi) in treatment areas are 0.0407

Document of

The World Bank

Report No: ICR00003281

IMPLEMENTATION COMPLETION AND RESULTS REPORT

(IBRD-80790)

ON

A LOAN

IN THE AMOUNT OF US$ 531.19 MILLION

TO THE

REPUBLIC OF INDONESIA

FOR THE

FOURTH NATIONAL PROGRAM FOR COMMUNITY EMPOWERMENT

IN RURAL AREAS

January 27, 2015

Social, Urban, Rural and Resilience Global Practice

Indonesia Social Development Unit

East Asia and Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

ii

CURRENCY EQUIVALENTS

(Exchange Rate Effective June 30, 2014)

Currency Unit = Rupiah

IDR 1.00 = US$ 0.000083

US$ 1.00 = IDR 11,990

FISCAL YEAR

January 1 – December 31

ABBREVIATIONS AND ACRONYMS

ASF Administrative Services Firm

BAPPENAS Badan Perencanaan Pembangunan Nasional (National Planning Board)

BKPG Bantuan Keuangan Pemakmue Gampong (Village Prosperity Grant)

CPS Country Partnership Strategy

CDD Community-driven Development

EIRR Economic Internal Rate of Return

FM Financial Management

GoI Government of Indonesia

IBRD International Bank for Reconstruction and Development

ICR Implementation Completion and Results

IDR Indonesian Rupiah

IRI Intermediate Results Indicator

ISR Implementation Status Results

IGSES Implementation Guidelines for Social and Environmental Safeguards

JMC Joint Management Committee

KDP Kecamatan Development Project

KPI Key performance Indicator

M&E Monitoring and Evaluation

MIS Management Information System

NGO Non-government Organization

NMC National Management Consultant

O&M Operations and Maintenance

PDO Project Development Objective

PMD Pemberdayaan Masyarakat Desa (Village Community Empowerment)

PNPM Program Nasional Pemberdayaan Masyarakat (National Community

Empowerment Program)

PSF PNPM Support Facility

PTO Petunjuk Teknis Operasional (Operational Technical Guideline)

RESPEK Rencana Strategis Pembangunan Kampung (Strategic Plan for Village

Development)

RMC Regional Management Consultant

UPP Urban Poverty Project

iii

Regional Vice President: Axel van Trotsenburg

Country Director: Rodrigo A. Chaves

Senior Global Practice Director Ede Jorge Ijjasz-Vasquez

Acting Practice Manager: Kevin Tomlinson

Project Team Leader: Sonya Woo

ICR Team Leader: Lily Hoo

iv

INDONESIA

FOURTH NATIONAL PROGRAM FOR COMMUNITY EMPOWERMENT

IN RURAL AREAS

CONTENTS

Data Sheet

A. Basic Information ........................................................................................................... v

B. Key Dates ....................................................................................................................... v C. Ratings Summary ........................................................................................................... v

D. Sector and Theme Codes ............................................................................................... vi E. Bank Staff ...................................................................................................................... vi

F. Results Framework Analysis ........................................................................................ vii G. Ratings of Project Performance in ISRs ....................................................................... xi H. Restructuring (if any) .................................................................................................... xi

I. Disbursement Profile .................................................................................................... xii

1. Project Context, Development Objectives and Design ................................................. 1 2. Key Factors Affecting Implementation and Outcomes ................................................ 6 3. Assessment of Outcomes ............................................................................................ 11

4. Assessment of Risk to Development Outcome ........................................................... 18

5. Assessment of Bank and Borrower Performance ....................................................... 19 6. Lessons Learned.......................................................................................................... 21 7. Comments on Issues Raised by Borrower/Implementing Agencies/Donors .............. 22

Annex 1. Project Costs and Financing .............................................................................. 23 Annex 2. Outputs by Component...................................................................................... 24

Annex 3. Economic and Financial Analysis ..................................................................... 31 Annex 4. Loan Preparation and Implementation Support/Supervision Processes ............ 33 Annex 5. Beneficiary Survey Results ............................................................................... 35 Annex 6. Stakeholder Workshop Report and Results ....................................................... 38 Annex 8. Comments of Cofinanciers ................................................................................ 45

Annex 9. List of Supporting Documents .......................................................................... 46

MAP IBRD33420

v

A. Basic Information

Country: Indonesia Project Name:

FOURTH NATIONAL

PROGRAM FOR

COMMUNITY

EMPOWERMENT IN

RURAL AREA (PNPM

IV)

Project ID: P122810 L/C/TF Number(s): IBRD-80790

ICR Date: 11/04/2014 ICR Type: Core ICR

Lending Instrument: SIL Borrower: REPUBLIC OF

INDONESIA

Original Total

Commitment: USD 531.19M Disbursed Amount: USD 530.77M

Revised Amount: USD 531.19M

Environmental Category: B

Implementing Agencies:

Directorate General of Village Community Empowerment (PMD) Ministry of Home Affairs

Cofinanciers and Other External Partners:

B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 03/23/2011 Effectiveness: 09/01/2011

Appraisal: 05/31/2011 Restructuring(s):

Approval: 07/14/2011 Mid-term Review: 03/18/2013 11/26/2012

Closing: 06/30/2014 06/30/2014

C. Ratings Summary

C.1 Performance Rating by ICR

Outcomes: Moderately Satisfactory

Risk to Development Outcome: Moderate

Bank Performance: Satisfactory

Borrower Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)

Bank Ratings Borrower Ratings

Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory

Quality of Supervision: Satisfactory Implementing

Agency/Agencies: Moderately Satisfactory

vi

Overall Bank

Performance: Moderately Satisfactory

Overall Borrower

Performance: Moderately Satisfactory

C.3 Quality at Entry and Implementation Performance Indicators

Implementation

Performance Indicators

QAG Assessments

(if any) Rating

Potential Problem

Project at any time

(Yes/No):

No Quality at Entry

(QEA): None

Problem Project at any

time (Yes/No): No

Quality of

Supervision (QSA): None

DO rating before

Closing/Inactive status:

Moderately

Satisfactory

D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

General water, sanitation and flood protection sector 19 19

Irrigation and drainage 19 19

Pre-primary education 19 19

Rural and Inter-Urban Roads and Highways 19 19

Sub-national government administration 24 24

Theme Code (as % of total Bank financing)

Child health 25 25

Other rural development 12 12

Participation and civic engagement 13 13

Population and reproductive health 25 25

Rural services and infrastructure 25 25

E. Bank Staff

Positions At ICR At Approval

Vice President: Axel van Trotsenburg James W. Adams

Country Director: Rodrigo A. Chaves Stefan G. Koeberle

Practice Manager/Manager: Kevin A Tomlinson Franz R. Drees-Gross

Project Team Leader: Sonya Woo Susanne Holste

ICR Team Leader: Lily Hoo

ICR Primary Author: Andre Oosterman

vii

F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document) The Project Development Objective (PDO) is for villagers in PNPM rural locations to

benefit from improved socio-economic and local governance conditions

Revised Project Development Objectives (as approved by original approving authority)

Not applicable

(a) PDO Indicator(s)

Indicator Baseline Value

Original Target

Values (from

approval

documents)

Formally

Revised

Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 1 :

Improved HH expenditure rates and improved access to economic and social

services in a minimum of 4,800 sub-districts in 2011 (impacts taken from

representative sample)

Value

quantitative or

Qualitative)

2% increase above

control group per project

cycle (based on previous

evaluation of KDP2)

More than and

equal to 2.0% 9.1%

Date achieved 06/14/2011 07/15/2011 12/31/2011

Comments

(incl. %

achievement)

According to an impact evaluation completed in 2011, during 2007-2009 the

increase in the consumption rate of poor households in project locations was

9.1%, substantially higher than the target in PAD (2.0%).

Indicator 2 : EIRRs more than 30% for major rural infrastructure types

Value

quantitative or

Qualitative)

EIRR between 39%-68%

for KDP2 More than 30% 35%-50%

Date achieved 06/14/2011 07/15/2011 06/30/2014

Comments

(incl. %

achievement)

A study undertaken in 2012 found that the economic internal rate of return (EIRR)

of a sample of 48 small-scale infrastructure projects financed by predecessor

projects during 2007-2011 ranged from 35% to 50%, which was higher than that

the targeted EIRR of 30%.

Indicator 3 : More than 80% satisfaction levels from beneficiaries regarding improved

services and local level governance

Value

quantitative or

Qualitative)

Previous satisfaction

levels more than 80% for

KDP2

More than 80% 92.4%

Date achieved 06/14/2011 07/15/2011 09/30/2014

Comments

(incl. %

achievement)

A study undertaken in 2012 estimated that 92.4% of project beneficiaries

benefited from project investments, and that more than 80% beneficiaries

claimed that project investments met their needs. According to a technical

evaluation, which was also conducted in 2012, 89% of PNPM-financed

subprojects that were implemented during 2007-2011 are functional and are ade-

quately utilized by communities.

Indicator 4 : Project beneficiaries

viii

Value

quantitative or

Qualitative)

49 million 49 million 29 million

Date achieved 06/14/2011 07/15/2011 06/30/2014

Comments

(incl. %

achievement)

In 2012 and 2013, the project reached 29.9 million and 28.0 million beneficiaries,

respectively. This was substantially lower than the target of 49 million. Part of the

shortfall may be explained by a reduction in the average block grant amount per

kecamatan, which decreased by about 25% from 2010 to 2013. It is also worth

noting that the target was roughly estimated based on the population size of each

village. If indirect beneficiaries are included, the total number of beneficiaries in

2012 and 2013 became 62.2 million and 58.2 million respectively.

Indicator 5 : Of which female beneficiaries

Value

quantitative or

Qualitative)

24 million 24 million 14.6 million

Date achieved 06/14/2011 07/15/2011 06/30/2014

Comments

(incl. %

achievement)

During 2011-2012, the project financed more than 225,000 proposals, more than

50% of which were formulated by women’s groups.

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target

Values (from

approval

documents)

Formally

Revised

Target Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 1 :

Component 1:

a) Min. 50% participation rate of women and poorest community members

in planning and decision-making meetings

Value

(quantitative

or Qualitative)

In 2008, 48% for women

and 56 % for poorest

community members

More than 50% 45% of women;

51% of poor

Date achieved 06/14/2011 07/15/2011 06/30/2014

Comments

(incl. %

achievement)

At loan closing, overall women’s participation was 45%, while participation of the

poorest was 51%.

Indicator 2 : Component 1 :

b) More than 85% of agreed work plans completed each year

Value

(quantitative

or Qualitative)

in 2006, 95% completed More than 85% 95%

Date achieved 06/14/2011 07/15/2011 06/09/2014

Comments

(incl. %

achievement)

95% of participating villages complied with PNPM Rural project cycle

according to the PTO

Indicator 3 : Component 1:

c) More than 70% of infrastructure works are evaluated as of high quality

ix

Value

(quantitative

or Qualitative)

PNPM Rural 2008

Annual

Report showed 65 %

evaluated as high quality

More than 70% 75%

Date achieved 06/14/2011 07/15/2011 06/09/2014

Comments

(incl. %

achievement)

75% from sub project infrastructure works complied with the technical standards

as mentioned in PTO

Indicator 4 :

Component 1:

d) O&M arrangements are in place and/ or functioning for more than 85%

of infrastructure works

Value

(quantitative

or Qualitative)

More than 85% More than 85% 90%

Date achieved 06/14/2011 07/15/2011 06/09/2014

Comments

(incl. %

achievement)

According to PSF’s technical evaluation of 2012, 90% of infrastructure projects

built by the program are still functional 1- 5 years after construction.

Indicator 5 :

Component 1:

e) By 2011 4,500 sub-districts with completed subprojects ( #/type of

infrastructure works, economic, and education and health subprojects/ activities)

Value

(quantitative

or Qualitative)

4,100 sub-districts in

2010 4,500 4,616

Date achieved 06/14/2011 07/15/2011 06/30/2014

Comments

(incl. %

achievement)

In the participating sub-districts, the project successfully built almost 35,000 km

of roads, more than 4,000 small bridges, around 3,000 irrigation systems and

nearly 35,000 clean water supplies among others.

Indicator 6 :

Component 1:

f) Through the MDG/Community Conditional Cash Transfers pilot,

improvements in priority health and education indicators.

Value

(quantitative

or Qualitative)

Taken from Generasi

(2007) baseline survey More than 0.03 0.0407

Date achieved 06/14/2011 07/15/2011 06/09/2014

Comments

(incl. %

achievement)

The impact evaluation showed that standard deviation of achievements in 12 key

health and education indicators of the CCT pilot (PNPM Generasi) in treatment

areas are 0.0407 higher than in control areas.

Indicator 7 :

Component 2:

a) More than 90% of local government councils provide funds and/or

oversee PNPM by 2011

Value

(quantitative

or Qualitative)


Date achieved 06/14/2011 07/15/2011 12/31/2012

Comments

(incl. %

achievement)

99% of local government council provided cost sharing at the maximum 20%

from the block grant.

x


b) More than 90% of planned facilitator positions are filled by 2012

Value

(quantitative

or Qualitative)


Date achieved 06/14/2011 07/15/2011 12/31/2012

Comments

(incl. %

achievement)

By the end of 2012, 87% of planned positions were filled, mainly because of

poor performance of the relatively isolated provinces of Papua and West Papua.

PMD improved facilitator conditions in the course of 2013 and 2014, which

decreased the vacancy rate to 9% in 2013.

Indicator 9 :

Component 2:

c) More than 90% of facilitators receive the agreed upon number of pre and

in service training days by 2011

Value

(quantitative

or Qualitative)

N/A More than 90% 100%

Date achieved 06/14/2011 07/15/2011 12/31/2011

Comments

(incl. %

achievement)

To improve the quality of facilitator PMD has increased the training period for

new facilitators from 11 days to the previous duration of 21 days


a) More than 90% of planned consultant positions are filled by 2011

Value

(quantitative

or Qualitative)


Date achieved 06/14/2011 07/15/2011 12/31/2011

Comments

(incl. %

achievement)

As of end of 2011, 96% of planned consultants position are filled.

Indicator 11 :

Component 3:

b) Audit sample size increases to min. of 20% of all PNPM-Rural sub-

districts and audit results are made public

Value

(quantitative

or Qualitative)

13.5% in 2010 20% 18.6%

Date achieved 06/14/2011 07/15/2011 06/09/2014

Comments

(incl. %

achievement)

Both in 2012 and 2013, audit coverage was slightly below the target of 20%

(17.6% and 18.6%, respectively). This, however, was mainly because the

calculation of audit coverage only considered reports submitted on or before 30

June. If reports received after this date were to be included, coverage is

substantially higher than 20%.

Indicator 12 :

Component 3:

c) More than 70% of sampled villages receive socialization material

packages for PNPM in 2011 and onward

Value

(quantitative

or Qualitative)

More than 70% 70% 100%

Date achieved 06/14/2011 07/15/2011 12/31/2011

xi

Comments

(incl. %

achievement)

All PNPM Rural locations have received socialization material packages

delivered by sub-district facilitators.


a) Number of project management units established by March every year

Value

(quantitative

or Qualitative)

More than 350 in 2009 More than 400 425

Date achieved 06/14/2011 07/15/2011 06/09/2014

Comments

(incl. %

achievement)

Each participating districts are required to establish project management unit as

mandated by PTO. There are about 425 districts participated in this project.

Indicator 14 :

Component 4:

b) All oversight consultants and facilitators paid on time, in full every

month

Value

(quantitative

or Qualitative)


Date achieved 06/14/2011 07/15/2011 06/09/2014

Comments

(incl. %

achievement)

N/A


c) Kecamatan grants disbursed and accounted for

Value

(quantitative

or Qualitative)

More than 75% in 2009 More than 80% 98%

Date achieved 06/14/2011 07/15/2011 06/09/2014

Comments

(incl. %

achievement) 98.8% of the loan proceeds were disbursed by the end of December 2012

G. Ratings of Project Performance in ISRs

No. Date ISR

Archived DO IP

Actual

Disbursements

(USD millions)

1 03/24/2012 Moderately Satisfactory Moderately Satisfactory 346.10





H. Restructuring (if any)

Not Applicable

xii

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal

From the late 1960s until the mid-1990s, Indonesia experienced a sustained and uninterrupted

reduction in poverty. The share of the population below the national poverty line dropped from

over 40% in 1976 to less than 12% in 1996. However, the Asian financial crisis, which started

in July 1997, wiped out much of these gains. By 1998, the poverty rate had doubled to over

24%, and the number of Indonesians living below the poverty line increased to almost 50

million. The financial crisis did not only cause economic hardship but also led to greater

demand for public participation in government affairs, both by individual citizens and by sub-

national governments. In 1999, the National Parliament passed two new decentralization laws,

partially in response to the effect of the crisis on Indonesia's political landscape. These laws

shifted many responsibilities for government affairs from the centre to the provincial and

district (kota and kabupaten) governments, and also provided these sub-national governments

with the financing needed to implement their new responsibilities.

Against this background, the Government of Indonesia (GoI) created a national development

program (Program Pembangunan Nasional or PROPENAS) for 2000-2004 aimed at increased

public participation, improved governance, and economic recovery. The economic recovery

program prioritized poverty alleviation and the development of a community-based economic

system. PROPENAS 2000-2004 emphasized the need to establish partnerships between

government, communities and the private sector, and to increase efforts to overcome poverty

and social instabilities. To help achieve these objectives, the Bank assisted GoI with the

implementation of a series of poverty reduction projects, initially consisting of the Kecamatan

Development Project (KDP) series in rural areas, and the Urban Poverty Project (UPP) series

in urban areas. Implementation of the first KDP and UPP commenced in 1998.

In 2006, GoI launched the National Program for Community Empowerment, which is better

known by its Indonesian acronym PNPM (Program Nasional Pemberdayaan Masyarakat).

PNPM was established as the operational umbrella for all community-based development

(CDD) programs in the country, including the KDP and UPP series. An inter-ministerial

working group (Kelompok Kerja Pengendali PNPM Mandiri or “Tim Pengendali”) provides

oversight of PNPM at the national level. The group is chaired by the Coordinating Minister for

People’s Welfare and its members include representatives from the Ministry of Home Affairs,

the National Development and Planning Board (BAPPENAS), the Ministry of Finance, and

relevant line ministries. Coinciding with the creation of PNPM, GoI also established the PNPM

Support Facility (PSF) to harmonize and coordinate development partner efforts, including the

planning and targeting of financial assistance, as well as monitoring and evaluation of the

program’s operations and impact. The PSF is managed by a Joint Management Committee

(JMC) with representatives from GoI agencies and development partners contributing at least

US$1 million to PNPM projects. The Bank has been co-chair of the JMC since its inception.

In 2011, the Bank had financed––or was financing––PNPM in rural areas through a series of

three (sector investment) projects, which formed the basis of a sector wide programmatic

approach: PNPM-Rural I, II, and III (the Bank originally planned to co-finance PNPM Rural

for the period 2007-2012, but the financing requirement for a five-year program would exceed

Indonesia’s Single Borrower Limit). The focus of this ICR is PNPM-Rural IV (“the project”),

which was mainly a continuation of PNPM-Rural III, to help consolidate PNPM as a national

program. The PNPM IV marked a key turning point in the program, not just in terms of its

2

sheer size and scale––as the world’s largest community driven development program––but

demonstrating GoI’s full commitment to poverty reduction and community empowerment.

The project would cover 4,978 sub-districts (kecamatan) in 32 provinces, 4,791 of which were

already covered by its predecessor projects (Table 1). PNPM-Rural IV did not only address

GoI’s development objectives, but was also fully consistent with the Bank’s Country

Partnership Strategy (CPS) for 2009-2012, which emphasizes engagements with government

counterparts and other stakeholders who are committed to addressing critical governance and

institutional challenges. At the time of appraisal (June 2011), the total cost of PNPM-Rural IV

was estimated at approximately US$1,284 million. The project would be financed from the

proceeds of an IBRD Loan (about US$531m), grants from other development partners (about

US$26m), and resources from the Borrower (about US$753m, including community

contributions of approximately US$35m). The expected effectiveness date of the IBRD Loan

was September 1, 2011, and the expected loan closing date was June 30, 2014. Grants from

other development partners would be channeled through a multi-donor trust fund in PSF.

Table 1: Key Features of PNPM-Rural Projects

(at time of appraisal of PNPM-Rural IV)

Key Feature PNPM-

Rural

PNPM-

Rural II

PNPM-

Rural III

PNPM-

Rural IV

# Provinces covered* 30 30 32 32

# Kecamatan covered 2,864 4,371 4,791 4,978

Bank financing (US$ million)** 231 300 785 531

Closing month Jun 2011 Dec 2011 Dec 2012 Jun 2014

Source: World Bank (2014)

* Indonesia has 33 provinces, of which one (the capital district of Jakarta) has no rural

areas, and is therefore not eligible for block grants from PNPM-Rural.

* *As estimated at appraisal.

1.2 Original Project Development Objective and Key Performance Indicators

Project Development Objective

The PAD stated that the project development objective (PDO) is “for villagers in PNPM rural

locations to benefit from improved socio-economic and local governance conditions”.

Key Indicators1

Outcome indicators

1. Improved household expenditure rates and improved access to economic and social

services in a minimum of 4,800 sub-districts in 2011 (impacts taken from representative

sample) 2

1 These indicators were taken from the PAD. The Loan Agreement itself did not list performance indicators. 2 The PAD mentions a minimum of 4,978 sub-districts in the main text, and a minimum of 4,800 in Annex

1. Aide-memoires and ISRs measured progress of this outcome indicator against a target of 4,800.

3

2. Economic internal rates of return (EIRRs) > 30 % for major types of rural infrastructure.

3. > 80% satisfaction levels from beneficiaries regarding improved services and local level

governance (impacts taken from representative sample)

Intermediate results indicators

The PAD identified 15 results indicators, aimed at measuring intermediate results in the four

components described in Section 1.5 (indicators are in brackets):

1. Villagers participate in a process to plan, select and manage basic social and economic

infrastructure provided through block grants (participation rate of women and poorest

community members in planning and decision-making meetings, percentage of agreed

work plans completed each year, percentage of infrastructure work evaluated as of high

quality, percentage of infrastructure works for which O&M arrangements are in place

and/or functioning, number of sub-districts with completed infrastructure subprojects,

improvements in priority health and education indicators).

2. Consultants at the national, provincial and district levels are providing assistance to

communities and local governments to implement PNPM (percentage of local government

councils that provide funds and/or oversee PNPM, percentage of planned facilitator

positions filled, percentage of facilitators receiving agreed number of pre-service and in-

service training days).

3. Project stakeholders use results of M&E activities and studies to improve project

performance (percentage of planned consultant positions filled, percentage of PNPM-Rural

sub-districts in audit sample size and audit results are made public, percentage of sampled

villages that receive socialization material packages for PNPM from 2011 onwards).

4. Project management teams established in a timely manner and functioning (number of units

established by March every year, oversight consultants and facilitators paid on time and in

full every month, kecamatan grants disbursed and accounted for).

1.3 Revised PDO and Key Indicators

The PDO and key indicators remained unchanged during project implementation.

1.4 Main Beneficiaries

The intended primary beneficiaries of the project consisted of some 49 million persons living

in 4,978 target kecamatan located in 32 of Indonesia’s 33 provinces (the Special Capital

District of Jakarta does not contain rural sub-districts and therefore did not form part of the

project). The population of these kecamatan would benefit both directly and indirectly from

improved community infrastructure and services, grant assistance for poorest and most

vulnerable groups (such as scholarships and training courses), and improved access to

microcredit––all planned and implemented through a community-driven development (CDD)

process. Secondary beneficiaries would be: (i) Directorate of General for Community and

Village Empowerment in the Ministry of Home Affairs (the executing agency), which would

benefit from increased capacity; (ii) village level facilitators and partner agencies providing

implementation support; (iii) kabupaten governments in which the 4,978 target kecamatan

were located, who would benefit from training, field studies, and institutional strengthening

aimed at improving their planning capabilities; and (iv) village representative councils (Badan

Perwakilan Desa or BPD), which would benefit from technical assistance and advisory

4

services in the process of establishing and strengthening their role in channeling CDD funds to

appropriate community activities and investments.

1.5 Original Components

Component 1: Kecamatan Grants. Project Cost: US$1,097.6 million (36% Bank financing)

This component would provide block grants to participating kecamatan to finance activities

identified in village plans, including: (i) specific high-priority infrastructure investments; (ii)

microcredit loans for women’s groups; (iii) social safety programs for the benefit of the poorest

and most vulnerable groups or individuals; and (iv) investments aimed at addressing disasters,

emergencies or catastrophic events. Kecamatan grants would not finance activities on the

project’s negative list, such as weapons, religious buildings or government offices. Each

participating kecamatan would receive a one-time block grant of IDR 750 million to IDR 3

billion to finance investments in one or more of the above categories, in accordance with their

community development plans (with a maximum allocation of 25% to revolving funds for

women’s savings and loan groups).3 The block grant amount would depend on the population,

poverty incidence and location (Java/Bali or elsewhere) of the kecamatan. In addition to the

core sub-district planning and block grant scheme, PNPM-Rural IV would also provide block

grants for four special programs:

1. PNPM-Generasi (US$375 million). The incentivized block grant scheme, which

commenced in 2007 is co-financed by GoI and donors to support the achievement of

Millennium Development Goals (MDGs) related to health and education. From 2011

to 2012, US$94.3 million was allocated to the pilot project. Generasi was subsequently

delinked in 2012 once the proof-of-concept stage was successfully completed. It

attracted substantial donor funding as well as GoI support to warrant its own identity

as a standalone project. An additional grant financing operation in the amount of

US$151.7 million was approved in 2014.4

2. PNPM-Green (US$34 million). This pilot program, which commenced in 2008 funded

block grants and technical assistance to promote community investments in natural

resource management and renewable energy. An additional grant financing operation

in the amount of US$18 million was approved in 2012 to finance additional block

grants and technical support for eight target provinces. The pilot project closed in 2013.

Of the total amount allocated, US$30.68 million was disbursed.

3. PNPM-BPKG (US$100 million). This is a regional block grant scheme initiated in

2009 and financed by the Government of Nanggroe Aceh Darussalam (“Aceh”)

through the Special Autonomy fund granted to Aceh as part of the 2005 peace

agreement. While the block grants are not co-financed by the Bank, the program

improved the community welfare by providing financial assistance through a

provincial allocation for each (village) but using the project’s facilitators and operating

3 Except sub-districts that were classified as “less poor” or “near poor” that had received block grants for

more than three cycles under PNPM-Rural. These would receive block grants of IDR 450 million and IDR

650 million, respectively. 4 The PAD mentions a minimum of 4,978 sub-districts in the main text, and a minimum of 4,800 in Annex

1. Aide-memoires and ISRs measured progress of this outcome indicator against a target of 4,800.

5

systems. In the period of 2007– 2012, BKPG disbursed over IDR1.5 trillion (around

$120 million USD) for a wide variety of activities, with a focus on village infrastructure

improvements, savings and loans activities for women’s groups, education and health

programs, and the strengthening of village government. In 2013, IDR70 million were

allocated for each of 6,464 villages in the province. Unlike the national version of

PNPM, BKPG provides the same financial allocation for each village in Aceh. The

program is still ongoing.

4. PNPM-RESPEK. This is also a similar regional block grant scheme, initiated in 2008.

It covers the provinces of Papua and West Papua, and is funded by the governments of

these provinces. A provincial top up to the block grant in the amount of IDR100 million

was provided to each village. PNPM-RESPEK finances activities in the fields of: (i)

nutrition and food security, (ii) education, (iii) primary health care, (iv) village

infrastructure, and (v) economic livelihoods. The program is still ongoing.

In May 2009, GoI had signed a financing agreement with the International Fund for Agricul-

tural Development (IFAD) in an amount of about SDR42.3 million, of which SDR42.0 million

was provided as a loan and the remainder as a grant.5 The IFAD grant was used to finance

block grants, facilitation support and implementation support and technical assistance in

parallel with funds provided by the project and was therefore not formally linked to the project.

Component 2: Facilitation Support. Project Cost: US$110.8 million (89% Bank financing)

This component would strengthen the capacity of district and sub-district government

institutions and communities in development planning and investment through sub-district

(kecamatan) and district (kabupaten) social and technical facilitators.

Component 3: Implementation Support and Technical Assistance. Project Cost: US$36.8

million (84% Bank financing)

The project would be managed by a Project Management Unit (PMU) in the Directorate-

General of Village Community Empowerment (DJ-PMD or simply “PMD”) in the Ministry of

Home Affairs. The PMU would hire consultants to assist with project implementation;

facilitators would be hired by the participating provincial governments. Technical assistance

would be provided through National Management Consultants (NMC) at the central level, and

Regional Management Consultant (RMC) teams at the province level, with RMC offices in the

participating district governments, and facilitators and community cadres at the kecamatan

level. In addition, this component would finance strategic development of PNPM, the project’s

monitoring and evaluation program, and an enhanced audit program that expands sampling,

and provides capacity development support for district government auditors. This component

would also finance most project training other than community-level training.

Component 4: Project Management Support. Project Cost: US$38.5m (2% Bank financing)

This component would provide technical advisory services and other material support to

strengthen PMD and support the management of PNPM’s special programs, including pilot

projects, government add-ons, and crisis-response activities (such as natural disasters and

5 At the time the agreement was signed, the combined amount was equivalent to about USD 65.2 million.

6

financial crises). Most of these operational costs would be covered by the Government from

central and local government budgets.

1.6 Revised Components

The components remained unchanged during project implementation.

1.7 Other significant changes

Delinking of PNPM-Generasi and PNPM-Green. At appraisal, it was envisaged that part of

the proceeds of the US$531 million loan would be allocated to co-finance PNPM-Generasi.

For managerial purposes, the Bank decided to delink this pilot project from PNPM-Rural III in

December 2012, and the project was not linked to PNPM-Rural IV. PNPM-Green was not

linked to either project, and continued to be implemented on a stand-alone basis.

Acceleration of disbursement. The PAD envisaged that 87.8% of the loan would have been

disbursed by July 2013. In actual fact, 98.8% of the loan proceeds were disbursed by the end

of December 2012, reportedly because GoI wished to free up fiscal space for other programs.

The unexpected acceleration of disbursement accelerated the preparation of a successor loan

(National Program for Community Empowerment in Rural Areas 2012-2015, informally

known as “PNPM-Rural V”), which was signed in November 2012.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry

Soundness of background analysis and lessons learned from previous projects. The design

of PNPM-Rural IV was based on PNPM-Rural III, which was being implemented at the time

of appraisal, but with several modifications to reflect lessons learned from the implementation

of its predecessor projects. The most important of these lessons were the following: (i) need to

grow institutional and managerial capacities of the implementing agency along with the

program; (ii) need to develop a robust and integrated management information system; (iii)

need to strengthen governance and community level accountability systems (especially an

improved complaints handling system); (iv) need to strengthen the quality of facilitation since

facilitators are the backbone of PNPM; and (v) need for additional implementation assistance

from the Bank through a portfolio approach (PAD, pp 6-7).

Assessment of project design. As indicated above, project design followed the successful

models of other CDD projects, which had been verified through previous studies and also

incorporated lessons learned from previous PNPM-Rural projects. However, the project design

also anticipated implementation challenges due to the immense size of the project by among

others: (i) acknowledging the need for an improved MIS and leveraging additional donor

funding to support its redesign; (ii) introducing a “portfolio approach” to supervision (i.e.,

combining supervision missions of two or more projects) as part of the project design to allow

the Bank to help GoI harmonize and supervise projects more cost-effectively while the project

grew in size; and (iii) supporting GoI with some of the operational responsibility for monitoring

and evaluation of key performance indicators through PSF and other diagnostic studies, which

would be used to help inform operations. The RF PDO level indicators were designed to bring

out the achievement of the PDOs and the team, given the loss of the control groups, decided to

compensate by augmenting the 2011 impact evaluation results with 2 additional technical

surveys: technical evaluation of infrastructure and beneficiary satisfaction survey.

7

Government commitment, stakeholder involvement, and participatory processes. The

Government of Indonesia played a central role in helping to prepare the project in accordance

with a roadmap for the development of a nationwide PNPM until 2015. This roadmap was

prepared with the involvement of stakeholders at the central and sub-national level, including

NGOs and community-based organizations.

Risks and risk mitigation measures. The overall project risk was rated “Medium” (low

likelihood-high impact), and this rating was deemed appropriate. The key identified risks

include: (i) institutional and managerial capacity of PMD, (ii) formal and informal fiduciary

and governance controls, and (iii) availability of qualified facilitators. Although the project risk

analysis in the PAD did not mention the systemic delays in budget approvals by GoI as a risk

factor, the Bank and GoI sought to mitigate this problem by setting up the block grant

reimbursement mechanism as a way to limit the extent of the delays and extending the project

cycle from 12 to 18 months. Furthermore, notwithstanding the delays in the release of the

budget, the vast majority of sub-projects were still completed on time.

2.2 Implementation

Factors outside the control of the government or the implementation agency

Not applicable.

Factors subject to the control of the government or the implementation agency

Supporting factors

Strong political commitment. The project was able to secure support at the highest levels

of government. Virtually all participating district governments co-financed part of the

project cost from their own resources, and a substantial minority provided funds over and

above the minimum required amount.

Effective anti-corruption strategy. Individuals and independent organizations in bene-

ficiary communities used channels of communication established by the project (such as

the complaints handling unit or individual facilitators) to report incidences of suspected

corruption. Follow-up actions, including local government coordination and withholding

of funds distribution, proved effective in combating corruption. Corruption largely

consisted of isolated cases.

Challenging factors

Centralized program management. In spite of strong political support, the implementing

agency’s capacity to take corrective and timely actions was hampered by its centralized

management structure, which sometimes did not allow for the timely escalation and

redress of the problems of a project with such a wide national coverage. Limitations in

coordinating with other PMD Directorates at times also resulted in inconsistencies in the

application of HR and other operational policies across various PNPM projects managed

by PMD.

Delays in the recruitment of facilitators. Given the scale and size of the project (i.e. the

need to retain more than 12,000 facilitators a year), some delays in procurement of

facilitators occurred, especially in remote areas such as Papua and West Papua where

provincial governments were often unable to recruit facilitators with the required

8

experience and academic background at the salary level offered. To address the shortfall,

GoI increased facilitators’ salary in mid-2013 and made exceptions to the recruitment of

technical facilitators in the two provinces under the Barefoot Engineering program, which

was designed to provide on-the-job training.6

Mid-term review. At the end of 2012 a mid-term review was conducted, three months ahead

of the original schedule (March 2013). The review resulted in two sets of recommendations.

Improve the performance of PNPM-Rural in remote areas. It was recommended to

modify the implementation approach of PNPM Rural in remote areas (especially in the

provinces of Papua and West Papua) by, inter alia, adjustment of the project cycle from

12 to 18 months to ensure that sub-projects could complete; strengthen fiduciary

safeguards applied to the RLFs, which in effect, help to control and stem disbursements;

and improve the remuneration, benefits and travel allowance for facilitators.

Improve program management by PMD. The Bank urged PMD to strengthen program

oversight and management by accelerating the redesign of the management information

system (MIS), improving the operations manual (Petunjuk Teknis Operasional or PTO),

establishing a joint secretary for PNPN-Rural and related programs, and by temporarily

excluding kecamatan with problems related to fraud and corruption or with one or more

facilitator positions that had been vacant for at least four months.

Because 98.8% of the loan funds had been disbursed at the time the mid-term review was

conducted, the recommendations were aimed at improving the performance of PNPM-Rural in

general. At the loan closing date, PMD had implemented most recommendations on program

management, except those on improving performance in remote areas. These were delayed

because of operational difficulties and would be addressed during the implementation of the

successor project PNPM Rural 2012-2015

Effectiveness of risk mitigation. Mitigation of project risks was generally good. Specific risks

related to pro-poor targeting and fiduciary controls were mitigated effectively during

implementation; and the risks to project management and the recruitment of faciliators was

partially effective. For instance, the revision of the PTO in March was a significant milestone

in the project as the PTO had not been revised for 5 years; and according to the reports on

facilitation, the national vacancy rate for facilitators was relatively low (3% at District level

and 5% at the Sub-District level by early 2014. However, technical facilitator vacancy rates

tend to be higher in remote areas (8%). Risks related to the sustainability and performance of

RLFs may have been mitigated less effectively, although the shortcomings did not

fundamentally undermine achievement of the project outcomes.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

6 The Barefoot Engineers project trains local high school graduates to become PNPM technical facilitators

in the remote and undeveloped Indonesian provinces of Papua and Papua Barat, and provides them with

mentoring and refresher training over the course of their work. 624 technical facilitators have graduated from

the program and have been recruited by PNPM since 2003; the latest class of 290 (of which 29% were

women) graduated in March 2013, and at the time of writing, 246 remain employed with the program.

9

The project measured progress against key performance indicators (KPIs) using a management

information system, augmented by a series of qualitative and quantitative surveys and the

results of regular supervision missions.

(a) M&E design. At the time of appraisal, a management information system was in place

to monitor and evaluate PNPM-Rural III. The same system would be used to monitor and

evaluate the implementation of PNPM-Rural IV. In addition, the PAD listed a series of

surveys and studies that would be undertaken during project implementation to

complement the data to be collected by the MIS. And as mentioned above in Section 2.1,

the loss of the control groups had prompted the team to add 2 technical surveys to

augment the 2011 impact evaluation results in order to assess PDO indicators

performance.

(b) M&E implementation. All surveys and studies defined in Annex 1 of the PAD were

completed before the project closing date.7 Both the Government and the Bank conducted

monitoring and supervision as planned and findings from surveys/studies, monitoring and

supervision were all used to improve project performance. As anticipated, the project

continued to encounter problems with the MIS. MIS data was published on the project

website (http://www.pnpm-perdesaan.co.id) and updated on a monthly basis until

December 2011, i.e. shortly after loan effectiveness in September 2011.Throughout the

project implementation period, reporting was delayed due to difficulties in transmitting

vast volumes of project data from village level up to national level. At the suggestion of

the Bank, PMD had requested Australia’s Department of Foreign Affairs and Trade

(DFAT) to recruit a firm to redesign the MIS. However, this activity was delayed (one

year behind the original schedule, partly because the design specifications were not

finalized). At the loan closing date, the new MIS was not yet operational.

(c) M&E utilization. The utilization of the MIS was compromised by the fact that the system

was unable to generate timely information until most of the PNPM-Rural IV loan funds

were already disbursed. To compensate for this shortcoming, both GoI and the Bank

relied more heavily than expected on the findings of supervision missions and the planned

surveys and studies to monitor, reconcile, verify and evaluate project implementation.

Data collected from these sources helped the project to (i) identify and remedy quality

problems and to incorporate better controls as the project progressed, (ii) improve control

of fund flows and accountability for sub-grants, and (iii) escalate issues to PMD

management, as appropriate.

2.4 Safeguard and Fiduciary Compliance

PNPM Rural IV was a Category B project. The Implementation Guidelines for Social and

Environmental Safeguards (IGSES) was satisfactorily prepared and disclosed in 2013, and was

re-disclosed in March 2014 as part of the revised PTO. Although some project activities

triggered safeguards related to environmental assessment (OD/BP 4.01) and indigenous

peoples (OD/BP 4.10), none of these activities caused significant adverse impacts. The project

did not finance land acquisition, but did receive voluntary land donations. Given the small size

the subprojects financed by the project (the cost of which often did not exceed US$20,000

7 The beneficiary survey was conducted in 2012 but the report was completed in October 2014. See Annex

5 for summary of the survey results.

10

equivalent), it was a priori expected that any adverse social and environmental impacts of such

subprojects would be small or negligible. Indeed, the project generated substantial social and

environmental benefits, especially through investments in drainage, water supply and sani-

tation. Some aspects of the design and monitoring of mitigation measures of potential social

and environmental impacts remained poor, especially with regards to voluntary land donations,

and required significant improvement in the final stages of project implementation. In addition,

in some provinces the participation of indigenous peoples was initially substandard and a

review mission was carried out in mid-2014 aimed at improving compliance with OP/BP 4.10

to mitigate the potential adverse impacts of social exclusion.

As to fiduciary compliance, the project received an unqualified opinion from an external

auditor in 2012 and 2013, and according to the ISR of June 2013 (when most loan funds were

disbursed), the project’s financial management was generally acceptable.8 PMD ensured that

local governments addressed the findings of the external auditor, although the recording of this

process in the MIS was often incomplete or late. Both in 2012 and 2013, audit coverage was

slightly below the target of 20% (17.6% and 18.6%, respectively). This, however, was mainly

because the calculation of audit coverage only considered reports submitted on or before 30

June. If reports received after this date was to be included, coverage is substantially higher than

20%.

The project did encounter some problems with financial management of the revolving funds

component throughout implementation, especially those related to the accounting of the RLFs

and implementing pro-active management strategies to deal with loan loss reserves. In addition,

some provinces experienced difficulties in finding qualified financial facilitators. Despite these,

the repayment rate of RLF groups in general are still high (>90%). Nonetheless, to address

these problems, PMD raised facilitator salaries in 2014 to attract more qualified candidates,

took action to improve the RLF reporting system, and mandated a moratorium on block grants

for RLFs with high levels of non-performing loans and idle funds.

Procurement at the closing of the project was rated as moderately satisfactory as significant

improvements were noted in the simplification of community procurement procedures which

were also aimed to promote further transparency; and the transition to applying the improved

procurement procedures under the revised PTO is underway. Furthermore, no significant issues

were noted during ex-post reviews that were conducted by the Bank twice a year.

8 The audit report published in 2014 (and covering the calendar year of 2013) is not relevant in this

context because 98.8% of the IBRD loan funds for the project were disbursed by 31 December 2012.

11

2.5 Post-completion Operation/Next Phase

(a) Transition arrangements. Presidential elections were held in July 2014, shortly after the

loan closing date. At the time of writing, GoI and the Parliament are still discussing how

the PNPM program would need to be modified to operate in conjunction with the

Government’s new Village Law Program, which was approved by the outgoing President

in January 2014.

(b) Operations and Maintenance. Typically, the responsibility for operation and maintenance

(O&M) has been assumed by the community through the establishment of O&M teams

during the accountability meetings. These teams are responsible for collecting fees for

operating as well as for the provision of regular infrastructure maintenance. This

arrangement has worked well for infrastructure where the community can levy standard

fees such as irrigation, clean water supply systems, and early childhood education centers

but less so for public infrastructure such roads and drains where maintenance is provided

on an ad-hoc basis. However, in some villages, the village government helps to provide

additional funds for the maintenance of roads.

(c) Follow-up projects. In addition to the Additional Financing operations that were

processed for PNPM Generasi and Green, in the first half of 2012, GoI requested the

Bank to co-finance the National Program for Community Empowerment in Rural Areas

2012-2015. An agreement for US$650 million IBRD loan was signed in November 2012.

(d) Future impact evaluation. A new program evaluation is being designed and will be

implemented in the first quarter of 2015 prior to the completion of PNPM Rural 2012-

2015.

3. Assessment of Outcomes

3.1 Relevance of Objective, Design and Implementation

Rating for Relevance of Objective: Substantial

The project complemented the Government’s efforts to improve socio-economic development

and empower local communities, especially in rural areas. The President recently confirmed

his commitment to community-driven development and empowerment projects, and in

accordance with this vision, the new Village Law Program further decentralizes fiscal and

decision making responsibilities to Village Governments. The CDD approach is also a key

component of the Bank’s Country Partnership Strategy for 2013-2015, which emphasizes

engagements with government counterparts and other stakeholders who are committed to

addressing critical governance and institutional challenges. The CPS identifies six thematic

areas that are expected to form the core of the Bank’s engagement in Indonesia, three of which

are directly relevant to PNPM-Rural IV: (i) Promoting Communities, Protecting the Vulnerable

and Improving Health Outcomes, (ii) Ensuring Sustainable Development and Improving

Disaster Resilience, and (iii) Gender and Governance. Therefore, at completion the project was

still highly relevant to the objectives of both the Bank and the GoI.

12

Rating for Relevance of Design and Implementation: Substantial

The design of PNPM Rural IV remained relevant as it was built upon the previous PNPM Rural

projects with some corrective actions to strengthen the MIS (hiring of consultant by DFAT to

improve design of the MIS and to make the system online/web-based); and improve RLF

reporting and tracking (through the RLF MIS under the RLF pilot project). In addition,

implementation arrangements were enhanced through the Bank- and Recipient-executed Trust

Fund projects such as the Barefoot Engineer III project, which aimed to improve the

availability of technical facilitators in Papua and West Papua); RESPEK Trust Fund, which

aimed to strengthen the overall program management and coordination of community

empowerment activities among development partners in the remote areas of Papua and West

Papua; and the RLF pilot, which supported the capacity building, restructuring and

institutionalization of the PNPM RLF scheme; and to help strengthen linkages between the

PNPM RLFs and the commercial microfinance sector so as to help ensure growth and

sustainability in the provision of financial services to the poor.

3.2 Achievement of Project Development Objective

Rating: Substantial

Overview. Over the course of the project (2011–2013), PNPM-Rural IV has built, inter alia,

over 50,000 km of roads, more than 12,000 irrigation systems, and nearly 62,000 clean water

facilities. The project also generated temporary employment for around 1.5 to 2 million persons

annually, of which more than 80% were classified as poor, and provided revolving loan funds

for around 47,000 women’s groups annually (see Annex 2 for details on outputs). These

achievements are significant. Based on an analysis of the logical chain and potential changes

attributed to the project’s outputs, the investments in rural infrastructure have, among other

things, improved basic socio-economic conditions for communities through the provision of

access to transportation, access to basic services (such as health and clean water), and new

economic opportunities for villagers in the project areas. Access to transportation and basic

services have helped households to reduce their transportation costs while new economic

opportunities, along with temporary employment provided by the project and access to the

revolving loan fund, contributed to project outcome of increased household expenditure and

improved access to economic and social services. 9

Participation in the project remained strong–which resulted in high satisfaction rates and a

perception that the project is more transparent than other projects at village level. 10 The

efficacy of the project with respect to improvements in local governance conditions can also

be verified by the following achievements: (i) the project’s external audits, undertaken by

GOI’s auditing entity BPKP, achieved its audit target of 20 percent of the sub-districts

benefiting from the project and continued to enjoy unqualified ratings for the entire project

period.11 In addition, the practice of sensitizing communities at the sub-district and district

9 PNPM-Rural Impact Evaluation. PSF. April 2012, A Qualitative Study: Impact of PNPM Rural in East

Java, West Sumatera, and South Sulawesi. PSF. April 2012. Economic Impact Analysis of Kecamatan

Development Program Infrastructure Projects. DSF. 2005, Ex-post Evaluation of KDP Infrastructure

Projects. World Bank. 2001 10 The Local Level Institutions III: Overview Report. World Bank. 2013. 11 While at project closing, the actual audit sample is reported as 18.57 percent, the task team confirms that

the audit sample was achieved if reports received after June 30 were included.

13

levels to audit findings (also known as “audit expose”) continued to be promoted through the

project; (ii) annual audit reports, including all district audit reports were published on the

PNPM website on an annual basis thereby meeting a loan convent requirement; (iii) relevant

heads of districts and sub-districts in each project location disclosed reports on the results of

project implementation and accounted for funds used on an annual basis thereby meeting a

loan convent requirement; (iv) 100 percent of villages received socialization material packages

for the project, compared with an end-line target of 70 percent; (v) community procurement

issues, which were previously identified as somewhat problematic in the ICR for PNPM Rural

III (largely due to a lack of simplicity and clarity in procedures) were subsequently resolved

by the closing of PNPM Rural IV and formally adopted through the revision of the PTO; (vi)

additional measures to improve RLF management and transparency, including term limits for

UPK staff and the recruitment of additional RLF facilitators were also adopted in the new PTO;

and (vii) by project closing, the IT-based Complaints Handling System showed improvements

in the management of fraud and corruption cases, including a decrease in “potentially

problematic locations” and increase in the overall case recovery rate. This can be attributed to

the fact that the implementing agency took appropriate actions to apply sanctions and suspend

disbursements in select areas where corruption had been identified and where no appropriate

action was assessed to have been taken. In addition, PMD together with TNP2K and

Menkokesra took important steps to continue to promote good governance measures through

the implementation of GoI’s Better Governance Action Plan (BGAP). This helped to send a

strong message to the public and communities that corruption will not be tolerated.

Outcome indicators. As indicated above, the project’s achievements are underscored by the

following outcome indicators:

Improved household expenditure rates and improved access to economic and social

services in a minimum of 4,800 sub-districts in 2011. Since PNPM-Rural became a

national program in 2010, the task team was no longer able to conduct an impact

evaluation to compare treatment and control areas as a way to measure outcomes.

However, supplementary data was used to verify the project’s findings by comparing

them to national trends. according to the National Socio-Economic Survey (Survei Sosial

dan Ekonomi Nasional, Susenas), which is conducted annually in Indonesia, per capita

expenditure has shown an increasing trend from 2008 (when the PNPM commenced its

national scale up) to 2013 while poverty rate was declining for the same period (Table 2).

However, two rounds of impact evaluations (completed in 2011 and 2008) for both

PNPM-Rural and Kecamatan Development Program (KDP, predecessor of PNPM-Rural)

showed that consumption rate of poor households in project locations was higher than

that of the poor households in non-project locations (9.1% and 11% higher, respectively).

In terms of access to services, the impact evaluation showed that households in PNPM

Rural areas saw a 5.1% increase in outpatient care access compare to non-PNPM Rural

areas.12 In addition, the 2012 beneficiary survey, which was designed to augment the 2011

impact evaluation findings, further verified the original findings and showed that 66% of

households reported better access to transportation as benefit from PNPM Rural sub-

projects.13

12 PNPM-Rural Impact Evaluation. PSF. April 2012. 13 Overview Findings of Incidence of Benefits Survey 2012. PSF. 2014 .

14

Table 2: Poverty Rate and Per capita Expenditure in Indonesia (2008 – 2013)14

Year 2008 2009 2010 2011 2012 2013

% population below national poverty line

15.42 14.15 13.33 12.36 11.66 11.47

Per capita expenditure (IDR)

293,256

296,123

333,850

387,787

434,300

475,952

Source: World Bank staff analysis of the Susenas 2008 – 2013 data

Economic Internal Rates of Return (EIRRs) >30 % for major rural infrastructure types.

A study undertaken in 2012 found that the economic internal rate of return (EIRR) of a

sample of 48 small-scale infrastructure projects financed by predecessor projects during

2007-2011 ranged from 35% to 50%, which was higher than that the targeted EIRR of

30%.15 This rate is consistent with previous EIRR assessment of similar sub-projects built

using similar approach in Indonesia (KDP), which showed EIRR rate of between 38.62%

to 67.64%, and a recent EIRR assessment of similar infrastructure projects built under

PNPM-Rural Post Disaster Management Support (DMS) project, which concluded an

average EIRR of 60%.16 However, as explained in detail in Annex 3, the 2012 study may

have used some over confident assumptions. (For example, the assumption that benefits

generated by a sub-project remained constant over time until end of its service life).

Despite this, and adjusting for such assumptions, it is assessed that the average EIRR of

most sub-projects will remain comfortably above the 12% economic opportunity cost of

capital, but not necessarily above the project target of 30%.

>80% satisfaction levels from beneficiaries regarding improved services and local level

governance. A study undertaken in 2012 estimated that 92.4% of project beneficiaries

benefited from project investments, and that more than 80% beneficiaries claimed that

project investments met their needs.17 According to a technical evaluation, which was

also conducted in 2012, 89% of PNPM-financed subprojects that were implemented

during 2007-2011 are functional and are adequately utilized by communities.18 A recent

technical evaluation of PNPM DMS also showed that of 29 sub-projects evaluated (built

in 2012-2013), 27 (93%) has functional community O&M team.19 These findings serve

as proxy for satisfaction levels: communities do not merely state that they are satisfied,

but benefited from and actively use and maintain the subprojects for the intended purpose.

In 2012 and 2013, the project reached 29.9 million and 28.0 million beneficiaries, respectively.

14 Susenas sampling frame in 2008-2010 used different data base and sampling method compare to 2011-

2013 (revised based on data from 2010 Census). All PCEs are deflated to National CPI 2007. All data are

weighted to the population weight. 15 Laporan Akhir Studi Skala Kecil Analyisis Manfaat Ekonomi Proyek Infrastruktur PNPM Mandiri

Perdesaan. PSF. July 2012. (This report mentions substantially higher average EIRRs, and these EIRRs

were also mentioned in the ISRs. However, EIRRs for most subprojects in the sample for Central Java

were estimated at over 200% and considered outliers by this ICR; see Annex 3 for details.) 16 Economic Impact Analysis of Kecamatan Development Program Infrastructure Projects. DSF. 2005; EIRR

and Technical Evaluation Report for PNPM Disaster Management Support. PSF (draft, January 2015). 17 Overview Findings of Incidence of Benefits Survey 2012. PSF 2014 . 18 Technical Evaluation of Infrastructure – PNPM-Rural & Other Funding Sources. PSF. July 2012. 19 EIRR and Technical Evaluation Report for PNPM Disaster Management Support. PSF (draft, January

2015).

15

This was substantially lower than the target of 49 million. Part of the shortfall may be explained

by a reduction in the average block grant amount per kecamatan, which decreased by about

25% from 2010 to 2013. It is also worth noting that the target was roughly estimated based on

the population size of each village. An assumption was made that every village under each

kecamatan would be covered. At that time, a village poverty mapping was not done before the

project was started and the project did not have the methodological tools to make more accurate

statistical approximations either ex-ante or ex-post. The program over time (at least from 2008-

2014) has only covered 60,000 out of the 73,000 villages in Indonesia20. In addition to that, if

indirect beneficiaries are included, the number of beneficiaries for 2012 and 2013 became 62.2

million and 58.2 million respectively (half of which are women and the poor).

Intermediate results indicators. During 2011-2012, the project financed more than 225,000

proposals, more than 50% of which were formulated by women’s groups. (see Annex 2 for

details).

The targets of most of the intermediate results indicators were also achieved, the exceptions

being:

Min. 50% participation rate of women and poorest community members in planning and

decision-making meetings. At loan closing, overall women’s participation was 45%,

while participation of the poorest was 51%.

> 90% of planned facilitator positions are filled by 2012. By the end of 2012, 87% of

planned positions were filled, mainly because of poor performance of the relatively

isolated provinces of Papua and West Papua. PMD improved facilitator conditions in the

course of 2013 and 2014, which decreased the vacancy rate to 9% in 2013.

3.3 Efficiency

Rating: Substantial

Efficiency of infrastructure subprojects. Efficiency was gauged by the economic rate of

return, and unit rate norms of subprojects financed by the predecessor projects of PNPM-Rural

IV. As mentioned in Section 3.2, a study undertaken in 2012 found that EIRRs of subprojects

financed by predecessor projects ranged from 35% to 50%, higher than the targeted EIRR of

30% (these figures do not include EIRRs for subprojects in Central Java, which were unusually

high and were therefore considered outliers; see Annex 3 for details). It is reasonable to assume

that the EIRRs of subprojects financed by PNPM-Rural IV could be lower than EIRRs of

subprojects presented in the study because the study: (i) does not include subprojects that were

no longer functional (estimated at 11% of the total according to the PSF study mentioned in

footnote 9), (ii) assumes a low shadow wage rate factor for unskilled labor, and (iii) assumes

that benefits generated by a subproject remains constant over time until the end of its service

life, even though benefits will in practice gradually decline because of maintenance. However,

even if more conservative assumptions are used, the EIRR of most subprojects will, in all

20 The highest beneficiaries figure was reached in 2010 with about 47 millions – that was when the project

reached national coverage with the highest block grant amount per kecamatan.

16

likelihood, remain above the economic opportunity cost of capital, (which the Bank estimates

at 12% for developing member countries), but may fall below the target of 30% for some of

the project-financed subprojects (see Annex 3 for details). It is also worth noting that EIRRs of

subprojects financed by kecamatan grants are expected to decrease over time, as most

kecamatan in the country have by now received several rounds of block grants, and may well

have used the first rounds to finance the most economically viable subprojects.

Unit rate norms are highly favorable for community-executed infrastructure projects. An

independent evaluation of KDP-financed subprojects found that the cost of small-scale

infrastructure built by villagers was on average 56% lower than equivalent works built through

government contracts. The cost savings were mainly attributed to the high level of voluntary

labor contributed by residents in community-driven development projects. These findings were

confirmed by an analysis of 48 subprojects financed by PNPM-Rural. According to this

analysis, the total cost of the subproject was 36% lower than if they would have been under-

taken by local government contractors. In 2012, PSF commissioned a technical evaluation of

1,128 subprojects financed by PNPM-Rural. 21 According to this study, PNPM-sponsored

infrastructure projects are typically 15 to 25% less costly to construct than similar projects

undertaken by line ministries. Cost savings are substantially higher (sometimes exceeding 50%

of line ministry costs) for works that involve large amounts of unskilled labor.

Efficiency of RLF subprojects. About 20% of kecamatan block grants were eligible for

reimbursement under block grants for RLFs and training. In 2012 and 2013, percentages

reimbursed amounted to 11% and 6% respectively. The efficiency of investments in these

components was not captured in the project’s results framework although PMD collects regular

data on RLFs performance, especially on non-performing loans, groups’ repayment rates and

maturity (defined as groups that are ready to be channeled to other/regular sources of financing

such as bank and cooperatives) as can be seen in Table A.2.3 in Annex 2. In addition, the Bank’s

RLF pilot project was designed to address some of the specific shortcomings of the RLF,

provide performance data, promote good financial practices, and test a set of financial practices,

procedures, as well as tools, which are expected to help improve RLF performance. An

evaluation of the pilot project will be conducted in the first quarter of 2015.

3.4 Justification of Overall Outcome Rating

Rating: Moderately Satisfactory

Relevance. The PDOs and design remained relevant throughout the implementation period and

was rated as “substantial.” PNPM-Rural IV was not only central in efforts to alleviate poverty

in villages covered by the project, but was also designed to consolidate and improve the

Government’s nationwide CDD program. In addition, the project explicitly addressed the

Government’s roadmap for further development of PNPM, especially by the successful

incorporation of related poverty alleviation projects.

Achievement of PDO. As described in Section 3.2, the project has achieved its development

objective as measured by its three outcome indicators. While limitations were noted in the

project’s analysis of improvements in socio-economic and governance conditions, as the

analysis was mostly derived from surveys undertaken a year prior to the project’s

implementation, the underlying assumptions made by the task team regarding limited changes

21 Ibid.

17

to the project context and circumstances from one year to the next are considered to be

reasonable given the loss of the control group. Furthermore, to further verify its PDO findings,

the Bank team expanded its methodology through the implementation of two additional

studies; undertook an analysis of the project’s intermediary results indicators; and extrapolated

data from other external sources, which were used to depict national level trends at the

household level. Intermediate results indicators and complementary in-depth perception

surveys which are used to substantiate the findings also suggest that the major share of the

project’s beneficiaries are satisfied and use project-financed infrastructure for its intended

purposes.

Efficiency. As described above, the cost of CDD projects tends to be significantly lower than

similar programs financed by government contractors. It should be noted, however, that these

cost savings could be influenced by: (i) limited capabilities for O&M for some types of

infrastructure, especially roads), and (ii) heavy reliance on facilitators, which results in

relatively high project implementation costs (project management costs of CDD projects are

usually in the order of 15-20% of total costs, against less than 10% for most other types of

projects). As mentioned in Section 3.3, there is also reason to believe that EIRRs of subprojects

financed by PNPN-Rural IV are lower than EIRRs estimated by PSF-commissioned studies.

Despite some of the shortcomings noted in the project’s monitoring and evaluation, the project

found other ways to augment the verification means and confirm the project’s outcomes, which

were substantiated through the 2012 technical evaluation and beneficiary surveys; even in the

absence of an impact evaluation or updated economic analysis. The overall outcome of the

project is therefore rated “moderately satisfactory”.

3.5 Overarching Themes, Other Outcomes and Impacts

After more than a decade of implementation of community-driven development (CDD)

projects in Indonesia, lessons learned from various evaluations/studies have shown that CDD

projects – including this project and its predecessors – have had positive impacts on poverty

reduction. Both PNPM Rural and KDP helped increase household consumption, move

household out of poverty, and improve access to economic and social services. Evaluations

also showed that PNPM Rural and KDP are more effective in reaching poor households and

households in poor sub-districts.22 The project continues to see strong women participation in

project’s planning meetings as well as high percentage of women’s proposals being funded

each year (more than half).

By and large, PNPM Rural remains one of the most well-known development projects at the

community level – typically around 60% of respondents in PNPM-related surveys would said

that they have heard of or know the name of the project.23 A study on local level institutions in

Indonesia indicates that, in addition to being well-known, the project is also associated with

good transparency and in general has higher satisfaction rate compare to other development

projects at the local level.24 All these positive achievements have contributed to the GoI

22 PNPM Rural Impact Evaluation. PSF. 2012, Second Phase KDP Impact Evaluation. World Bank. 2008. 23 See for example, PNPM Rural Impact Evaluation (2012) and PNPM Beneficiary Survey (2012). 24 The Local Level Institutions III: Overview Report. World Bank/PSF. 2013.

18

decision to incorporate CDD principles such as participation/inclusion, community

empowerment and accountability into the Village Law as mentioned in Section 3.1.

3.6 Summary of Findings of Beneficiary Survey and Stakeholder Workshops

Beneficiary surveys. PSF undertook evaluations of the project’s impact on household expen-

diture, and reviewed the technical and economic performance of the subprojects used by project

beneficiaries. The key findings of these studies are summarized in Section 3.2 and will not be

repeated here (see Annex 5 for details on the beneficiary survey).

Stakeholder workshop. On 22 September 2014, PSF invited key stakeholders to a workshop

in Jakarta to comment on the initial findings of the project evaluation presented in this ICR.

The majority of the participants agreed that PNPM has brought major benefits to the country

and should be continued beyond 2014. To further improve PNPM-Rural, it was deemed neces-

sary to encourage innovation, strengthen the role of local governments in the implementation

of the program, and improve the monitoring and evaluation framework (see Annex 6).

4. Assessment of Risk to Development Outcome

Rating: Moderate

The “Risk to Development Outcome” is defined as the risk that the (expected) development

outcomes of the project—at the time of this assessment—will not be maintained or realized.

As described in Section 3.4, the project’s development outcomes remain highly relevant. By

achieving these outcomes, the project has contributed to establishing effective, participatory

and representative community committees that can mobilize funds through the preparation of

community development plans, thereby enabling villagers in project locations to benefit from

improved socio-economic and local governance conditions. Some risks related to the

development outcome are described below:

The political risk is considered to be low. While it is true that PNPM Rural’s achievements

would not be sustainable in the event of a withdrawal of local and national governmental

support for community-driven development, such a withdrawal is considered highly unlikely.

Both the central and local governments have indicated a strong and long-term commitment to

the financing of community-based poverty alleviation programs; and the new Village Law is a

strong testament to the success of PNPM.

The technical risk is rated “moderate.” While O&M village level committees have been

established under the project in order to inculcate a sense of ownership, certain weaknesses in

identified aspects of the operation and maintenance (O&M) for village infrastructure remains

a concern. The 2012 technical evaluation of PNPM investments demonstrates that O&M is

problematic for infrastructure projects that do not provide direct personal benefits (especially

roads and drains).

The social risk is rated as “moderate.” While local governance remains relatively strong as can

be seen in high number of households benefited from the project and high number of

infrastructure still being used by community after a number of years, the solidity of community

ownership of the project could be undermined if some issues, particularly related to transfer of

project assets to the community, are not addressed in advance. The Bank and the Government

are well aware of these potential issues and have started to discuss arrangements that are needed

19

to ensure smooth transfer of assets to the community.

Based on the above, the overall “risk to development outcome” is rated “moderate”.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance

(a) Bank Performance in Ensuring Quality at Entry


As mentioned above, the design of the project was based on the design of PNPM-Rural

III and its predecessors, which were considered at the time of appraisal to be successful

but with some corrective actions to reflect lessons learned from these projects (notably

the need for additional support to the implementing agency and the need for a robust and

integrated MIS), but otherwise using the same implementation arrangements. The Bank

kept design changes to a minimum in order to consolidate PNPM-Rural as a full-fledged

nationwide program and served as a successful incubator for pilot programs. To introduce

substantial design changes during the scaling up process was not considered prudent.

There are however some weaknesses in the project’s results framework, the most important of

which are the following:

Outcomes of investments in RLF are not reflected in the results framework. About 10%

of the PNPM-Rural IV block grants were allocated for RLFs and training. The results

framework measured overall outcomes of project investment (including infrastructure,

RLF) on households’ consumption level but did not contain any indicators to measure the

discrete impact of these investments on the socio-economic conditions of the recipients,

nor did the Bank or PSF conduct studies to measure the outcomes of these two sub-

components. However, as mentioned earlier, PMD did track and measure RLF outputs

and the RLF Pilot will be commencing its program evaluation in 2015.

Formulation of selected intermediate result indicators. The following IRIs attempted to

measure more than one achievement (emphasis added): “minimum 50% participation rate

of women and poorest community members in planning and decision-making meetings”,

“O&M arrangements are in place and/or functioning for >85% of infrastructure works”,

“>90% of local government councils provide funds and/or oversee PNPM by 2011”, “all

oversight consultants and facilitators paid on time, [and] in full every month”, “audit

sample size increases to minimum of 20% of all PNPM-Rural sub-districts and audit

results are made public”. The indicator “by 2011 4,500 sub-districts with completed

subprojects” was of limited relevance, because most loan funds would be disbursed after

2011. Moreover, the number of completed subprojects is presumably more relevant than

the number of kecamatan in which subprojects are completed.

In addition to that, as mentioned in Section 2.1, 2.3 and 3.2, the team used data from 2011

impact evaluation for one of the main PDO indicators due to loss of control groups and cost-

benefit analysis of repeating the impact evaluation. However, in view of the fact that the team

compensated for this by adopting other studies and analyses to augment its findings, the Bank’s

quality at entry is rated as “moderately satisfactory.”

(b) Quality of Supervision

Rating: Satisfactory

20

The Bank mobilized a multi-disciplinary supervision team with substantial experience in

supervising previous Bank-financed CDD projects in Indonesia. The team was able to identify

and proactively address key issues adversely affecting achievement of the PDO in an early

stage, notably problems with data accuracy and quality, budget execution and revision,

consultant staff performance, recording of potentially adverse environmental impacts,

community procurement, facilitator recruitment, and the performance of revolving loan funds.

Furthermore, the Bank was able to leverage significant trust funds to provide enhanced

implementation support, ensure timely support to GoI and develop a strong evaluation program.

Bank supervision missions resulted in a series of important recommendations, many of which

were adopted by PMD and helped improve performance in the abovementioned-areas.

At the Bank’s suggestion, PMD, inter alia, suspended block grants to RLFs with high levels of

non-performing loans or idle funds (accounting for at least 10% and 15% of total assets,

respectively), introduced more competitive bidding for project management services through

the use of Administrative Service Firms (ASFs), improved conditions for facilitators, and

helped introduce Implementation Guidelines for Social and Environmental Safeguards

(IGSES) to help improve recording of potential social and environmental impacts by

communities. Bank supervision also sought to address the shortcomings in the Results

Framework by helping to design the RLF Pilot Evaluation to assess RLF outputs and processes

as well as provide insights into women loan and saving (SPP) group characteristics. With

respect to the bunching of IRIs, given the recommended limits on KPIs in the Results

Framework, there is always a tendency to combine indicators. However, PMD was nevertheless

able to monitor and track results as the indicators were broken down into specifically

measurable pieces. It was therefore decided that no modifications would be made to the IRIs.

As such, the Bank’s quality of supervision was therefore rated as “satisfactory.”

(c) Justification of Rating for Overall Bank Performance


In light of the “satisfactory” rating of the quality of supervision, the “moderately satisfactory”

rating of the overall outcome of the project, and the “moderately satisfactory” rating of the

Bank’s performance in ensuring quality at entry, the Bank’s overall performance is rated

“moderately satisfactory.”

5.2 Borrower Performance

(a) Performance of the Government


Throughout project implementation, the project continued to enjoy the strong support of the

Government. The Government still provided majority of the project costs and has been

successful at ensuring that funds were provided in timely manner to the community – in fact,

98.8% of the funds were already disbursed by December 2012. At the same time, the quality

of policy guidance and oversight from the Government has slightly declined, resulting in less

pro-active approach that influenced timely resolution of implementation issues. The inter-

ministerial working group for PNPM (Tim Pengendali) and BAPPENAS often reacted to

suggestions for corrective action from PSF or the Bank instead of actively providing oversight,

partly because it was unable to collect and review the M&E data in a timely manner. Despite

this, the Government is still able to ensure quality of preparation and relatively smooth

21

implementation as well as compliance with Loan Agreement. Thus the performance of the

Government was consequently rated “moderately satisfactory”.

(b) Implementing Agency Performance


As the Implementing Agency, the PMD played an important role throughout the life of PNPM

Rural in the implementation of a large number of sub-projects. It was able to manage the

unprecedented scale-up of PNPM Rural from around 2,800 kecamatan in 2007 to nearly 5,000

in PNPM Rural IV (2012). PMD continued to play an essential role in ensuring that the project

was implemented on time and consistently met its fiduciary obligations. And despite the

difficulties inherent in the management of a project of the scale and complexity of PNPM-

Rural, PMD was open to ways of improving the program. During project implementation, the

agency made a series of strategic decisions that improved the quality of the outcomes and the

overall performance of the project, including the revision of the PTO, which also included

improvements to, and an overall simplification of, community procurement procedures; an

increase in the facilitators’ salary, housing and benefits allowances; and a moratorium of RLF

activities in areas which were assessed to be performing poorly.

However, some of these decisions were delayed (such as the issuance of IGSES, revision of

the PTO and increase of facilitator salaries) and were only implemented after strong insistence

from both the Bank and the Tim Pengendali. The performance of PMD was therefore rated

“moderately satisfactory”.

(c) Justification of Rating for Overall Borrower Performance


Because the performance of both the Government and the Implementing Agency was rated

“moderately satisfactory”, the overall performance of the Borrower was also rated “moderately

satisfactory.”

6. Lessons Learned

A clear and robust results framework is essential for a nationwide CDD program. An

adequate results framework will set clearly articulated targets for outcomes that the

project intends to achieve over time, and periodically measure to what extent the targets

have been achieved (instead of measuring these on an ad hoc basis or after the project has

been completed). For a project such as PNPM-Rural, relevant targets include the

outcomes of infrastructure block grants (which was already measured in the existing

results framework), the outcome of RLF investments on the formation of micro-

enterprises, the outcome of investments in training on employment of the recipient, and

well as impacts on local governments as perceived by participating communities.

Platform for support innovation. The project’s basic operating system and facilitation

platform has proven to be sufficiently robust to support the incubation of pilot programs

and to test new and innovative ideas. However, some of the experiences gained from the

implementation of pilot programs, such as RESPEK in Papua, may suggest that the project

may not be an effective platform for targeting specific groups or for customizing responses

without specialized facilitation and additional resources to ensure the achievement of pilot

outcomes. A good example where these additional resources have been successfully brought

to bear is PNPM Generasi. At the same time, from the outset, a robust institutional capacity

22

assessment and transition plans need to be well established and agreed upon by the different

implementing entities to ensure sufficient integration and reporting across the various pilot

programs as well as their sustainability.

Facilitators require incentives to promote innovation. To date, kecamatan block grants

have been allocated to a narrow range of subproject types (even though PNPM-Rural

offers an “open menu”). This discourages CDD planning (because every year, the same

narrow range of subprojects are being discussed), and adversely affects economic benefits

(as subprojects outside the familiar range of subproject types are not considered). To

encourage innovation, facilitators need to be given financial and organizational incentives

to help communities consider a broader range of subproject types.

Investments in facilitator training should be based on timely evaluations and

accompanied by competitive salaries as well as adequate resources for supervision. The

quality of facilitators is the main driver of success for community-based project

implementation. For this reason, it is not only essential that facilitators receive adequate

training before they are deployed but that they also receive continued refresher training

and mentoring to ensure that they are well oriented to revisions that are made to the

project’s Standard Operating Procedures and other technical guidance. It is also essential

that facilitators receive competitive salaries (including appropriate allowances), which

are regularly increased in accordance with prevailing market conditions. This will help

to ensure that the program can attract good candidates, has continuity and that the quality

of supervision remains high.

Procurement of Administrative Service Firms (ASFs) is cost-effective. Through this

procurement method, the implementing agency was able to provide oversight to the

program at a substantial lower cost than through traditional oversight consultants (mainly

because overheads on specialist staff are no longer included in the cost price).

The above lessons have general applicability for similar operations in Indonesia and

comparable countries.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Donors

(a) Borrower/Implementing agencies

No comments provided.

(b) Cofinanciers/Development Partners


(c) Other partners and stakeholders

On 22 September 2014, PSF organized a stakeholder workshop to discuss experiences and

lessons learned during the implementation of the project. Refer to Annex 6 for a summary of

the key issues raised during this workshop.

23

Annex 1. Project Costs and Financing

a) Project Cost by Component (in US$ million equivalent)

Components

Appraisal

Estimate

(US$ millions)

Actual/Latest

Estimate

(US$ millions)*

Percentage of

Appraisal

Kecamatan Grants 1,097.6 1,112.5 101

Facilitation Support 110.8 205.0 185

Implementation Support and Technical

Assistance

36.8 80.0 218

Project Management Support 38.5 43.8 114

Total Baseline Cost 1,283.7 1,441.3 112

Physical Contingencies 0.0 - -

Price Contingencies 0.0 - -

Total Project Costs 1,283.7 1,441.3 112

Front-end fees 0.0 - -

Total Financing Required 1,283.7 1,441.3 112

*Provisional figures (minor differences may arise between these data and final loan closing data)

b) Financing

Source of Funds

Appraisal

Estimate

(US$ millions)

Actual/Latest

Estimate

(US$ millions)*

Percentage of

Appraisal

Borrower 691.1 875.2 127

Communities 35.3 41.2 116

International Bank for Reconstruction

and Development (IBRD) 531.2 525.0 99

Co-Financiers 26.1 26.1 100

Total 1,283.7 1,441.3 112

*Provisional figures (minor differences may arise between these data and final loan closing data)

24

Annex 2. Outputs by Component

Component 1: Kecamatan Grants.

Overview. This component provided block grants to kecamatan to finance competitive grant

proposals selected according to priorities identified in community development plans for

investments in: (i) physical infrastructure, (ii) social programs, and (iii) revolving loan funds

(RLFs). From 2011-2012, the project financed more than 150,000 proposals in over 5,000

kecamatan throughout Indonesia. More than 50 percent of these proposals were developed by

women’s groups (Table A2.1). Block grant amounts ranged from IDR 750 million

(US$ 80,000) to IDR 3 billion (US$320,000) per kecamatan per year, depending on population,

poverty incidence and location (except non-poor and near-poor kecamatan that already

received block grants for more than three cycles; these would receive IDR 450m and IDR 650m,

respectively). Communities were allowed to allocate a maximum of 25% of the total kecamatan

block grant amount to revolving loan funds (provided that the recipient RLF had a repayment

rate of at least 80%).

Most block grants for infrastructure and social programs were allocated to village roads, water

supply facilities and communal sanitation, followed by investments in new education and

health facilities (Table A2.1). Infrastructure financed from the kecamatan block grants

generated temporary employment for around 1.5 - 2 million persons annually, of which more

than 80% were classified as poor.

Block grant allocations to revolving loan funds benefited around 47,000 women’s groups on

average annually (each group consisted of 10 – 15 members). Overall the performance of RLFs

improved from 2011 to 2013, as evidenced by an increase in loan repayment rates. However,

the apparent improvement in performance was partially offset by a substantial increase in the

total amount of idle funds, which was almost IDR 2.3 trillion (about US$190 million) in

December 2013 (A2.2).

The project’s investment in rural infrastructure mentioned above has, among other things, provided

access to transportation, access to basic services (such as health and clean water), and new

economic opportunities for villagers in the project areas. Access to transportation and basic services

help households reduce their transportation costs while new economic opportunities, along with

temporary employment provided by the project and revolving loan fund, contributed to higher

income and consumption rate.

Table A2.1: Size of block grants and number of proposals funded by the Project

Type of Proponent Volume

2011 2012 2013 Total

Block grants allocation

(Rp) 8,300 billion 8,100 billion 8,000 billion 24,400 billion

Proposed by women’s

groups

41,648 40,001 36,047 117,696

Proposed by other

groups

34,240 36,876 36,339 107,455

TOTAL PROPOSAL 75,888 76,877 72,386 225,151

25

Table A2.2: Types of Sub-project Financed by the Project

Type of Infrastructure Unit Volume Total

2011 2012 2013

Village roads km

18,209

16,770

15,086

50,065

Small bridges #

2,613

2,895

2,681

8,189

Irrigation systems m

3,478,114

3,515,827

2,984,978

9,978,919

#

3,164

4,662

4,880

12,706

Water supply facilities #

13,603

29,798

18,589

61,990

Communal sanitation

facilities/MCK

#

7,211

7,657

8,076

22,944

Public markets

508

688

607

1,803

Electrification #

3,396

3,399

2,732

9,527

Docks and jetties m

15,301

15,418

12,119

42,838

Education Infrastructure #

8,389

6,726

7,048

22,163

Education Non Infrastructure #

191,301

195,524

236,085

622,910

Health Infrastructure #

28,208

41,339

30,282

99,829

Health Non Infrastructure #

291,863

345,560

338,628

976,051

Source: Project MIS System (30 June 2014)

Table A2.3: Key Indicators of Revolving Loan Funds

Type of Proponent 2011 2012 2013

Women’s Loan and Savings Group (SPP)

Total capital (IDR) or accumulated

capital (since 2002)

5,809 billion 7,621 billion 8,950 billion

Repayment rate (cumulative since

2002)

91% 94% 94%

Idle Funds (IDR) 1,568billion 2,088billion 2,432billion

SPP Block Grant allocation (IDR) 1,028 billion 924 billion 506 billion

Number of SPP Group per year 67,228 48,749 24,898

Source: Project MIS System (31 August 2014)

Sustainability. According to PSF’s technical evaluation of 2012, 90% of infrastructure projects

built by the program are still functional 1- 5 years after construction. This showed that sub-

projects built by the program are well maintained and used by the community. Improvements

in local governance is likely caused by the project’s transparent processes and procedures,

26

which require community’s active participation in the project’s decision-making forum

throughout the planning, implementation and maintenance stages.

Although poor operations and maintenance (O&M) remains a major concern for village

infrastructure throughout Indonesia in general, the project has tried to mitigate this by requiring

the establishment of an O&M committee at the village level, as a means to instill a sense of

ownership for the infrastructure financed by the project. This requirement is especially useful

to maintain sub-projects with direct personal benefits such as kindergarten buildings, clean

water facilities and irrigations. This is less useful however for infrastructure types without

direct personal benefits (notably roads and drains). Some villages have taken measures to

support maintenance of such infrastructure by including a regular maintenance fund in the

village budget. This effort would likely be strengthened in the future with the implementation

of the Village Law that will see significant increase in funds allocated to villages for

development purposes.

Component 2: Facilitation Support.

This component financed 13,000 facilitators in 2012 and more than 13,400 facilitators in 2013

(25% of which are women) to support the development of community capacities for planning

and project management through support for subdistrict (kecamatan) technical, financial and

social facilitators and district (kabupaten) facilitators. Facilitators also helped strengthen inter-

village organizations and forums, and local government coordination and oversight. The

primary outcome of this component was facilitation of over 225,000 grant proposals in

accordance with PNPM-Rural guidelines. In some provinces – especially remote ones such as

Papua and West Papua, the number of facilitators was often lower than the planned number of

facilitators, partly because of delays in recruitment but also because provincial governments

were often unable to recruit facilitators with the required experience and academic background

at the salary level stipulated by PMD (this problem especially hampered the recruitment of

technical facilitators, even after lowering entry requirements). PMD has tried to address this

by increasing the salary of facilitators in 2013 and by conducting special recruitment for

technical facilitators in remote areas through the Barefoot Engineers programs. In addition, to

improve the quality of facilitators, PMD has increased the training period for new facilitators

from 11 days to the previous duration of 21 days.

Component 3: Implementation Support and Technical Assistance.

This activity was managed by a Project Management Unit (PMU) in PMD, with technical

assistance provided through a National Management Consultant (NMC) at the central level,

and Regional Management Consultants (RMCs) at the central and provincial level. This

component financed consultants to assist in various aspects of project implementation. The

outcome was the provision of the technical assistance needed to properly manage the project

and the monitor and coordinate the extensive field presence of facilitators at the local level that

is required to successfully implement this type of community-driven development project.

Component 3 also financed several studies and evaluations to support the Government with the

evaluation of PNPM-Rural, including a large-scale beneficiary survey (see Annex 5 for details).

The list of studies/evaluations conducted can be seen in the following table.

27

Evaluations/Studies Methodology Main Findings

1 Rapid Assessment of

Women’s Participation

in PNPM

(http://www.pnpm-

support.org/publication/p

npm-womens-

participation-study-

report)

Qualitative (2012) Quality of women’s participation was

mixed in study areas and is still not

maximized

Women’s involvement in the project cycle

is still low beyond the initial project

stages of socialization and needs

assessment

Women’s proposals that are funded are

usually infrastructure, rather than e.g.,

capacity building and trainings that are

requested when the local actors do a more

in-depth analysis of women’s needs

Local initiatives and strategies around

women’s participation existed but yet to

be integrated into more general local

decision making processes

2 Local Level Governance

Review

(http://www.pnpm-

support.org/publication/g

overnance-review-pnpm-

rural-community-level-

analysis-final-report)

Qualitative/ Action

Research (2012) The state of local governance in PNPM is

mixed: it is still remarkably strong, but

faces problems.

Participation rates are still mostly high,

but the quality of participation has

declined in some places.

There are weaknesses in transparency and

information sharing: performance of

accountability mechanisms in the program

varies and the incidence of serious

corruption is up.

Despite this, the ‘foundations’ of PNPM

are still strong. These foundations,

though, are being eroded by serious

pressure from ‘higher-up’ problems

related to the broader governance

environment, changes in the program

design, and problems with

implementation and management.

PNPM must address these program

design, implementation and management

issues to avoid undoing its years of good

work at community level.

3 Rate of Return Analysis

(EIRR) of PNPM-Rural

Infrastructure Sub-

Projects

(http://www.pnpm-

support.org/publication/l

aporan-akhir-studi-skala-

kecil-analisis-manfaat-

ekonomi-proyek-

infrastruktur-pnpm)

Quantitative

(economic

analysis), small

scale study in 20

villages assessing

48 sub-projects

(2012)

Similar with 2005 results (Torrens, 2005),

the EIRR varied among sub-projects with

median value of 30 – 50%

The average general income multiplier is

1.3

Sub-projects are generally 25 – 30%

cheaper than project built using typical

local government contractor

4 Village Infrastructure

Census

Quantitative

(village level Comprehensive data on basic infrastructure

availability and quality (including health

http://www.pnpm-support.org/publication/pnpm-womens-participation-study-report





http://www.pnpm-support.org/publication/governance-review-pnpm-rural-community-level-analysis-final-report





http://www.pnpm-support.org/publication/laporan-akhir-studi-skala-kecil-analisis-manfaat-ekonomi-proyek-infrastruktur-pnpm






28


(http://www.pnpm-

support.org/publication/i

nfrastructure-census-

report-infrastructure-

supply-readiness-

indonesia-achievements-

and)

census) –

piggybacking

PODES 2011

and education facilities) in nearly all urban

and rural villages in Indonesia (over 76,000

villages).

A consistent picture of geographic variation

in the supply of basic infrastructure across

Indonesia. In general, the island of Java and

the province of Bali perform best, while

local needs for investment still exist in these

regions, and particularly so in the provinces

of Jawa Barat and Banten.

The largest gaps in infrastructure supply

readiness are found for the Papua region,

the Maluku islands, NTT, as well as for the

interior of Kalimantan.

Data has been used to create a supply

readiness index for health and education (to

support PNPM Generasi in location

selection/targeting).

Further analysis will be done to calculate

financing gaps (pending data availability on

costs) and to provide more information for

PNPM-Rural in terms of location selection

and allocation of block-grants

5 Local Level Institutions

3 (LLI3)

http://www.pnpm-

support.org/publication/l

ocal-level-institution-

study-3-overview-report

Mixed-methods

(longitudinal study,

1st one in 1996 and

the 2nd in 2001); 40

villages in 3

provinces for HH

survey, 20 villages

for qualitative

methods

A decade after the last round of LLI, almost

half of the villages studied maintained the

same local capacity (defined as ability to

solve common problems collectively), about

a third experienced decline in capacity, and

a quarter saw their capacity improved

Assessing capacity from the aspects of

assets, local political economy, and sources

of capacity shifts (capturing pattern of

cooperation between actors), declined in

capacity mostly caused by deteriorating

access to natural resources, nascent signs of

reduced reciprocity, and unresponsive

village leaders who did not work in

villagers’ interest.

On the other hand, increased in capacity

was mostly due to villagers’ own efforts to

improve their livelihoods, increase control

over natural resources, and sustain

mechanisms to ensure that village leaders

were oriented towards solving collective

problems. Reformist officials and external

actors (such as NGOs) also contributed to

increased problem-solving capacity

More village heads are responsive to

villagers’ interests and are more powerful;

however, strengthening of the village head

does not translate directly to strengthening

local capacity. Additional accountability

http://www.pnpm-support.org/publication/infrastructure-census-report-infrastructure-supply-readiness-indonesia-achievements-and







http://www.pnpm-support.org/publication/local-level-institution-study-3-overview-report




29


mechanisms are needed to engender

synergy between strengthened village heads

and their constituents

12 Integration of

Community-based

Poverty Reduction

Programs at Community

Level

Qualitative, 18

villages in 3

provinces

Proliferation of programs is largely

considered to be beneficial for villagers,

with greatest benefit being the availability

of additional facilities and improved quality

of public services, improved access

(transport) and increased incomes, evenly

distributed development, availability of

capital and more business opportunities, and

improved local community and village

government capacities

Integration between empowerment

programs is generally not done

systematically, usually in the form of

sporadic initiatives to synergize programs

that have the same targets and beneficiaries

to avoid overlap. There are no initiatives at

the district level that encourage actors at the

village level to synergize by forming

program implementing

organizations/institutions at the village level

The general model of empowerment

program integration in the sample villages

is to have just one development plan per

village. In this way, empowerment

programs no longer make their own

development plans in the villages, but

follow the plan produced by the community

at the village development planning meeting

Factors enabling empowerment program

integration are: program design that is

directed towards integration, which takes

the form of program technical operating

procedures that are then communicated to

all stakeholders; having regulations and

institutions/forums that promote and

facilitate integration or collaboration

between programs; having initiatives to

build the quality of village medium term

development plans; having a district head

and village heads who understand – and are

committed to – integration; having civil

society groups (NGOs and community

leaders in the village) who have knowledge

of – and commitment to – improving the

effectiveness of village development

planning and implementation of

empowerment programs; and having the

commitment of national and local

government

30


13 Community Groups

Study

Qualitative, 6

villages/urban

wards in 5

provinces

Within the programs, community groups in

PNPM Rural and Urban performed well, but

almost no spill-over of PNPM principles

and processes outside of PNPM, both at the

village and kecamatan level (PNPM is seen

as merely project)

Community’s experience participating in

PNPM did not foster collective awareness

to demand their government (esp. village

government) to do the same despite the

benefits felt from PNPM

At the village level, power is often

concentrated at the hand of village head and

a few elites who then dominated PNPM

activities (although often with benevolent

intent). This power is not balanced with

power from other institutions (e.g. BPD) to

serve as control mechanism (checks and

balances)

Facilitation – which is key for

empowerment – is still weak and unable to

bridge the gap between elites and

community in general. Facilitators often too

busy with administrative duties and do not

have time to engage with community

Component 4: Project Management Support.

At the time of appraisal, it was envisaged that this component would provide support to

PNPM’s special programs, including pilot projects (notably PNPM-Generasi and PNPM-

Green) and crisis-response activities. As described above, PNPM-Generasi was delinked from

the project, whereas PNPM-Green never formed part of the project. PNPM-Respek and PNPM-

BKPG grants were directly added onto regular block grants for PNPM-Rural IV, and their

implementation did not require dedicated project management support. As a result, there were

few Bank-financed outputs recorded under this component.

31

Annex 3. Economic and Financial Analysis

Introduction. In 2012, PSF commenced a study to analyze the economic benefits of small-

scale infrastructure financed by PNPM-Rural. This annex summarizes the final report of the

consultant, which contains an analysis of a sample of 48 small-scale infrastructure projects

equally spread out over three sectors that have thus far absorbed most the Government’s

investment in PNPM-Rural: water supply, roads and bridges, and irrigation.25 The sample was

taken in 39 villages in four provinces (East Nusa Tenggara, West Sulawesi, West Kalimantan

and Central Java), and covered projects constructed between 2007 and 2011. The report

presented the results of three types of economic and financial analysis: (i) economic internal

rate of return, (ii) general income multiplier, and (iii) cost-efficiency. The study used followed

the methodology that was developed for a similar study conducted in 2005.26

Economic internal rate of return. The World Bank’s Handbook on Economic Analysis of

Investment Operations requires the real economic rate of return (EIRR) of Bank-financed

subprojects to be equal or higher than the opportunity cost of capital of the country (which the

Bank estimates at 12% per annum for developing country members, including Indonesia). The

average unweighted economic internal rate of return (EIRR) of the 48 projects analyzed by the

consultant was approximately 97%, far above the minimum required rate, and also far above

the target for the project itself (30%). Average unweighted EIRRs were highest for irrigation

projects (112%), followed by roads and bridges (94%) and water supply (87%). For reasons

not explained in the report, the average unweighted EIRR for projects in Central Java (239%)

was far higher than in the three other provinces, where average EIRRs were in the order of

35%-50% (this ICR considers EIRRs for subprojects in Central Java as outliers). The median

EIRR was approximately 40%. List of EIRRs by sub-projects and by province can be seen in

the following table.

Type of Infrastructure Number of Projects Average EIRR

Roads/ Bridges

- Central Java 4 241.8%

- West Kalimantan 4 39.78%

- West Sulawesi 4 30.75%

- NTT 4 64.80%

Irrigation




- NTT 4 38.83%

Clean Water Channels




- NTT 4 34.25%

Total Projects 48

25 Laporan Akhir Studi Skala Kecil Analisis Manfaat Ekonomi Proyek Infrastruktur PNPM Mandiri

Perdesaan. PNPM Support Facility. July 2012. 26 Economic Impact Analysis of Kecamatan Development Program Infrastructure Projects. Anthony

Torrens. January 2005.

32

The average EIRR of subprojects financed by PNPM-Rural IV are considered likely to be lower

than EIRRs presented in the study, the most important being:

The study did not take into account subprojects that were no longer functional, which

consist of about 11% of all subprojects financed by PNPM-Rural during 2007-2011.27 If

benefits are reduced by 11%, the EIRR of a subproject with an EIRR of 40% will drop to

37% (40% was used because this is the median EIRR of the 48 subprojects in the sample.) 28

The study assumes that benefits generated by a subproject remains constant over time

until the end of its service life, even though benefits will in practice gradually decline

because of inadequate maintenance. Assuming that benefits decrease by 5% per year, the

EIRR drops from 37% to 33%.

The study assumes an unusually low shadow wage rate factor for unskilled labor (0.13,

whereas other WB-financed projects in Indonesia and other developing countries assume

a factor of 0.6 to 0.8). Assuming a more realistic conversion factor of 0.7 and average

labor content of 25%, the EIRR drops further, to 28%.

After these adjustments, it is likely that the average EIRR of most project-financed subprojects

will remain comfortably above the 12% economic opportunity cost of capital, but not

necessarily above the project target of 30%. It is also worth noting that EIRRs of subprojects

financed by kecamatan grants are a priori expected to decrease over time, as most kecamatan

in the country have by now received several rounds of block grants, and may have used the

first rounds to finance the most economically viable subprojects.

General income multiplier. The second part of the consultant’s report presents estimates of

the income multiplier of infrastructure projects financed by PNPM-Rural IV. Page 22 of the

report defines the term income multiplier as “the circulation of money in a village caused by

the utilization of funds for the development of PNPM infrastructure projects”. It was assumed

that “the circulation of money” referred to “the increase in circulation of money in the year the

infrastructure project was constructed”, although the report does not explicitly say so. The

highest weighted multipliers were observed in the irrigation sector (1.34), followed by water

supply (1.28) and roads (1.18).

Cost efficiency. The cost of the 48 projects was compared to theoretical cost of the same

projects, but assuming that they would have been undertaken by local government contractors.

The results of the analysis are comparable to the findings of similar studies on cost savings of

community-driven development projects, especially when undertaken by the communities

themselves (swadaya) instead of using community contracting. Cost savings were highest for

projects in the roads and bridges sector (22%-33%), followed by water supply (19%-28%), and

irrigation (20%-25%).

27 Technical Evaluation of Infrastructure – PNPM-Rural & Other Funding Sources. PSF. July 2012. 28 Assuming a subproject with constant benefit streams during an economic lifetime of 10 years

(assumptions that were also used for the estimation of the EIRRs of the 48 sample subprojects).

33

Annex 4. Loan Preparation and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/

Specialty

Lending

Jan Weetjens Lead Social Development

Specialist

GSURR Sector Manager

Susanne Holste Lead Social Development

Specialist

GSURR Task Team Leader

Sentot Surya Satria Senior Social

Development Specialist

GSURR Co-TTL

Natasha Hayward Senior Social


GSURR Monitoring and Evaluation

Ahsan Ali Lead Procurement

Specialist

GGODR Procurement

Achmad Zacky Wasaraka Procurement Analyst GGODR Procurement

Zulfi Novriandi Operations Analyst GSURR Procurement

Juan Martinez Senior Social Scientist GSURR Social safeguards

Indira Dharmapatni Senior Operations Officer GSURR Environmental Safeguards

Yogana Prasta Operation Adviser EACIF FM and Disbursement

Unggul Suprayitno Senior. Financial

Management Specialist

GGODR Financial Management

Melinda Good Senior Counsel LEGES Legal

Alexander B. Setiadji Social Development

Specialist

GSURR Governance

Supervision /ICR

Jan Weetjens Lead Social Development

Specialist

GSURR PSF Manager

Susanne Holste Lead Social Development

Specialist

GSURR Task Team Leader from June

2011 to September 2013

Sonya Woo Senior Operations Officer GSURR Task Team Leader from

October 2013 to project

closing in June 2014

Sentot Surya Satriat Senior Social


GSURR Operations

Yogana Prasta Operation Adviser EACIF Operations

Sentot Surya Satria Senior Social


GSURR Operations

Sri Kuntari Social Development

Specialist

GSURR Operations

Ahsan Ali Lead Procurement

Specialist

GGODR Procurement

Yash Gupta Senior procurement

Specialist

GGODR Procurement

34

Names Title Unit Responsibility/

Specialty

Juan Martinez Senior Social Scientist GSURR Social Safeguards

Achmad Zacky Wasaraka Procurement Analyst GGODR Procurement

Zulfi Novriandi Operations Analyst GSURR Procurement

Unggul Suprayitno Senior. Financial

Management Specialist

GGODR Financial Management

Festina Lavida Operations Officer GSURR Financial Management

(Audit)

Alexander B. Setiadji Social Development

Specialist

GSURR Governance

Hanggar Irawan Financial Management

Analyst

GSURR Budget & Disbursement

Ani Himawati Operations Analyst GSURR Social Safeguards

Favio Chaves Consultant GENDR Environmental Safeguards

Franciscus Prahastanto Operations Analyst GSURR Operations

Griya Rufiane Operations Analyst GSURR Operations

Muslahuddin Daud Social Development

Specialist

GSURR Social Development

Robert Anders Anderson Operations Officer GSURR Operations (Papua)

Joseph Pieter Seumahu Operations Analyst GSURR Operations (Papua)

Lily Hoo Monitoring & Evaluation

Specialist

GSURR ICR Team Leader

Yoseph Lucky Consultant GSURR Facilitation and Training

Octaviera Ratna Herawati Consultant GSURR Monitoring

(b) Staff Time and Cost

Stage of Project Cycle

Staff Time and Cost (Bank Budget Only)

No. of staff weeks US$ Thousands (including

travel and consultant costs)

Lending

FY11 27.55 206.77

FY12 2.00 18.03

Total: 29.55 224.80

Supervision/ICR

FY12 10.77 75.06

FY13 17.53 120.86

FY14 9.98 106.14

FY15 - 1.57

Total: 38.28 303.62

35

Annex 5. Beneficiary Survey Results

Introduction. In 2012, PSF commissioned a study to assess the incidence of benefits of

PNPM-Rural. The annex first describes the background, purpose and objectives and

methodology of the study. It then summarizes the main findings.

Background. The approach of PNPM-Rural to delivery of public infrastructure and access to

credit have placed communities in control of a development process attempts to ensure a more

participatory, transparent and accountable decision-making process for the use of community-

level development funds. Recent research on the impact of PNPM-Rural has demonstrated

positive impacts on key household welfare indicators such as consumption per capita, transition

out of poverty and access to health and education services.29 While these findings indicate an

average overall positive benefit among households residing in kecamatan which receive PNPM,

they are not able to detail how these benefits are distributed at the kecamatan and village level.

Although the potential for elite capture at the local level is present for any approach to project

implementation, there has long been a concern in the CDD literature that village level control

of the development process could lead to systematic distribution of benefits toward elites and

those connected to them.

Additional qualitative research on PNPM has discussed the perception of benefit from the

program with respect to prevailing supply of basic infrastructure and poverty status. In areas

where infrastructure gaps exist, the needs/interests of the poor with respect to PNPM

infrastructure projects are aligned with communal interests to address gaps. However, in

villages where sufficient infrastructure exists, needs/interests of the poor tend to be less aligned

with communal proposals opting for skill and capacity building for employment or

health/education service delivery instead of standard roads/bridges/irrigation which occupy

70% of all project block grant funds under PNPM. Moreover, the findings also showed that

PNPM was frequently not perceived as a poverty reduction program by community members

in general and among the poor. Rather, the project is perceived as for the community, in some

cases in direct counterbalance to household-based poverty programs.30

The findings above have significant potential with respect to the future impacts and targeting

of PNPM to locations where it has the greatest benefit for the poor and the community as a

whole. If the project is perceived as working more effectively in certain contexts such as in

poor kecamatan and areas with low levels of existing infrastructure, it suggests options for

potential targeting of the program more heavily to these areas. However, while these questions

have been looked at using well-developed qualitative research methods, the overall sample size

was limited to the nature of the exercise. The Incidence of Benefit Study attempts to further

investigate these findings by looking at patterns of perception of benefit and targeting of PNPM

using a large sample quantitative approach with the goal of determining the validity of the

qualitative results as well as providing a more detailed understanding of the issues presented.

Purpose and objectives. The primary objective of the Incidence of Benefit study is to

determine the of (1) distribution of perceived benefits and (2) the perception of PNPM as a

29 Voss (2013). PNPM Rural Impact Evaluation, April 2012. PSF. 30 Syukri et al (2012). Research Report: A Qualitative Study of Impact of PNPM Rural. PSF.

36

poverty reduction program and (3) change in consumption per capita, across a set of defined

sub-groups based on the following factors:

• Gender

• Poverty

• Education of adults

• Relationships to Village Government/Tokoh Masyarakat

• Household-based poverty aid (RASKIN, BLT etc.)

Methodology. The survey was fielded in 600 villages across 3 provinces: Lampung, South

Sumatera, and Central Java. Ten households were selected randomly in each village during the

survey. Power calculations suggested that the number of clusters (villages) is more than

sufficient to create valid inferences with respect to sample distributions of indicators.

Main findings. Table A5.1 summarizes the main findings of the beneficiary survey by income

level, as measured by monthly per capita expenditure. In general, the majority of respondents

state that PNPM-financed infrastructure meets community needs and over 60% believe that the

infrastructure helps to improve access, followed by cleaner living environment (32%) and

protection against floods or landslides (25%). Most respondents also state that PNPM-financed

economic activities (i.e. revolving funds) meet community needs. However, a relatively small

number of respondents believe that revolving funds increase business capital (25%) or improve

business (13%). There is little variation in opinions about PNPM’s perceived benefits across

income groups, although respondents in higher income quintiles tend to be more favorable

about the program’s benefits than those in lower quintiles.

37

Table A5.1: Selected Findings of Beneficiary Survey

(percent of all respondents)

Type of Benefit All Quintile (1 = lowest, 5 = highest)

1 2 3 4 5

Households receiving benefits from PNPM

infrastructure activities

76.1 69.5 71.8 76.9 78.8 83.7

Benefits received:

- Open/improve access to transport 66.0 60.7 63.0 65.9 68.5 71.9

- Protect against floods/landslides 24.7 22.6 20.7 26.0 25.6 28.8

- Cleaner living environment 31.7 26.8 28.2 31.4 33.8 38.3

- Improve sanitation 9.8 7.7 9.4 10.5 9.9 11.2

- Improve health 10.0 8.8 8.5 11.1 10.3 11.2

- Address daily needs 3.9 3.5 3.5 3.8 4.3 4.3

- Reduce workload 3.2 2.9 3.3 3.7 3.2 3.0

- Improve business 4.8 4.1 4.0 5.1 5.2 5.7

PNPM infrastructure activities meet community

needs 80.4 75.6 76.7 80.8 82.2 86.8

Benefits received by communities from PNPM

infrastructure activities

- Open up business opportunities 11.5 10.9 10.6 12.2 11.5 12.1

- Improve access 73.7 69.8 70.0 74.1 75.5 79.0

- Address primary needs 39.2 36.8 36.4 38.6 38.3 46.0

- Help reduce poverty 7.5 5.9 5.7 8.4 8.3 9.4

Households receiving benefits from PNPM

economic activities

38.8 34.2 40.4 38.2 40.9 39.5

Benefits received:

- Address daily needs 10.4 12.6 13.3 10.1 9.4 7.6

- Reduce workload 3.1 3.5 3.8 2.7 3.0 2.7

- Improve business 13.7 10.5 12.2 16.0 15.3 13.7

- Increase business capital 25.4 18.7 24.9 25.9 28.9 27.3

PNPM economic activities meet community

needs 89.2 88.9 89.9 89.5 90.1 87.7

Benefits received by communities from PNPM

economic activities

- Open up business opportunities 78.9 74.7 78.2 79.9 82.0 79.0

- Improve access 4.3 4.1 3.3 5.0 4.6 4.3

- Address primary needs 9.0 11.9 9.9 9.8 6.9 7.3

- Help reduce poverty 19.8 20.0 18.5 23.4 18.5 18.6

Source: Overview Findings of Incidence of Benefit Survey (2014)

38

Annex 6. Stakeholder Workshop Report and Results

On September 22, 2014 PSF organized to a workshop in Jakarta to collect comments on the

initial findings of the project evaluation presented in this ICR. Attendees consisted of repre-

sentatives of the implementing agency, as well as consultants and facilitators involved in the

implementation of PNPM-Rural IV. To further improve PNPM-Rural, it was deemed necessary

to encourage innovation, strengthen the role of local governments in the implementation of the

program, and improve the monitoring and evaluation framework. Other key observations on

the Bank’s initial findings and lessons learned can be summarized as follows.

Observations about the Bank’s initial findings.

The project has been successfully implemented and has achieved most of its target

indicators. It also remains one of the most popular programs at the community level

which community still wants to see continuing to help fulfill their basic needs. PMD

agrees that this has been the case despite shortcomings in program implementation.

Although most of the targets have been achieved, the results framework does not

adequately capture capacity building elements which are the core business of PMD.

There is anecdotal evidence that PNPM has empowered people, made them more

independent, capable of collective action, and has developed behavior and mindsets to

be involved in solving their development problems compared to before. However,

these achievements have not been captured in the results framework. Existing

indicators place too much emphasis on infrastructure outputs, although achievements

in infrastructure could illustrate the increased capacity of communities to build and

maintain PNPM-financed infrastructure. The ultimate goal for PMD is not the number

of infrastructure subprojects that were constructed, but a more empowered community.

It is recommended that the ICR provide lessons onprocess-related performance

indicators to capture the empowerment aspects of the program and identify indicators

that need to be changed or added to measure the impacts of community empowerment

and sustainability of the project at the community level for PNPM Rural 2012 - 2015.

PMD has implemented several pilot projects to help improve PNPM Rural

performance. However, the ICR report only covers the agreed framework in PNPM-

Rural IV loan agreement. PMD will utilize the Government’s Project Completion

Report to cover other relevant issues, such as implementation of PNPM Integrasi pilot

project (which aims to link PNPM Rural planning processes to the regular village

development planning process), which has now transitioned to a regular program; and

the pilot project to support better RLF implementation and reporting.

Some indicators in the results framework measured more than one achievement. The

intermediate results indicator “participation rate of women and poorest community

members in planning and decision-making meetings” for example, is confusing – does

the project want to measure participation of women or poor? It is thus recommended

that separate data––participation of women and the participation of poor – are reported

in the ICR.

39

Observations about lessons learned.

Monitoring and evaluation requires improvement. PMD, NMC and RMCs have

different views about targets, the objectives of measurement, and how to report the

measurement results. There is also a lack of clarity about responsibilities for

monitoring and evaluation. In addition, the MIS is still not functioning and has

influenced PMD’s capability to monitor and redress problems in a timely manner.

PMD will work with AusAid to ensure that the new MIS will be operational soon to

allow more timely and regular data collection for monitoring and evaluation purposes.

More relevant indicators are needed to adequately capture program sustainability.

Information on outputs may be used to illustrate the sustainability of PMPM-Rural but

it will be better to have indicators that can capture the process of empowerment as well.

Regional government and local institutions play an important role in supporting the

implementation of PNPM-Rural IV. Hence, the program needs to continue working

closely with the regional governments and build the capacity of local institutions to

strengthen program performance even further.

A special approach is needed for the implementation of PNPM Rural in remote areas.

Although this has been discussed for a while, PMD needs to take immediate action to

start operationalizing the special arrangements for remote areas.

A new approach is needed to produce more diverse types of activities in the field.

Because most facilitators have a technical background, activities in the field have

focused on building infrastructure. A new approach is needed to address the lack of

variety, such as MP3KI (the Master Plan for Accelerating and Expanding Poverty

Reduction in Indonesia) and P2B (the Livelihood program).

40

Annex 7. Borrower's Completion Report and Comments on Draft ICR

(a) Borrower’s Completion Report

On December 16, 2014, the Borrower submitted a project completion report to the Bank, the

executive summary of which is copied below.

Implementation Completion Report PNPM III

National Rural Community Empowerment Program (PNPM Mandiri Perdesaan)

Directorate General of Community and Village Empowerment

Ministry of Home Affairs

Executive Summary (Brief Version)

GENERAL

PNPM Mandiri is a national community empowerment-based poverty reduction program

implemented through system harmonization and development, as well as program mechanism

and procedure, provision of facilitation and stimulus funds to encourage community initiative

and innovation for sustainable poverty reduction efforts.

PNPM Mandiri Rural mission is to: 1) increase community capacity and its institutions, 2)

institutionalization of participatory development system, 3) enhance the effectiveness of local

government roles and capacity, 4) increase the quality and quantity of basic social

infrastructure and facilities and communities’ economy, and 5) develop partnership networks

in development.

FOCUS AND SCOPE OF PROGRAM ACTIVITY

PNPM Rural key activities are those that pertain to fulfillment of community basic needs,

including: infrastructure development, economy, education and health. Activities that are

prohibited in the program include those that relate to ethnic, racial and religious issues (e.g.

building of religious facilities), environmental destruction, narcotics use, personal and group

interest, and so forth. This list of items is called the “Negative List” in the program.

PNPM Mandiri Rural FY 2011, 2012 and 2013 (PNPM IV) has been implemented in 32

provinces with target locations that include all sub-districts in Indonesia that is implemented

in stages and does not include sub-districts under the “problematic” category. Target groups in

PNPM Mandiri Rural include: a) poor rural communities, b) rural community institutions, and

c) local community institutions. PNPM-Penguatan (supplement programs to strengthen the

core PNPM programs) and other supporting programs include sector-based community

empowerment programs or replications to support poverty reduction.

The implementation of these programs at the community level refers to PNPM Mandiri Rural

policy framework. Aside from core planning at the sub-district level and the block grant

scheme, PNPM Mandiri Rural IV also provided grants to four specific programs:

41

a. PNPM-Generasi, an incentive grant scheme to support the achievement of Millenium

Development Goals that relate to health and education;

b. PNPM-Green, a pilot program that funds grants and technical facilitation to promote

community investments in natural resource management and renewable energy;

c. PNPM-BPKG, a regional block grant scheme funded by the Government of Nanggroe

Aceh Darussalam specifically aimed to sub-districts in that province;

d. PNPM-RESPEK, also a regional block grant scheme that includes Papua and West Papua

Provinces and funded by the Provincial Governments. PNPM-RESPEK activities fund

projects in sectors including: (i) nutrition and food security, (ii) education, (iii) primary

health services, (iv) rural infrastructure, and (v) economic livelihoods.

LOCATION AND BUDGET ALLOCATION

PNPM Mandiri Rural Regular FY 2011 included 32 provinces, 393 districts and 4,979 sub-

districts. The total block grant allocation was IDR 8.28 trillion. The total block grant allocation

for this activity was supported by National Budget financing in the amount of IDR 6.56 trillion,

comprising of allocations for Operational Activity Funds for Planning (DOK Perencanaan) in

the amount of IDR 276 billion, and Operational Activity Funds for Training (DOK Pelatihan)

in the amount of IDR 360 billion. PNPM Mandiri Rural Regular FY 2012 included 32

provinces, 393 districts and 5,094 sub-districts. The total block grant allocation was IDR 8.14

trillion. This block grant allocation was supported by National Budget financing in the amount

of IDR 7.23 trillion, and Regional Budget financing of IDR 913 billion. The total DOK

allocation was IDR 543 billion. PNPM Mandiri Rural Regular FY 2013 included 32 provinces,

392 districts, and 5,146 sub-districts. The total activity block grant allocation was IDR 8.42

trillion. This block grant allocation was supported by National Budget financing in the amount

of IDR 7.99 trillion. The total DOK allocation was IDR 533 billion.

The number of villages funded and number of PNPM-IV beneficiaries

Year 2011 2012 2013

Villages funded (village) 62,952 61,598 63,169

Beneficiaries (persons) 31,359,160 32,024,728 28,671,752

Poor beneficiaries (persons) 17,617,445 18,754,952 15,850,403

BUDGET DISBURSEMENT PROGRESS

The national recap of block grant disbursement for PNPM Mandiri Rural Regular FY 2011-

2013 both from National and Regional Budgets reached IDR 23.93 trillion, equivalent to

97.97% of the total activity block grant ceiling of IDR 24.42 trillion. The total PNPM Mandiri

Rural block grant funds disbursed for regular FY 2011 to FY 2013 was IDR 23.76 trillion or

99.30% of the total disbursed funds. The national recap of DOK PNPM Mandiri Rural

disbursement for Regular FY 2011 – FY 2013 was IDR 1.71 trillion. The total DOK

disbursement was IDR 1.56 trillion, or an equivalent of 92.13% of the total disbursed funds.

INFRASTRUCTURE ACTIVITIES

Infrastructure activities in FY 2011 – 2013 are almost all completed. However, there are some

infrastructure development activities that continue to see technical obstacles and management

issues, although not many. Based on the data input from Protan (the management information

42

system), the infrastructure built during FY 2011 included 44,695 proposals with budget of IDR

6.48 trillion and almost IDR 194 billion of community contributions. In FY 2012, there were

47,054 total proposals with a budget of IDR 6.84 trillion and IDR 193 billion of community

contributions. In FY 2013, there was a decrease in the number of infrastructure activities to

45,379 proposals, absorbing block grants of IDR 6.75 billion and IDR 205 billion of

community contributions.

FINANCIAL MANAGEMENT OF REVOLVING LOAN FUND

During PNPM IV implementation, the Revolving Loan Fund (RLF) activities have been

running for 14 years since 1998. With time, there have been some improvements in the

technical operational implementation and implementation policy made. In terms of community

institutions, the RLF activity has been protected by establishing community as the owners

through the Inter-Village Communication Body (BKAD) institution (Law 32 of 2004,

Government Regulation 72 of 2005 and Minister of Home Affairs Circular of 2006). Currently,

the RLF activity has developed well and strongly. Some of the results are as follows.

In terms of activity management, the total RLF funds managed by communities through UPK

currently has exceeded IDR 8.5 trillion (block grants and capital seeding) with a total

repayment above 91% (Productive Economy Activities (UEP) and RLF), almost 7% higher

than in the previous year. This data was taken from 4,672 sub-districts that reported.

PROGRAM IMPLEMENTATION AUDIT

Results of the internal audit of PNPM Mandiri Rural FY 2011 implementation up to the fourth

quarter of 2012 are as follow:

a. Regional Management Consultant (RMC) level, included 50 districts from 79 target district

locations (64%) and 91 sub-districts from 252 target sub-district locations (36%) with a

sample of 151 villages.

b. Provincial level, included 338 districts from 393 target districts (86%) and 1,287 sub-

districts from 1,324 target sub-districts (97%) with a sample of 2,331 villages.

c. District level, included 3,752 sub-districts from 5.037 target sub-district locations (74.49%)

and a sample of 10,427 villages

The audit findings are mainly dominated by incompliance to applicable rules (PTO and PTO

Elaboration), such as incompliance to principles and procedures of goods and services

procurement, incompliance to the principle of priority-based-on-need in the planning process,

incompliance provisions and mechanisms of revolving fund management, and so forth.

External Audit. PKP conducted annual audits and control reviews of program financial

management. The audit procedure was complicated by the rather large contribution from

regional government funds (DDUB) that is audited by inspectorates. The audited sample of

sub-districts was around 20 percent of the total PNPM Mandiri Rural locations and the joint

audit results at the district level was posted at the PNPM Mandiri Rural website, in the media

and by NGOs. PNPM Mandiri implementation is actively monitored by NGOs and the media

(national and local newspapers, community radios, social media and several local TV stations).

PNPM is involved with media, and publishes reports of fraud and corruption and actions taken

43

to resolve these cases, where special advertorial funds are made available in every province to

publish the outcomes and processes of case handling.

Complaint and Problem Handling. PNPM Mandiri Rural has redesigned the complaint and

problem handling mechanism that is enshrined in the Complaint Handling System SOP. The

goal is to resolve, as much as possible, the complaints in the locations where they are reported.

Beneficiaries, or anyone concerned about this program, may easily file a complaint, through

facilitators by sending an SMS, email, letter, and so forth. Grievances can be reported through

SMS (phone number 085710301234 and 082112109495), e-mail ([email protected]) or a

form available on www.pnpm-perdesaan.or.id.

There has been an increase in the number of cases solved between 2011 until 2013, but there

has been a decrease in the number of new cases reported. 2,684 new cases were reported in

2011, 1,803 in 2012, and 1,578 in 2013 (even though the total amount of funds misused and

the number of abuse cases remained constant). On average 40.53% cases were resolved. In

2011, 42.65% of cases were resolved, in 2012 33.45% and in 2013 45.49% cases.

Sanctions. Sanctions for abuse/corruption have been integrated in the PNPM Mandiri Rural

cycle and, most importantly, socialized widely. PNPM facilitators, consultants, UPK or TPK

and other actors risk to be terminated or prosecuted if they commit corruption. In unresolved

corruption cases, the disbursement and payment of funds to villages can be suspended until

proper measures are taken. This has proven to be an effective mechanism to prevent and a good

way to promote the resolution of corruption cases at the village level. If there is no proper

resolution, a sub-district can be declared “potentially problematic location”. Sanctions will also

apply if program funds are used for political or political party activities. However, every

potentially problematic sub-district is still provided with facilitators to help resolve the problem

and facilitated the planning process, even though the funds are suspended until the problem is

resolved.

LEARNING

Some of the experiences gained throughout the implementation of PNPM Mandiri Rural,

especially for 2011 until 2013, have provided a lot of practical experience for all parties and

actors, both those who are involved directly and indirectly in the implementation. These

experiences has the positive potential to be used as material for study, reflection, as well as

development, or to enrich the knowledge to be contributed or replicate in other programs or

activities regarding poverty reduction and community empowerment in rural areas. Some of

the lessons learned in the implementation of PNPM IV include:

a. Providing an Administrative Service Firm (ASF) can save cost. Through this service

provision method, the implementing institution can provide program supervision with much

lower cost compared to using a traditional supervising consultant (especially due to the

overhead of specialist staff that are no longer included in the financing).

b. The Social and Environmental Safeguard Policy is an effective way to help promote public

awareness to social and environmental impact potential. Additional technical training in

project safeguards implementation need to be included in the pre-deployment and on-the-

job training for facilitators.

http://www.pnpm-perdesaan.or.id/

44

c. A clear and robust outcome framework is very necessary for a national CDD program. An

adequate outcome framework will set clear targets, highly articulated to elaborate the

outcomes that the project intends to achieve from time to time, and periodically measure the

extent to which the targets have been achieved (rather that measure them in an ad hoc

manner or only after the project is completed). For projects such as PNPM Mandiri Rural,

the relevant target include outcomes from infrastructure grants (that have been measured in

the existing outcome framework), RLF investment outcomes in establishment of micro

enterprises, outcomes from job training investment beneficiaries, as well as impacts on the

local government as perceived by the participating communities.

(b) Borrower Comments on Draft ICR


45

Annex 8. Comments of Cofinanciers No comments provided.

46

Annex 9. List of Supporting Documents

Project Appraisal Document. World Bank. June 2011.

Aide Memoires (various). World Bank. 2012-2014.

Implementation Status Reports (various). World Bank. 2012-2014.

Economic Impact Analysis of Kecamatan Development Program Infrastructure Projects.

Anthony Torrens. January 2005.

Country Partnership Strategy for Indonesia FY2009-2012. World Bank. August 2008.

Country Partnership Strategy for Indonesia 2013-2015. World Bank. December 2012.

First Quarterly Report for PNPM Rural. MoHA. June 2014.

Indonesia’s PNPM Generasi Program – Final Evaluation Report. PSF. March 2012.

PNPM-Rural Impact Evaluation. John Voss. PSF. April 2012.

Research Report :a Qualitative Study : The Impact of PNPM Rural East Java, West

Sumatra, Southeast Sulawesi. Muhammad Syukri et al. PSF. April 2012

Peta Jalan PNPM Mandiri – Menuju Keberlanjutan Program Pemberdayaan

Masyarakat. Kelompok Kerja Pengendali Program Penanggulangan Kemiskinan Berbasis

Pemberdayaan Masyarakat (PNPM Mandiri), Tim Nasional Percepatan Penanggulangan

Kemiskinan (TNP2K). Jakarta, September 2012.

PNPM Gender Study 2012: Increasing the Quality of Women’s Participation – Final

Report. PNPM Support Facility. 2012.

Laporan Akhir Studi Skala Kecil Analisis Manfaat Ekonomi Proyek Infrastruktur PNPM

Mandiri Perdesaan. PNPM Support Facility. July 2012.

Technical Evaluation of Infrastructure – PNPM-Rural & Other Funding Sources. PNPM

Support Facility. July 2012.

Overview Findings of Incidence of Benefits Survey. PSF. October 2014.

1