Indoneisa, Participatory Irrigation Sector Project

EvaluationIndependent

Performance Evaluation

Report

Indonesia: Participatory Irrigation Sector Project

Reference Number: PPE:INO 2016-03 Project Number: 32359 Loan and Grant Numbers: 2064, 2065 and 4299 Independent Evaluation: PE-785

Performance Evaluation Report February 2016

Indonesia: Participatory Irrigation Sector Project

This document is being disclosed to the public in accordance with ADB’s Public Communications Policy 2011.

NOTE

In this report, “$” refers to US dollars.

In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgments as to the legal or other status of any territory or area.

The guidelines formally adopted by IED on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of the management of IED, there were no conflicts of interest of the persons preparing, reviewing, or approving this report.

Director General V. Thomas, Independent Evaluation Department (IED) Director W. Kolkma, Independent Evaluation Division 1, IED

Team leaders A. Brubaker, Senior Evaluation Specialist, IED E. Gozali, Principal Evaluation Specialist, IED (until 27 Nov 2015) Team members M. Dimayuga, Senior Evaluation Officer, IED C. Marvilla, Evaluation Assistant, IED

Abbreviations

ADB - Asian Development Bank DGAIF - Directorate General of Agricultural Infrastructure and Facilities DGWR - Directorate General of Water Resources DS1 - first dry season DS2 - second dry season EIRR - economic internal rate of return ha - hectare HYV - high yielding variety IAMIS - irrigation asset management information system IED - Independent Evaluation Department IOB - Inspectie Ontwikkelingssamenwerking en Beleidsevaluatie

(Policy and Operations Evaluation Department) IPDMIP - Integrated Participatory Development and Management of

Irrigation Project O&M - operation and maintenance PCR - project completion report PISP - Participatory Irrigation Sector Project PPER - project performance evaluation report RP2I - Rencana Pengembangan dan Pengelolaan Irigasi (district

irrigation development and management plan) SDR - special drawing right SI-PAI - Sistim Informasi Pengelolaan Aset Irigasi (irrigation asset

management information system) WISM - Water Resources and Irrigation Sector Management WUA - water users association WUAF - water users association federation

Currency Equivalents

Currency Unit – rupiah (Rp)

At Appraisal (3 November 2003)

At Completion (25 March 2013)

At Independent Evaluation (30 June 2015)

Rp1.00 = $0.000117 $0.000103 $0.000075 $1.00 = Rp8,513 Rp9,700 Rp13,340

Contents

Acknowledgements vii

Basic Data ix

Executive Summary xi

Chapter 1: Introduction 1 A. Evaluation Purpose 1 B. Process 2

Chapter 2: Design and Implementation 3 A. Formulation 3 B. Rationale 3 C. Cost, Financing, and Implementation Arrangements 5 D. Outputs 7

Chapter 3: Performance Assessment 13 A. Overall Assessment 13 B. Relevance 13 C. Effectiveness 15 D. Efficiency 17 E. Sustainability 18

Chapter 4: Other Assessments 21 A. Impact 21 B. ADB Performance 22 C. Borrower and Executing Agency Performance 22

Chapter 5: Issues, Lessons, and Follow-Up Actions 24 A. Issues and Lessons 24 B. Follow-up Actions 26 Appendixes 1. Evaluation Matrix 29 2. Coverage of Primary Data Gathering 34 3. Project Performance Against Design Framework 35 4. Institutional Assessment 41 5. Economic Assessment 50

Supplementary Appendix (available upon request)

A. Economic Analysis Model

Acknowledgements

This report is a product of the Independent Evaluation Department (IED) of the Asian Development Bank (ADB). Emile Gozali, Principal Evaluation Specialist, led the evaluation, with support from Ma. Juana Dimayuga (Senior Evaluation Officer) and Christine Grace Marvilla (Evaluation Assistant).

The IED team worked in partnership with a team from the Policy and

Operations Evaluation Department (IOB) of the Ministry of Foreign Affairs of the Government of the Netherlands, comprising IOB inspectors Ferko Bodnar and Joep Schenk, and consultants Jetze Heun and Tangkas Panjaitan. The IOB team, which focused their evaluation of the impact of the project, was assisted by Equator, a community development foundation.

National consultant, Suhardi Suryadi prepared portions of the draft and

supported the institutional analysis. Agricultural development specialist consultant, Frederick Roche contributed to the evaluation and produced a first version of the economic analysis of project results based on project completion report data. The analysis was subsequently modified and finalized by IED staff, incorporating additional data supplied by the IOB team based on their field survey of the project.

The evaluation team is grateful to ADB staff members Eric Quincieu, Pantja

Putih Wardani, Steven Tabor, and Ferny Suhandi, as well as officials from the Government of Indonesia, for engaging in dialogue, providing support, and sharing information. The team also acknowledges the substantial help provided by Willem Van Diest and his colleagues, who are former consultants to the project.

The report was peer reviewed by Andrew Brubaker, Senior Evaluation Specialist,

and Garrett Kilroy, Evaluation Specialist of IED. Mary Grace Agapito, Evaluation Officer, provided help with the analysis of safeguards measures. The report was prepared under the overall guidance of Vinod Thomas, Director General, and Walter Kolkma, Director, Division 1, IED. IED retains full responsibility for the report.

Basic Data Indonesia: Participatory Irrigation Sector Project (Project Number 32359, Loans 2064 and 2065, Grant 4299)

Key Project Data

At Appraisal ($ million)

Actual ($ million)

Total project cost Foreign exchange cost Local currency cost

ADB loan amount Netherland’s grant

126.0 30.6 95.4

73.0 15.0

127.5 29.1 98.4

74.3 15.0

Key Dates

Fact-finding mission Loan negotiations Board approval Loan signing Loan effectiveness First disbursement Loan closing

11–19 Sep 2001 12 Nov 2003 19 Dec 2003

2 Feb 2005 2 Jun 2005

19 Mar 2007 31 Dec 2012

Borrower Republic of Indonesia Executing Agencies Ministry of Public Works and Housing Ministry of Home Affairs Ministry of Agriculture Mission Data

Type of Mission No. of Missions No. of Person-Days Fact-finding 1 36 Appraisal 1 50 Inception 1 38 Loan review 11 320 Project completion review 1 48 Independent evaluation 2 40

Executive Summary

The Participatory Irrigation Sector Project (PISP) in Indonesia was approved on 19 December 2003 with an estimated cost of $126.0 million equivalent. The project was financed by two loans to a value of $73.0 million from the Asian Development Bank (ADB) and a $15.0 million grant from the Government of the Netherlands. Counterpart funds were made available by the Government of Indonesia and by subnational governments (provinces and districts), while farmers provided in-kind contributions through the water users associations. Expected impacts were increased rural income and reduced poverty in the targeted irrigation schemes in 25 districts in six provinces (Lampung, Banten, West Java, Central Java, East Java, and South Sulawesi). The project’s more immediate outcomes were expected to be sustainable decentralized management of irrigation systems and higher crop yields.

The PISP was implemented mostly in line with its original design. Executing

agencies were the Ministry of Public Works and Housing, the Ministry of Home Affairs, and the Ministry of Agriculture. Implementation was coordinated among these three, the provincial and district governments, and the National Development Planning Agency. The PISP’s cost at completion was $127.5 million, compared to its original estimated cost of $126.0 million. However, the project took 8.5 years to implement rather than the scheduled 6.0 years because of prolonged revisions to the national legal framework for irrigation, the government’s decentralization process, and delays in the release of counterpart funds. The delay and price inflation resulted in a revision of the scope of the project in 2009.

The project had four outputs: (i) upgraded district capacity to manage irrigation responsibilities, (ii) improved irrigation performance at scheme level, (iii) improved extension services and farmer access to agricultural inputs and services, and (iv) an improved system for water resource management information and asset management.

Project Performance Assessment

This evaluation rates the project successful, a rating also provided by the project completion report (PCR) and PCR Validation Report.

The project is rated relevant as the project goals and approaches were consistent with strategies of the government and ADB in Indonesia during and after the project period. The project’s institutional arrangements were appropriate as the PISP aligned agency responsibilities with their respective budget allocations. This improved the capacities of local agencies and beneficiaries. The project’s focus on irrigation reforms and institution building was timely, and its modality as a sector project was appropriate. The relatively narrow scope of activities, however, posed challenges since improved irrigation alone (with limited extension support) was not sufficient to produce consistent yield increases in many locations. There was a need to better match project activities with other projects providing non-irrigation support. The fairly limited choice of activities and budget also constrained PISP support to the bottom-up

xii Participatory Irrigation Sector Project

planning exercise that was being promoted as part of the decentralization process. Other design issues that provided lessons for future projects include the need for better capacity building indicators and financing arrangements.

The PISP achieved most of its output goals in irrigation capacity building at the subnational level, particularly in integrating tasks to better record assets and damages, prepare a more comprehensive medium-term plan, change local regulations, and establish irrigation commissions. The project provided training to 8,311 provincial and district staff, exceeding the target of 5,000. Similarly, the revised work scope for rehabilitating infrastructure and strengthening water users associations (WUAs) and water users association federations (WUAFs) was completed. The main shortfall was in the provision of agricultural extension. Although extension sessions were largely delivered, there was no evidence of changes in farming practice due to the supposedly enhanced extension services. Although there was progress made in achieving outcome 1 (sustainable decentralized irrigation management), a key outcome indicator—adequate progress and avoidance of deferred maintenance—has yet to be achieved at the district-managed irrigation schemes (60% of the total project area). Outcome 2 (increased irrigated crop yield) was considered achieved as rice production from the fixed-sized command areas was estimated to have increased by at least 17% (against a target of 14%). Based on the overall progress in outcome and output achievements, the PISP is rated effective.

The project was highly efficient in its use of resources, with an estimated

economic internal rate of return (EIRR) of 24%. Benefits arose mainly from more intensive use of the irrigation command areas through reduction in fallow or idle plots during the drier crop growing periods (April to October). These positive effects were most pronounced in the schemes with lower water availability. There was a modest benefit gain from yield increase—mainly on rice—on the irrigated and planted areas. Modest rehabilitation costs per hectare also contributed to the robust EIRR estimates. However, project implementation proceeded less than efficiently because of the challenges discussed. On balance, the project is rated efficient.

The project provided important support and impetus for institutional change in the irrigation subsector, but a continuing effort is needed to complete the PISP reform agenda. Some institutional reform actions were partly sustained after the project was completed, such as enhancements in asset inventory data. However, operationalization of the irrigation commissions was not sustained in all districts, and the improved irrigation plan approach/format introduced under the PISP was not adopted in any of the 27 districts. With regard to the sustainability of funding, the budgets provided are still insufficient to adequately maintain the rehabilitated irrigation structures. The project is therefore rated less than likely sustainable.

The project made a contribution to improving incomes and reducing poverty.

Household incomes in the project area were estimated to have increased by 25% in real terms at project closing, exceeding the targeted 10%–15% income increase. The poverty rate in the project areas fell from 24.4% in 2008 to 6.8% at project closing, exceeding the target of a one-third reduction. The impact assessment will benefit from the ongoing additional analysis of household survey data gathered in 2015, whose results will be reported separately.

Executive Summary xiii

Issues and Lessons

There has been increasing spending on O&M costs incurred by the national and provincially managed irrigation schemes. The main unresolved challenge under the PISP was inadequate commitment of districts to fund irrigation O&M (and capital investment). This affects 60% of the PISP area. As decentralization reform progresses, districts have more discretion to determine how their budgets are spent. Various options therefore need to be explored to give local governments incentives to allocate more of their budget to irrigation O&M (e.g., requiring them to match funds provided by the central government). There is also a need to monitor fund transfers from the national level for irrigation to make sure they are used as agreed.

The evaluation of PISP design suggests there was a dilemma in its original scope. PISP activities were narrowly focused on irrigation improvement, which helped simplify project implementation. However, the relatively narrow set of interventions did not consistently generate adequate benefits (yield and income increases) across the PISP locations. Project interventions needed to be better matched with deliveries of other projects or programs providing non-irrigation supports. The project experience highlights the continuing challenge for ADB to design projects that are both simple and implementable but with adequate scope of interventions to achieve the desired outcomes and impacts. Furthermore, the PISP experience suggests that a narrow and less flexible scope is not optimal for projects supporting decentralization, since decentralization requires more flexible open-menu supports that encourage bottom-up participation and decision making. Future project designs have to consider these needs.

The project arranged agency training modules well, as they aimed to strengthen staff skills in key functions (irrigation planning, participatory consultation, and construction management) and in-class seminars were followed up and strengthened by hands-on training. There were nonetheless areas that can be improved in capacity building: (i) quizzes and short tests should have been administered as part of project training to help assess the extent of knowledge transfer; (ii) sector and project indicators could have been sharpened by including more service quality indicators such as the level of water loss in irrigation systems, the extent of irrigation service coverage, and the rate of adoption of new farming practices/technologies introduced under the extension program; and (iii) progress on key irrigation/agriculture indicators should have been monitored by independent parties such as the local planning or finance agency, or an entity that reports to the local parliament.

A clear mechanism needs to be developed on how voluntary land contributions are agreed upon and compensated, if necessary, among beneficiary groups (WUAs).

The project impact assessment and household surveys pointed to emerging trends that follow-on projects need to recognize or address: growth of contract farming and emergence of more commercial farming, consolidation of small land plots, the more dynamic labor movement in the rural areas, and migration of youth away from agriculture. There are also challenges to irrigated agriculture in the conversion of fertile farm lands into nonagricultural uses and the increasing competition for water. Sector stakeholders need to continue their dialogue on how to balance the competing goals of national rice self-sufficiency (keeping rice imports to a minimum) and improving household incomes and nutrition status through crop diversification.

xiv Participatory Irrigation Sector Project

Follow-up Actions

ADB needs to continue its support to ensure that the PISP districts follow through with the irrigation reforms partially carried out under the project. Key to these reforms are the updating and adoption of the medium-term plans for irrigation. The roles and activities of irrigation commissions set up under PISP have to be integrated into the district budgeting cycle. Agency staff training and strengthening of the legal basis for continuing irrigation reforms have to be continued. Since institution building is a long-term undertaking, irrigation capacity building efforts planned in the medium-term in Indonesia should cover the PISP districts. Leaving the institution work only partially developed will waste the momentum of and investment already spent under the PISP.

To help overcome budget constraints within the districts, cost-effective construction solutions should be pursued such as engagement of WUAFs as contractors for civil works, since these were proven to be low-cost and of satisfactory quality. A review is needed to clear legal impediments (if any) to the engagement of WUAFs. Citizen/farmer groups have to be more actively involved to monitor local budget allocation and procurement processes. This monitoring can be supported by future projects.

ADB should support initiatives by the central government (Ministry of Finance and Ministry of Public Works to better monitor the utilization of the special irrigation budget transfers from the central level to the districts. A mechanism is needed to bind districts to agreed funding plans for irrigation.

CHAPTER 1

Introduction 1. The Participatory Irrigation Sector Project (PISP) was implemented in Indonesia from December 2003 to December 2012.1 The envisaged project outcomes were (i) development of a sustainable decentralized management of irrigation systems, and (ii) increased yields of irrigated crops. These outcomes were expected to contribute to two broader impacts: increased economic growth and reduced poverty. 2. The project was to cover 25 project districts2 in the provinces of Banten, Central Java, East Java, Lampung, South Sulawesi, and West Java.3 Activities under the project were organized into three components. Part A was on irrigation management comprising capacity building of district and provincial administrations, capacity building of farmers and water users associations (WUAs), rehabilitation of irrigation infrastructure, and support of agricultural (extension services and market empowerment). Part B improved water resources information and asset management, while Part C supported project coordination and management. The appraised project cost was $126.0 million. The Asian Development Bank (ADB) was to provide $73.0 million in loans, while the Government of the Netherlands provided a $15.0 million grant. The remaining costs were funded by the national and local governments and beneficiaries.4

A. Evaluation Purpose

3. Every year, as a part of its routine work program, the Independent Evaluation Department (IED) selects some completed ADB projects to be post-evaluated. The PISP was selected for two reasons. First, the evaluation unit of the project cofinancier, the Policy and Operations Evaluation Department (IOB) of the Ministry of Foreign Affairs of the Government of the Netherlands, proposed to conduct a joint impact evaluation of the PISP to provide inputs into IOB’s review of the Netherlands’ support overseas for sustainable water management. IED was interested in this joint evaluation, as impact evaluation helps enrich IED evaluation outputs. IOB will prepare a separate document to report the findings of the impact assessment. 4. Second, the evaluation of the PISP at this stage would help ADB’s preparation of a follow-on project, tentatively titled the Integrated Participatory Development and Management of Irrigation Project. The evaluation gives special attention to impacts on rural households and the project’s contribution to the wider national effort to devolve

1 Asian Development Bank (ADB). 2003. Report and Recommendation of the President to the Board of

Directors: Proposed Loans to the Republic of Indonesia for the Participatory Irrigation Sector Project. Manila.

2 District is known as kabupaten in Indonesia. 3 The project covered 27 districts at closing. The additional districts were due to a split in two of the

targeted districts. A new district, Pesawaran, split in 2006 from the South Lampung district in Lampung province, and North Toraja district split in 2009 from Tanah Toraja district in South Sulawesi province.

4 The tables in Basic Data provide comparisons between actual and planned investments. The grant from the Government of the Netherlands was administered by ADB.

2 Participatory Irrigation Sector Project

the national government’s mandates and resources to provincial and district governments. The follow-on project is expected to spur more robust ADB engagement in Indonesia’s irrigation and agriculture sector.

B. Process

5. The evaluation approach follows the Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations.5 The performance assessment is based on four criteria: the project’s (i) relevance, (ii) effectiveness in achieving outcomes and outputs, (iii) economic efficiency, and (iv) sustainability. An evaluation framework with questions on these criteria is presented in detail in Appendix 1. Other aspects assessed for the project are (i) the likely contribution to the stated impacts, i.e., income improvement and poverty reduction; and (ii) the performance of the government, executing agencies and ADB. Materials and data for the evaluation were sourced from existing documents and available databases, as well as field-based observation, interviews, and primary data collection. 6. Existing reports and data studied encompassed project design reports and reports on project implementation, including the project completion report (PCR) prepared by the government and project consultants.6 ADB completed a PCR in June 2014, which rated the project successful (relevant, effective, efficient, and likely sustainable according to the criteria in para. 5).7 IED subsequently validated this in January 2015,8 and also found the project successful (relevant, effective, and efficient), although the validation found the project less than likely sustainable. This project performance evaluation reviewed materials that were not accessed during the validation such as local government completion reports and a PCR household survey. 7. Primary data gathering was carried out during April to June 2015, comprising interviews with relevant agency staff and farmers’ representatives, spot checks of structures, and a survey of farm households. The coverage of primary data by district is shown in Appendix 2. Field interviews were carried out with agency staff in eleven districts and five provinces, and spot checks were done on completed project structures. For the impact assessment conducted for IOB, 900 households in 60 irrigation schemes in 15 districts were surveyed.9 A national community development foundation, Equator, facilitated the household survey. 8. This project performance evaluation report benefited from IOB inputs particularly in the assessment of project effectiveness in achieving its outcomes and its economic efficiency. Since this project performance evaluation report has been prepared ahead of the finalization of IOB’s study on the project’s impact, more findings on the PISP impact assessment will be reported in a separate document.

5 ADB. 2013. Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. Manila. 6 ADB. 2014. Completion Report: Participatory Irrigation Sector Project in Indonesia. Manila. 7 footnote 6 8 ADB. 2015. Validation Report: Participatory Irrigation Sector Project in Indonesia. Manila. 9 The survey covered eight PISP districts (Madiun, Banyumas, Cilacap, Kuningan, Lampung Timur, Lebak,

Sinjai, and Cirebon). The others were control districts and districts under the Water Resources and Irrigation Sector Management (WISM) Program (Phase 1) of the World Bank.

CHAPTER 2

Design and Implementation

A. Formulation

9. The project was designed in the early 2000s when the Government of Indonesia was planning the decentralization of its authority and responsibilities. It was anticipated that PISP would be implemented in an uncertain policy and institutional environment. The project was also designed to address the long-standing problem of inadequate routine operation and maintenance (O&M) of irrigation facilities. This problem emanates from a culture in which maintenance is routinely deferred, which necessitates more expensive periodic rehabilitation (footnote 1). 10. A feasibility study for the PISP was carried out in 2001, funded by an ADB technical assistant grant.10 A loan appraisal mission was subsequently fielded in November 2002. Because of the difficulties in assessing the impact of policy directions in decentralization and the ongoing review by the government of the Water Law No. 11 of 1974, the project’s preparation was slowed down. Interdepartmental comments during the PISP’s design focused on a need to simplify the project scope in view of past lessons learned and potential challenges from the upcoming government-wide decentralization process. The project design team responded by excluding a water resource management component that the feasibility study had proposed. The project scope was harmonized with other projects running parallel with the PISP, including the World Bank’s Water Resources and Irrigation Sector Management (WISM) Program, which was approved in June 2003 and supported improved water resource management.11 The design team was knowledgeable about Indonesia’s water sector issues, lessons from past irrigation projects, and activities of development partners. The project design team incorporated interdepartmental suggestions to improve project design (e.g., improving output indicators, enhancing the mechanism for district selection, and augmenting training content to support decentralization). ADB’s loans for the PISP were approved in December 2003, and the Government of the Netherlands approved its grant in December 2003.

B. Rationale

11. The sector opportunities and needs analyzed at appraisal focused mainly on the promising, albeit slow, progress that had been made in irrigation reforms since the late 1980s.12 The reforms had aimed to make irrigation services in Indonesia more efficient and sustainable, and have mostly involved the transfer of more management responsibilities to farmers, promotion of direct cost recovery for irrigation, and

10 ADB. 2001. Technical Assistance to the Republic of Indonesia for the Participatory Irrigation Sector Project.

Manila. 11 World Bank. 2003. Project Appraisal Document on a Proposed Loan in the Amount of US$45.0 million and

a Proposed Credit in the Amount of SDR17.9 million to the Republic of Indonesia for the Water Resources & Irrigation Sector Management Program.

12 see footnote 1


enhanced funding of irrigation O&M.13 In December 2001, an ambitious government regulation on irrigation (No. 77/2001) was issued, which envisaged the primary responsibility for irrigation O&M and periodic rehabilitation to be gradually transferred to WUAs. The PISP was largely premised on the opportunities available in the early 2000s to advance irrigation reforms as well as to support the sector’s adjustment to the upcoming government decentralization. 12. Outcomes and impacts. The project sought two outcomes: (i) sustainable decentralized management of irrigation systems, and (ii) increased yields of irrigated crops. Targets were to (i) provide 456,000 hectares (ha) with an adequate level of O&M by 2012, and (ii) increase the average paddy yield from 3.5 metric tons per ha in 2002 to 4.0 tons per ha (a 14.3% increase) by 2012. At appraisal, the original project area was 625,000 ha, but this coverage was reduced to 456,000 ha (73% of the original target) because of a 2.5-year delay in project implementation.14 The objective tree in Figure 1 and design and monitoring framework in Appendix 3 show how PISP outcomes were to be supported by outputs and activities, which are further discussed in paras. 22–33. The envisaged PISP impacts were (i) improved income, and (ii) reduced rural poverty. Average household income in the project area was expected to increase by 10%–15%, and the poverty rate was expected to decrease by one-third. 13. Indonesia’s agriculture sector was responsible for 15.2% of its total gross domestic product in 2003, decreasing to 13.7% in 2014.15 The share of the rural population, i.e., the share of population largely relying on farming-related activities, declined from 56% in 2003 to 47% in 2014.16 Nonetheless, robust agriculture growth is still very important for employment generation and poverty reduction nationwide. The contribution of rural areas to the national rate of deprivation declined from 59% in 2010 to 57% in 2014, while the contribution of the urban areas grew from 41% in 2010 to 43% in 2014. The government’s priority continues to be to rehabilitate and build new rural infrastructure, as the shortage of infrastructure constrains job creation and income growth in rural areas. 14. Analysis of the results chain. Figure 1 presents an overlay of the PISP results chain against the broader results framework for irrigated agriculture. It shows that PISP activities focused to a large extent on improving irrigation (water delivery) while providing relatively little or no support in other areas. This focus helped simplify project implementation. However, PISP activities were not sufficiently complete, at least in some locations, to achieve its envisaged outcome and impact to increase crop yields and incomes. Examples of excluded interventions were improvements in access to farm inputs (e.g., high-yielding varieties, fertilizers, farm equipment, and credit); and enhancement of transport infrastructure. There was no substantive analysis in the project design as to how these complementary interventions were to be provided in selected project areas, e.g., by parallel projects and programs being implemented at the same time as PISP. This issue is further discussed in the project relevance assessment (paras. 36–37).

13 The broad reform framework was outlined in an irrigation management reform program issued as a

national decree in 1999. 14 Despite the intent to support decentralization, the PISP outcome indicators did not include any indicators

that track the decentralization progress such as the proportion of irrigation budget or the types of decisions devolved to subnational governments.

15 World Bank. Agriculture, value added (% of GDP). http://data.worldbank.org/indicator/NV.AGR.TOTL.ZS 16 World Bank. Rural population (% of total population). http://data.worldbank.org/indicator/SP.RUR.TOTL.ZS

http://data.worldbank.org/indicator/NV.AGR.TOTL.ZS

Design and Implementation 5

Figure 1: Results Chains: Irrigation Subsector Compared to the Participatory Irrigation Sector Project

WUAFs = water users association federations Source: Independent Evaluation Department

C. Cost, Financing, and Implementation Arrangements

15. Project cost and financing. The actual project cost was at $127.5 million, compared to the appraisal estimate of $126.0 million. ADB provided two loans, one for SDR13.3 million ($19.0 million equivalent) from its concessional Asian Development Fund and the other for $54.0 million from its ordinary capital resources. The Government of the Netherlands grant for the equivalent of $15.0 million cofinanced all expenditure categories. An amount of $25.5 million was to be provided as counterpart funds by the central and subnational governments, while communities and farmers were to provide an in-kind contribution for incremental O&M efforts estimated at the equivalent of $12.5 million. At closing, the Asian Development Fund loan was valued $1.3 million higher than at appraisal because of the appreciation of the SDR against the United States dollar. Government counterpart funds were increased from $25.5 million to $31.9 million to compensate for higher costs as a result of project implementation delays (para. 20). The final beneficiary contribution was estimated at $6.2 million, half of the original amount, though it is difficult to value it as it was provided in-kind. Project spending by component is presented in Table 1.

Complementary interventions excluded from PISP

PISP interventions

LEGEND:

Increased income and reduced poverty (IMPACTS)

Higher yields of irrigated crops (OUTCOME 2). Improved On-Farm Prices (Outputs & Inputs).

Decentralized and Sustainable

Irrigation Services (OUTCOME 1)

Capacity building of District Water Resource and

Planning Agencies (OUTPUT 1)

Capacity building of

WUAFs

(OUTPUT 2)

Repair of irrigation structures

(OUTPUT 2)

Improved water resource and asset

management

(OUTPUT 4)

Improved know-how/access to

technology

Capacity building of District Agriculture

Agencies (OUTPUT 1)

Extension sessions provided; Extension

demonstration plots set up (OUTPUT 3)

Farmer organizations/ cooperatives Strengthened

Completed studies and training for

WUAFs (OUTPUT 3)

Access to credit/other inputs (seeds, fertilizers, equipment, labor)

Improved transport links

(on-farm and farm-to-market)

Increased irrigated crop profits Improved distribution of farm income/land tenure


Table 1: Project Costs by Component ($ million)

Component Appraisal Estimate

Actual

A. Irrigation Management 1. Capacity building of districts and provinces 8.0 8.9 2. Capacity building of water users associations 12.4 14.1 3. Sustainable operations of irrigation infrastructure 60.0 71.6 4. Irrigated agriculture support 6.8 12.9

B. Water Resources Information and Asset Management

2.0 3.4

C. Project Coordination and Monitoring 7.5 12.9 Subtotal A+B+C 96.6 123.7 D. Contingencies 23.4 0.0 E. Interest During Construction and Other Charges 6.0 3.8 Total 126.0 127.5

Source: ADB. 2014. Completion Report: Participatory Irrigation Sector Project. Manila.

16. Matching financing not pursued. To support decentralization, the PISP originally intended to institute a cofinancing requirement in irrigation rehabilitation funding between the national and subnational governments. The matching in financing from both the local and national budgets was intended to assure local governments’ commitment to the project. This requirement was abandoned in 2008 following issuance of the National Regulation No. 7 of 2008 on Financial Management and Responsibility in the Implementation of Deconcentration and Co-administration. This was due to the limited budgetary resources available at the local level and other political developments. As a result most of the project expenses after 2008 were charged to the national budget. Another adjustment made in the financing arrangement was in the project cost sharing between the external financiers (ADB and the Netherlands) and the government. Originally civil works under PISP was to be 83% financed by the external financiers and 17% by the government. To expedite project implementation, beginning in 2009, civil works contracts were separated into those that were 100% financed by external loans and grant and those that were 100% financed by the national government. These two changes in financing arrangement allowed the project to expedite construction and cope with the difficult realities surrounding the project.17 Nonetheless, the evaluation sees these changes as a cause for concern. ADB and the Government of the Netherlands as collaborating financiers of the PISP became less assured of the extent of the counterpart funds provided to the project. The change also prevented the external partners from involvement in the procurement and monitoring of project contracts fully funded by the government.18 17. Another issue recognized in the PCR and confirmed by the evaluation was that at appraisal all project expenditure categories were funded by the three funding sources (the government, ADB loans, and the grant from the Government of the Netherlands). This financing arrangement is rather unusual and added an extra burden to the PISP’s administration. Most contracts had to undergo three withdrawal processes and there was no centralized computer system in place for ADB to facilitate disbursement of the grant from the Government of the Netherlands (or other grants, for that matter). It would have been better if the PISP grant financing had been

17 Another adjustment pursued early in the project implementation was to enlarge imprest account

replenishment amounts. This reduced the account replenishment frequencies and expedited loan and grant disbursements.

18 Part of the counterpart contribution provided by districts and provinces was in the form of staff and administrative support. This support was fully accounted for in the final costing of the PISP.


targeted to support fewer and more specific activities, such as those critical for institution building and not immediately generating returns on investment (e.g., sector studies and upgrading of water resource management). 18. Implementation arrangements. The project had three executing agencies: (i) the Ministry of Public Works and Housing (previously the Ministry of Settlement and Regional Infrastructure) through the Directorate General of Water Resources (DGWR), (ii) the Ministry of Home Affairs through the Directorate General of Regional Development, and (iii) the Ministry of Agriculture through the Directorate General of Agricultural Infrastructure and Facilities. The implementation responsibilities were shared between the three national ministries and the provincial and district governments, primarily through their local planning, water resource/public works, and agriculture agencies. The implementation arrangements did not change during the project period. 19. A national steering committee was formed at the National Development Planning Agency (Badan Perencanaan Pembangunan National or BAPPENAS) to provide overall policy guidance. Each executing agency had a national project management unit, and each province and district had a project management office. 20. Scheduling. Project implementation faced substantial challenges mainly from 2003 to 2008. Implementation was slowed down to await the issuance of new Water Law No. 7 of 2004 and the national regulation on irrigation management (National Regulation No. 20 of 2006), which clarified the legal basis for reforms promoted under the PISP. Following the national election, there was also a change in government in October 2004, and there were broad changes in institutions and government procedures due to the government decentralization during 2003–2008. The PISP loans became effective only in June 2005, about 12–15 months behind schedule. Project activities began in earnest only in 2006, and progress was still slowed down during 2007–2008 by late releases of counterpart funds at the national level. Price inflation due to the delay led to a need to downscale the project scope. In 2009, the targeted number of irrigation systems and WUAs to be supported under the PISP were reduced (para. 30). The project loan’s closing was in December 2012, about 18 months behind the original schedule. 21. The evaluation review of PISP compliance with its loan covenants did not find additional observations beyond those reported in the ADB PCR (footnote 6) and government PCR.19 All 19 covenants were complied with. The main issue was the late releases of counterpart funds.

D. Outputs

22. The project envisaged four outputs. Output 1 was upgraded district capacity to manage devolved irrigation responsibilities; Output 2 was improved irrigation performance at scheme level; Output 3 comprised rationalized district agricultural services (Output 3a), and strengthened WUAs and improved farmer access to agricultural services/inputs (Output 3b); and Output 4 was an improved water resource management information and asset management system. Achievement of each output was defined by a number of targets described in paras 23–33. Figure 1 describes how these outputs supported higher-level PISP results. The four outputs are matched with

19 Government of Indonesia. 2013. National Project Completion Report: Participatory Irrigation Sector Project

2006–2013. Jakarta.


the project components as follows: component A (irrigation management) was to generate outputs 1, 2, and 3; component B (water resources information and asset management) supported output 4; and component C (project coordination and monitoring) was not intended to generate any direct output. This section describes the outputs and also reports on the PISP’s achievement of its intended outputs. These contribute to the assessment of the PISP’s performance (relevance and effectiveness) in Chapter 3. 23. Output 1: upgraded district capacity. This output was to be represented by three indicators for which targets were set: (i) effective functioning of the district governments as focal points for irrigation management, (ii) 5,000 provincial and district staff trained in irrigation planning and management by 2007, and (iii) preparation of a district irrigation development and management plan (Rencana Pengembangan dan Pengelolaan Irrigasi or RP2I) by 2005. By project’s end, 8,311 district and provincial staff were provided training in sector management (irrigation planning, design, construction management, O&M arrangements), water management, and agriculture from planning, water resources, and agricultural agencies. The topics and modules were developed based on consultations with the targeted agencies and training needs assessments. 24. The evaluation interviews with relevant agencies suggest that seminar training subjects were appropriate since these pertained to the general (rather than project-specific) tasks of targeted district agencies, such as strategic planning, irrigation construction and extension management, and administrative functions. Topics and materials were developed after a training needs assessment process. The in-class seminars were followed up by hands-on work in collaboration with consultants to implement project activities (medium-term irrigation planning, rehabilitation civil works, and agricultural extension). The evaluation found that there was a lack of quality control (e.g, certifications or quizzes) during the seminar sessions to help measure knowledge transfer. Appendix 4 provides an in-depth institutional assessment for the PISP. 25. All 27 districts worked to formulate an RP2I (para. 23), a centerpiece document in irrigation management which requires each district to (i) update the inventory of existing irrigation assets; (ii) consider other district medium-term plans such as those on land use and water allocation; and (iii) estimate in a more credible manner the irrigation investment and O&M requirements (Figure 2). The RP2Is were intended to be rolling 5-year plans to be updated yearly or periodically. At PISP completion, 26 districts had completed their RP2Is. The evaluation’s review of selected RP2Is suggested that these were an adequate initial document to facilitate enhanced irrigation investment budgeting. However, the RP2Is had not yet comprehensively covered all irrigation schemes in the district; 18 districts had incorporated planned investment for new irrigated areas (as opposed to only discussing rehabilitation needs for existing structures). Due to time constraints, RP2Is were prepared at the same time as the construction was carried out rather than before construction, as originally intended. This theoretically hampered the selection of the sites for the works.


Figure 2: Role of District Irrigation Development and Management Plans, Irrigation Commission, and Irrigation-Related Data in Influencing Irrigation Budget Allocation

O&M = operation and maintenance, RP2I = Rencana Pengembangan dan Pengelolaan Irigasi (district irrigation development and management plan). Source: Independent Evaluation Department

26. The project successfully supported the districts and provinces to enact regulations necessary to implement National Regulation No. 20 of 2006 (irrigation management). These included the (i) prevention of conversion of fertile irrigated lands into other uses, and (ii) establishment and organization of 33 irrigation commissions in the districts and provinces. Overall, output 1 (upgraded district capacities to manage irrigation) was largely achieved, particularly in helping to shift the focal point of irrigation management to the districts. 27. Output 2: Improved irrigation performance. This output which commanded the most funding resources comprised two areas: (i) WUA capacity building, and (ii) rehabilitation of irrigation infrastructure. The WUA institution building was gauged by four indicators and their targets: (i) 5,000 WUAs established or strengthened;20 (ii) WUAs within a scheme or in close proximity organized into WUA federations (WUAFs); (iii) the responsibility for O&M for primary and secondary systems assumed by the government, and the tertiary systems assumed by WUAs by 2012; and (iv) O&M in about 456,000 ha of the project area to be jointly managed by the government and WUAs as agreed in the participatory irrigation management agreements. 28. By its completion, the PISP had revitalized and strengthened 5,076 WUAs, which were organized into 643 WUAFs, meeting targets (i) and (ii) in para. 27.21

20 The original target of 6,250 WUAs was lowered, as the irrigation area was reduced at the midterm review. 21 The strengthening of the WUAs involved hands-on training in planning, construction works, and

agricultural extension dissemination. WUA capacity building was initially carried out by the district development planning agencies, but due to a change in assignment (per National Regulation No. 38 of 2007), this task was transferred to the Ministry of Agriculture and local agriculture agencies in 2008. Monitoring was conducted and efforts were made to ensure WUA women members received training

District/provincial medium-term

development plan (5-year horizon, static)

Annual government

work plan

Improved irrigation

work plan

Irrigation Development and Management Plan or

RP2I

(5-year horizon, rolling)

Irrigation commission deliberations

Data: Irrigation inventory Water resource

assessment Land/spatial plan Other assessments

Implementation of irrigation investment and O&M

Annual revenue and expenditure plan

Irrigation budget

Parliamentary review


Assignment of the irrigation facility responsibilities was achieved early on in the project period through issuance of National Regulation No. 20 of 2006 (para. 20), which clarified that WUAs (farmers) are responsible for the tertiary canal systems, the national and subnational governments are responsible for the primary and secondary canal systems and structures. This division of tasks was considered an improvement over the provision of National Regulation No. 77 of 2001, which aimed for WUAs to take the responsibility for all irrigation O&M and periodic rehabilitation, with the central and local government only providing assistance.22 The division of responsibilities among governments for irrigation management is demarcated into three levels: (i) the district governments are responsible for irrigation schemes (i.e., primary and secondary canals and structures) with command areas smaller than 1,000 ha, (ii) the provincial governments are responsible for irrigation schemes that range from 1,000 ha to 3,000 ha in size or that cross district boundaries, and (iii) the national government is responsible for schemes that are larger than 3,000 ha or that cross provincial boundaries. With this clarification of responsibilities, indicator (iii) target (para. 27) was met in principle, even though the actual ability of parties to carry out their responsibilities varied, as discussed in the project’s outcome assessment (paras. 43–46). 29. The relationship between the WUAs and the local agencies was critical in empowering the WUAs. With the demarcated assignments over the canal systems, the system of collaborative management between farmers and governments differed from that envisaged under the participatory irrigation management agreements.23 For the tertiary systems, cash assistance—under the government’s social assistance programs for the poor—has been given to the WUAs for irrigation O&M and repairs. The programs have been implemented by different national and local agencies and entail different levels of obligation on the farmers. For the primary and secondary structures, with the demarcated responsibilities there is no obligation on the part of WUAs to contribute to O&M and repairs. One significant contribution of the PISP was the capacity building and promotion of WUAs and WUAFs to be contractors for civil works in the primary and secondary systems. The WUAs and WUAFs became direct contractors for light rehabilitation works known as sustainable replacement of rehabilitation. They were also engaged in more complex rehabilitation as subcontractors to the main commercial contractors. The WUAFs reported that these contracting opportunities were important, as they raised WUAFs’ stature and financial standing, effectively empowering them.24 30. A critical part of output 2 was the aim to rehabilitate irrigation infrastructure to benefit 456,000 ha, which was revised downward from the original target of 625,000 ha (para. 12). The project completed rehabilitation of 1,614 irrigation schemes covering 457,220 ha, slightly above the revised target; sustainable replacement of rehabilitation works (light rehabilitation) covered 314,777 ha, while major rehabilitation works benefited 142,433 ha. Targeted irrigation schemes—also known as subprojects—were in the districts selected according to their readiness for irrigation reforms and level of

(19% in institutional strengthening and 7%–16% in farm practice and agricultural marketing) (PCR, Appendix 11).

22 ADB. 2003. Report and Recommendation of the President to the Board of Directors: Proposed Loans to the Republic of Indonesia for the Participatory Irrigation Sector Project. Manila. (para. 13)

23 It was originally envisioned that through the participatory irrigation management agreements the WUAs would cofinance a wider range of irrigation O&M and repair works, the extent of which depends on the WUAs’ capabilities.

24 Aside from the arrangements on irrigation O&M and repairs, governments and WUAs/farmers continue the arrangement on water allocation (e.g., operating of water gates), which is determined by agreed cropping plans/schedules.


poverty.25 Schemes were selected from proposals of the districts and existing WUAs, and narrowed down based on socioeconomic and technical profiling and feasibility assessments, including cost-benefit assessments.26 Field visits to irrigation structures by the evaluation team suggested that the work done was of satisfactory quality (Appendix 4). The project helped farmers rehabilitate their tertiary canals covering 86,416 ha, with linkage to the extension program (output 3). The project provided materials and the WUAs provided labor. Although, there were some shortcomings, output 2 was considered largely achieved in building WUA capacities and rehabilitating the irrigation infrastructure. The scope of this output was smaller than planned at appraisal, but the shortfalls were mainly related to the ambitious decentralization process, which led to delays but which was exogenous to the project scope. 31. Output 3a: rationalized district agricultural services. These services were measured by key performance targets set for the extension services: (i) adoption of improved irrigation agronomy techniques by members of 5,000 WUAs, and (ii) adoption of improved technology packages developed under other projects by farmers belonging to the 5,000 WUAs. Adoption of technology was not tracked when the project was implemented; instead, only input activity data were recorded, e.g., the establishment of 112 agricultural training and demonstration plots were ranging in size from 4 ha to 135 ha. Further, 511 sessions were provided for 11,800 farmers covering three main modules: (i) farm water management, (ii) crop intensification mainly for adoption of the System of Rice Intensification technique, and (iii) crop rotation and diversification.27 Despite the intent to promote crop diversification, interviews with the agriculture agencies suggest that most of the agronomy and water management extension support was aimed at rice cultivation. Given the lack of data on the progress in the actual adoption of improved technologies, output 3a is considered partly achieved. 32. Output 3b: water users association federations’ market role strengthened. This output was intended to support WUAFs in enhancing farmers’ bargaining power primarily against input and service providers. This effort was intended to be a pilot intervention. A pilot study on the potential market role of WUAFs was envisaged and subsequently completed for five districts. The project supported workshops for 2,400 farmers to gain more information on market prices and opportunities and sources of credit. Another type of module on value-adding agro-processing activities benefited 2,950 participants, mostly women. The modest targets under output 3b were generally accomplished. However, the evaluation observed that beyond the aforementioned study and market extension, the PISP was further involved in supporting an ongoing government program, mainly by the DGWR, to develop WUAFs as full-fledged businesses with more diverse ventures (e.g., in seed and livestock production). In the view of this evaluation, the various efforts at venture development in the WUAFs

25 footnote 22. 26 Irrigation schemes (subprojects) were selected from a long list of schemes submitted by the districts and

WUAs covering 1,317,000 ha. 27 The System of Rice Intensification technique can raise paddy yields and conserve water, which makes it

particularly well suited for dry-season cultivation. Compared with normal rice cultivation, System of Rice Intensification involves planting single seedlings instead of multiple seedlings in a clump, and not flooding the paddy fields during the rice vegetative growth stage. Planting is in a square pattern with more space, rather than in rows or at random, and is followed by more intensive weed control. The rice field should be kept moist rather than continuously saturated with water in order to improve root growth and support the growth of aerobic soil organisms. The wider spacing of rice plants enables more vigorous growth of roots and canopy. Higher yield is achieved with 80%–90% reductions in seed requirements and 25%–50% less irrigation water, but the method can be more intensive in its demand for labor or mechanical inputs. Fertilizers and pesticides can be used.


distracted from the effort under the PISP to strengthen the WUAFs and WUAs. ADB, upon completion of the studies, should have clarified its view with the government on the optimal role of the WUAs and WUAFs, which should focus on efforts to better manage and sustain irrigation services. Other non-water-related income improvement efforts should be handled by cooperatives or other associations that already exist at the village level. Furthermore, support for enterprise development, in terms of business know-how and financing, is better provided by more appropriate partners such as social enterprises, venture capital organizations, or financial institutions, rather than technical line agencies such as the DGWR. 33. Output 4: improved water resource management and asset information system. This output focused on (i) institutionalization of the water resource data management unit at the DGWR, and (ii) strengthening of the irrigation asset management system at the district level. By June 2006, the Water Resources Data Unit was successfully established. Subsequently, consistent with the more appropriate approach to manage water resources in a river basin concept, in 2008 the capabilities of the central data unit were distributed to the regional river basin authorities. Large river basin authorities are still under the supervision of the national DGWR. To strengthen asset management in the districts and provinces–related to the RP2I preparation–the PISP supported the development and use of the irrigation asset management information system. The evaluation found the system more user-friendly and less costly to use than the older software. Inventory data form an important input to the medium-term irrigation plan as contained in the RP2Is.

CHAPTER 3

Performance Assessment

A. Overall Assessment

34. Table 2 summarizes the assessment of the PISP’s overall performance. The project is rated relevant, effective, efficient, and less than likely sustainable. The project goals were consistent with Indonesia’s agriculture sector priorities and ADB’s operational priorities in Indonesia. While the project design was generally sound and appropriate, there were some design issues that affected implementation as noted in Section B. The project largely accomplished its planned outputs and made enough progress towards its envisaged outcomes. The pace of implementation suffered, particularly in the first 3 years, because of the government-wide decentralization reforms. This led to a reduction in scope. The economic analysis (Section D and Appendix 5) nevertheless shows that the project produced sufficient economic returns on investment. Despite the institution building achieved, the goal to adequately cover irrigation O&M has not been fully achieved, particularly for 60% of the rehabilitated irrigated lands. There is a continuing concern over the sustainability of PISP benefits, and the project area would benefit from further institutional support. Overall the project is assessed as successful.

Table 2: Overall Performance Assessment

Criterion Assessment Rating (0–3)

Weight (%)

Weighted Rating

Relevance Relevant 2 25% 0.50

Effectiveness Effective 2 25% 0.50

Efficiency Efficient 2 25% 0.50

Sustainability Less than likely sustainable

1 25% 0.25

Total ratinga 1.75 a A rating at or above 1.75 is considered successful as per the 2016 PPER Guidelines. Source: Independent Evaluation Department.

B. Relevance

35. The assessment of the project’s relevance looks at the strategic alignment between the goals of the project, the country, and ADB, as well as the quality of the PISP’s design. The PISP supported implementation of the National Development Program, 2000–2004 (at appraisal) and the National Medium-Term Development Plan, 2000–2004 and 2010–2014 (at completion),28 which aimed to strengthen local institutions and community empowerment as well as accelerate poverty reduction and address regional income disparity. PISP objectives were consistent with ADB’s country

28 Law No. 25 of 2000 on the National Development Program, 2000–2004; and Presidential Regulation No. 5

of 2010 on the National Medium-Term Development Plan, 2010–2014.


strategies for Indonesia. Four strategies were prepared during the project period in 2001, 2003, 2006, and 2012. All strategy documents identified agriculture, natural resources, and rural development projects as integral parts of ADB’s support to Indonesia for poverty reduction and improved public sector management. The country strategy and program, 2003–2005, prepared during PISP appraisal, identified rural development (including rural infrastructure, marketing and support services, and microfinance) as a priority operation area.29 ADB’s country partnership strategy, 2012–2014 continues to focus on agriculture and natural resources along with five other sectors: education, energy, finance, transport, and water supply and sanitation.30 36. Analysis of results framework. The sector objective tree in Figure 1 suggests that PISP activities were appropriately chosen to support outcome 1 (sustainable irrigation management) by providing capacity building to all stakeholders (i.e., local and national governments and WUAs) as well as infrastructure repairs. To provide support for enhanced water resource management—an area not directly addressed under the PISP—project activities were matched with those under parallel projects, such as the World Bank’s WISMP. To achieve outcome 2 (increased crop yields), the PISP provided limited non-irrigation interventions in extension services. While the design scope, which emphasized irrigation capacity building and rehabilitation, kept the project focused, the scope should be better matched with complementary activities or appropriate area selection to give the project a better chance to raise crop yields and incomes. 37. There were activities critical to achieving outcome 2 that the PISP excluded from its scope to maintain the focus of the project (Figure 1). There was not enough analysis at the design stage on how parallel agriculture support (e.g., provision of improved varieties, fertilizer, equipment, financial intermediation, and transport access) was to complement the PISP in raising yields. Agricultural extension and WUAF enterprise capacity building received a relatively small portion of total PISP expenditures (12%), while the remaining activities were targeted at irrigation (Table 1). Analysis of household farm survey data gathered in 2015 found there were an uneven yield responses to the focused support on irrigation under the PISP in different locations (para. 48 and Appendix 5), suggesting that the project might have benefited from more agriculture support. 38. The PISP design contended that greater government decentralization and local participation would bring about a higher irrigation O&M commitment. However, the project experience suggested that with greater decision-making powers over budget allocation and the annual work program at the district level, local leaders increasingly pursued work priorities that differed from those already agreed under the PISP—even within the agriculture sector. Intergovernmental relations are complex and take time to adapt to decentralization. As such, it was not realistic to assume that challenges such as deferred irrigation maintenance would be overcome during the PISP’s implementation period. 39. Financing modalities. The PISP financing modalities were largely appropriate. The $15 million grant from the Government of the Netherlands was substantial and an attractive enhancement to the ADB loans. As discussed in para. 17, the arrangement for the grant to finance all project expense categories hampered disbursements, but this could have been resolved with an improved computer system. The project used the sector loan modality, where individual subprojects (irrigation schemes) were selected

29 ADB. 2002. Country Strategy and Program: Indonesia, 2003–2005. Manila. (para. 58 and Table 4). 30 ADB. 2012. Country Partnership Strategy: Indonesia, 2012–2014. Manila. (para. 24).

Performance Assessment 15

gradually as the project implementation progressed with different activities in each scheme. This modality was considered suitable given the decentralization-related uncertainties faced by the PISP. 40. Presentation of project risk. The project design team had difficulties in assessing the pace of institutional and regulatory reforms as well as financial capacity issues arising from decentralization. In retrospect, the report and recommendation of the President would have benefited from a more overt presentation of the decentralization risks (footnote 1). 41. There were notable features of the PISP design that were groundbreaking for Indonesia. The effort to strengthen irrigation planning through the use of rolling medium-term planning incorporating rigorous and updated field data, and an introduction of the local commissions on irrigation were necessary to build support within the central and local governments to raise funding commitments for irrigation. Similarly, the scheme to engage WUAs and/or WUAFs as civil work contractors and subcontractors helped empower them and improved the quality and cost-effectiveness of the irrigation civil works. In spite of the aforementioned issues (paras. 36–40), the PISP’s overall design was largely sound, and its strategic objectives and design quality are assessed as relevant.

C. Effectiveness

42. The project’s effectiveness is assessed based on the extent expected PISP outcomes and outputs were achieved, and how outputs supported the attainment of outcomes. The output assessment in Chapter 2 (paras. 23–33) considers three of the four envisaged outputs achieved: (i) upgraded capacities of districts and provinces, (ii) WUAs and WUAFs strengthened and empowered, and (iii) improved water resource and asset management system. The output on agricultural extension was partly accomplished. Two outcomes pursued were (i) sustained decentralized management of irrigation, and (ii) increased yields of irrigated crops. 43. Sustained decentralized irrigation management. This outcome was pursued with two related targets: (i) an adequate level of routine irrigation O&M, and (ii) the avoidance of deferred maintenance at project closing. Government data, stakeholder interviews, and site visits suggested that there was adequate O&M budget for irrigation schemes under the responsibility of the national government for the maintenance of primary and secondary canals (allocation of irrigation responsibilities are explained in para. 28). On average, the budgetary allocations for O&M of national primary and secondary canals rose from around Rp180,000 ($19) per ha per year in 2012 to Rp250,000 ($21) per ha per year in 2014. Under the PISP, the national schemes covered 101,026 ha (22% of the total 457,220 ha rehabilitated) (Table 3). As shown in Table 3, maintenance in the provincial schemes (primary and secondary canals) is assessed as adequate since these schemes, in practice, have been supported by funding allocations from the central government.31 These account for 81,226 ha (about 18% of the total). Accurate O&M data for the provincial and district schemes, however, are not available since data on capital budget and spending versus routine O&M budget and spending have been poorly distinguished.

31 For most national schemes (larger than 3,000 ha), actual work to operate and maintain the schemes is

carried out by provincial water resources agency staff with funding from the central government.


44. The evaluation finds that O&M funding is inadequate for the district schemes (Table 3 and Appendix 3). This finding is based on interviews with agency staff, site inspections, and PCR report. District schemes account for 274,068 ha (60% of the total area). Selected site visits suggest that no or very little maintenance (e.g., weeding or minor repairs) has been done on the district-managed primary and secondary canals since PISP rehabilitation works were completed in 2009–2011. District agency staff reported that the total irrigation budgets have increased for all districts visited, but PISP infrastructure has not been targeted for routine maintenance because the available funds are limited and were used for major repairs in other locations.

Table 3: Irrigation Operation and Maintenance Adequacy

Irrigation Scheme (Responsibility)

Primary & Secondary Canals (National and Subnational

Governments)

Tertiary Canal (WUAs)

Project Area

National Schemes

101,026 ha (22% of total project area)

Provincial Schemes


District Schemes


ha = hectare, O&M = operation and maintenance, WUA = water users association. Source: Independent Evaluation Department.

45. No data is available on the adequacy of tertiary canal O&M in all three scheme types. Tertiary canals are the responsibility of WUAs. Discussions with WUAs and farmers confirmed that monetary contributions and labor are collected for O&M on an ad hoc basis, including payment for water allocation monitoring during the dry seasons, canal clearing, and repair. Since tertiary canals are small and earth-lined and simple to maintain in most locations, labor contributions are enough to keep the canals operational, provided the secondary system supplies enough water. For more costly tertiary canal repairs, particularly in the larger national and provincial schemes, cash grants are provided under the various social assistance programs of the national and local governments to WUAs or other farmer associations. There are, however, concerns that (i) social assistance may not reach the more remote areas, and (ii) utilization of the assistance funds is not adequately monitored and is prone to corruption (Appendix 4). 46. Initially, following implementation of National Regulation No. 20 of 2006, dedicated irrigation management funds were to be established in each district to help O&M funding in particular, but also for rehabilitation and new investment. These funds were to be sourced from central and local revenues. By project closing, however, this arrangement was never implemented, mainly due to the difficulty in accumulating and setting aside earmarked funds on a multiyear basis when the districts considered annual budgets insufficient. Further, district expenditure priorities shifted from year to year. Overall, there is a continuing concern over a culture of deferred maintenance, particularly in the district irrigation schemes. In all, the goal of sustainable decentralized irrigation management (outcome 1) is assessed as partly achieved.

Adequate Largely adequate. Earth-lined small canals require mostly labor & in-kind farmer contributions. Additional social assistance provided to WUAs for more costly works, but remote locations likely to receive less assistance.

Largely adequate, but varied by province. Better O&M data are required.

Inadequate


47. Support for decentralization. Outcome 1 implied that the PISP would support decentralization. However, the project did not have specific indicator targets to measure progress in decentralization. The PISP did not directly support giving greater budget authority to local governments, which was a part of the decentralization process. All external loans and grant funds (from ADB and the Government of the Netherlands) were channeled as part of Indonesia’s national budget through three national ministries. This was to prevent complications and implementation delays. No project budget was channeled to the provinces or districts in the form of fiscal transfers, which would have become part of the local government budgets, and as such helped empower the local governments. Such transfers are known as special allocations for irrigation, and this mechanism is increasingly being used outside the project areas to fund irrigation expenses (O&M and investment) that are the responsibility of districts and provinces. The PISP’s financing arrangement gradually became at odds with the government’s domestic financing arrangement and decentralization initiative, because under the PISP, the national budget was still being used to finance activities, while some of PISP works (e.g., rehabilitation of infrastructure assigned to the provinces and districts) were to be financed by local budgets (or transfers from the central to local governments). To support decentralization moving forward, future external support to Indonesia’s irrigation sector needs to further strengthen local budget management for irrigation. 48. Higher yields of irrigated crops. The IOB farm survey done in 2015 in conjunction with the project evaluation (para. 7) found a modest increase in rice yields of 11% during the wet season (November–March) and no significant increase in yields during the two dry seasons (April–July and August–October).32 This finding was based on a comparison between the yields of PISP schemes and control schemes. However, the improved irrigation under the project mainly led to expanded cropping areas during the dry seasons. This increase in cropping intensity and area was most pronounced in the dryer schemes (i.e., those with water availability less than 1 liter per second per ha). On an annual basis, rice production (in tons per ha per year) was estimated to increase by 17% from irrigation command areas, which sizes largely did not change before and after the project.33 This increase exceeds the outcome 2 target of 14%.34 Therefore, despite a somewhat modest increase, outcome 2 is considered achieved. 49. Based on achievements of the two outcomes and four output areas, the evaluation rates the PISP effective, with a note of concern for the ability of districts to maintain the irrigation systems at their current level of productivity. This concern is reflected in the efficiency and sustainability assessments below.

D. Efficiency

50. Economic efficiency. The re-estimated of economic returns of the project is based on a review of the data and the approach used in estimating the economic internal rates of return (EIRRs) at project completion and appraisal. The re-estimation took into account updated data gathered by the farm production and household

32 Details are in Table A5.3 of Appendix 5. 33 The increase was estimated conservatively, not on areas where rice replaces other crops. 34 As for the secondary annual crops (maize, legumes, peanuts, and vegetables), these are planted as both

irrigated and non-irrigated crops. Further, secondary crops are often intercropped (i.e., planted together in the same field at the same time), and such increase in yield and production per hectare per year could not be conclusively derived.


survey carried out in 2015 (paras. 7 and 61), production data from the subdistricts,35 and updated commodity prices. The economic analysis is detailed in Appendix 5. The evaluation adopted a similar analytical approach to what was done at project completion and appraisal. The evaluation reviewed in detail the investment and O&M cost estimates for 15 subprojects representing the PISP schemes as well as benefit estimates of higher farm incomes from higher cropped areas and relatively modest yield increases. Key adjustments made to the calculation made at project completion were (i) a lower economic price of rice based on 2015 market conditions, (ii) higher gaps in rice cropping intensity between the with- and without-project scenarios (as compared to the before- and after-project scenarios used in the PCR), and (ii) more realistic (i.e., lower) estimates of incremental increase in rice yields achieved under the project .36 The recalculated EIRR is 24%, compared to 54% at project completion and 29% at appraisal. 51. The project’s relatively high EIRR is typical for projects that carry out rehabilitation at fairly low cost per ha. The sensitivity analysis shows that the PISP’s economic efficiency is generally robust. However, economic return is susceptible to lower rice yields and prices, and lower revenues from secondary annual crops.37 The EIRR recalculation assumes that project benefits will be sustained over a 20 year period. The validity of this assumption will depend upon sustained O&M financing and continuing incentives for intensive rice cultivation. To date, a cost recovery assessment (paras. 43–46) raised concerns about the districts’ ability to maintain the irrigation facilities in good condition. 52. Implementation efficiency. The project at appraisal had a planned implementation period of 6 years. However, loan signing was delayed for more than 1 year pending issuance of Water Law No. 7 of 2004. The project loan was declared effective in June 2005, 18 months after ADB approval (12–15 months behind schedule), but it only became operational in mid-2006 after issuance of National Regulation No. 20 of 2006, which operationalized the Water Law no. 7 of 2004. According to the government PCR, no substantial subproject maintenance was done during 2004–2008.38 As a result of further structural deterioration and price inflation, the rehabilitation costs rose, the project scope had to be reduced, and project implementation was delayed by late delivery or shortfall of government counterpart funds. Further inefficiencies arose because the initial imprest account ceiling was too low, and cost sharing and allocation arrangements among the three financing sources were unnecessarily complicated. Implementation performance during 2010–2011 was rated highly satisfactory. To allow full achievement of project outcomes, the project closing date was extended for 18 months from July 2011 to December 2012. Project implementation in total was delayed by 30–33 months. However, given its robust estimated economic internal rates of return the project is rated efficient.

E. Sustainability

53. Institutional sustainability. The evaluation found that the project’s capacity building supports improved irrigation planning and budgeting at the provincial and district levels. This was primarily due to the acquisition of hard data to support requests for higher irrigation budgets. The effort to adopt RP2Is—more systematic medium-term irrigation plans—has not been completed. As of May 2015, 3 years after project

35 Known as Kecamatan in Indonesia. 36 footnote 32. 37 Details are in Table A5.5 of Appendix 5. 38 footnote 19.


closing, none of the PISP districts had updated the RP2Is that were produced under the project. 54. Commissions on irrigation have been established at the district and provincial levels. The commission membership comprises 50% agency staff (mainly planning and water resource staff) and 50% members of WUAFs and other farmer or citizen associations. The commissions are reviewing periodic managerial decisions such as the cropping plan, water demand and allocation, and planned repair work, as well as ad hoc issues between farmers and agencies. Interviews in the districts suggested that district staff continue to be unsure about the full function of the irrigation commissions. Following project completion, the commissions have convened to resolve disputes and issues, but it remains unclear how the commissions influence decisions on irrigation budgets. Consultations on investment and O&M prioritization are reserved for the village, subdistrict, and district-level annual planning consultations. Irrigation planning and budgeting have benefited from the improved asset inventory procedure and software developed under the PISP in about half the districts visited; the richer and updated data supports more credible budget requests for irrigation annually. Nonetheless, without the RP2Is being adopted and updated, the irrigation planning and budgeting horizon has been shortened to a yearly instead of multiyear framework. This does not take into account the longer-term forecasts in land use (e.g., farm versus non-farm uses) and longer-term water resources challenges. 55. The evaluation found that all PISP districts have discontinued the engagement of WUAFs as independent contractors or subcontractors for light and more complex construction works.39 This was one of the key advances made in the WUA and/or WUAF capacity building exercise. The cessation of contract issuance negatively affects WUAF institution building. Without the contracting work, the WUAs and WUAFs continue to perform their O&M functions on the tertiary canals, which had been transferred to them, as well as to coordinate social assistance if it pertains to irrigation. They also continue to coordinate with the authorities on operations of the primary and secondary canals and water resource management issues. 56. The PISP did not bring about any reform in agricultural extension, as compared to the irrigation subsector. The project mainly provided improved funding for staff training, farmers’ extension sessions, and extension demonstrations. These sessions were continued as part of the Ministry of Agriculture work programs after project completion. 57. Operation and maintenance funding. As discussed in connection with the cost recovery outcome assessment (paras. 43–46), to date, only the central government is funding O&M regularly (for primary and secondary canals in national schemes). Each year more funds are being made available for schemes larger than 3,000 ha, which are under the responsibility of the national government; in 2014 Rp250,000 per ha was made available. 58. For the provincial and district schemes, the evaluation could not collect the O&M data because of time constraints and inconsistent record keeping. There is a lack

39 The reason for the contract discontinuation, as reported by the districts, was that WUAs and WUAFs, as

private enterprises, are not allowed to be engaged directly by the public sector agencies without a bidding process. The direct engagement of WUAFs under the PISP was considered an exemption of applicable procurement laws in connection with implementation of an externally funded project. DGWR staff nonetheless confirmed in July 2015 that there is enough provision under existing Indonesian law to allow relatively small contracts (under Rp200 million) to be directly awarded without tender, and that this can potentially be a basis to continue direct WUAF engagement for irrigation civil works.


of distinction between the budget and expenditure items for capital versus routine expenditures. Spending on irrigation fluctuated from year to year depending on the priority of the local government. There is also a lack of comprehensive monitoring of funding given to a particular location from various national and subnational programs and one-time initiatives. Nonetheless, interviews with agency staff and site visits consistently suggested that irrigation O&M is inadequate particularly for the district schemes. For the district schemes visited by the evaluation team, no or very little maintenance has taken place since the PISP rehabilitation in 2009–2011. The practice of deferred maintenance is still prevalent. The structures are operational but show visible wear and tear (e.g., scouring erosion, tilting or collapsed canal walls, cracks in concrete from soil subsidence, structures covered with overgrown vegetation) and will likely require another rehabilitation project within the next 5 years to avoid a decline in performance. As discussed in para. 38, the lack of funding commitment to irrigation maintenance by the districts is partly related to the growing independence of the local governments in deciding their spending priorities. This issue is a key challenge in irrigation management moving forward. The evaluation rates the PISP benefits less than likely sustainable.

CHAPTER 4

Other Assessments

A. Impact

59. The project was intended to help increase household incomes and reduce poverty in the project area. At project closing, the household incomes in the project area had increased from Rp10.2 million before the project to Rp23.4 million in 2012 (Rp12.6 million in 2003 prices), or an increase of about 25% in constant prices, which exceeded the target of 10%–15%. The poverty rate in the project area fell from 24.4% in 2008 to 6.8% in 2012, exceeding the target of a one-third reduction. These findings were based on government statistics and a household survey done at project completion in 2013. Based on these figures, the project's impact is considered significant. 60. The farm input analysis done as part of the survey found an increasingly part-time nature of rice farming in the project area. The design of future irrigation projects should critically assess the long-run economic viability of intensive rice cultivation in an economy that is undergoing a transformation, and give greater attention to the unexploited opportunities for higher-valued crops that could respond to Indonesia’s changing food demand with a more economically sustainable impact on farm income than rice. Such a focus would be of particular benefit for poorer and smaller farm households that have few income sources besides seasonal crop production. The impact assessment highlights the necessity for farm families to continue diversifying their income sources. Another benefit of diversifying agricultural activities is the positive effect on household diet. The IOB survey in 2015 found many households consumed sufficient calories and protein, but insufficient minerals and vitamins. Land near urban centers is increasingly becoming attractive for production of higher value crops such as vegetables, which would reduce water consumption, since high-value crops could be planted on irrigated land in the dry season after rice. Another solution to improve productivity and income is to consolidate farm operations; more commercially oriented farmers could rent 1–2 ha of land from other farmers who would then be freed up to carry out off-farm work.40 61. As mentioned in paragraph 7, this evaluation will further benefit from the ongoing analysis of household survey data gathered in 2015. IOB will present the survey findings in a separate document. This will provide further insights into the impact of the PISP and into other socioeconomic trends in the project area. 62. The gender action plan introduced as part of the project design enabled participation of more women in training (as project officers and farmers), project management, and as WUA members. Additional provisions were entered in local regulations to promote and monitor the participation of women. Women’s

40 Households surveyed derived 40% of their incomes from non-irrigated agriculture (livestock, tree crops

and/or fruits, and non-farm employment). The younger rural generation, especially in Java, is becoming less interested in farming.


participation rates were monitored during project implementation and progress was made in women empowerment albeit slowly. At project closing, women’s membership on the boards of 931 WUAs and 235 WUAFs exceeded the target of 15%. However, the other 4,145 WUAs and 408 WUAFs still did not have any women on their boards. Women’s participation in project training was below the target of 33%, except for the training for cultivation of high-value crops and agriculture food processing, which was 100%.41 63. As all civil works under the PISP involved repairs of existing structures, there was no adverse environmental impact arising from these activities. The project required the acquisition of small parcels of land for minor realignment of headworks and canals in connection with the rehabilitation works. The Project’s Compensation Policy Framework and Procedural Guidelines of 2005 called for the project to either purchase lands following the approved procedure for involuntary land acquisition, or to solicit for land contributions from WUAs as a precondition for irrigation rehabilitation. As the project was implemented, no involuntary acquisition was carried out and the project’s land acquisition budget was not used. All necessary land acquisitions were done in the form of voluntary donation by WUAs, which is consistent with information received by the evaluation team during its site visits. However, the value of counterpart land contribution was not accounted for in the final project costing. The evaluation therefore recommends that a better accounting is done in future projects. In addition, there is a need for projects such as the PISP to specify how voluntary land contributions are agreed upon and compensated, if necessary, from amongst the WUA members.

B. ADB Performance

64. The project, as designed, met the priority needs of the government, and ADB was timely in its support for the impending government-wide decentralization reform in Indonesia. There were gaps in its conceptualization as discussed in paras. 36–40. The project implementation challenges and delays from 2003 to 2008 mostly arose from (i) issues with decentralization, and (ii) shortages in the counterpart budgets. Nonetheless, ADB assisted the project implementation by moving project supervision responsibility to the Indonesia Resident Mission in March 2007, revising the project scope in 2009, improving disbursement arrangements (enlarging allowable imprest account balances), and extending the project closing date. ADB fielded 14 review missions, which included site visits and consultations with local governments and farmers. The evaluation rates ADB’s performance as satisfactory.

C. Borrower and Executing Agency Performance

65. The institutional and legal framework uncertainties during 2003–2006 and shortfalls in counterpart funds were the main problems the project faced from the borrower side until 2008. These issues substantially delayed project progress from 2004 until 2008 and resulted in the project not being able to cover the envisaged 625,000 ha. The project national steering committee headed by the National Development Planning Agency and with the membership of three executing agencies responded well in coordinating responses to keep project implementation progressing. These included revising the project’s scope and modifying financing arrangements (para. 16). The three ministries, six provinces, and 27 districts involved in the PISP developed good horizontal and vertical coordination, and the National Steering Committee for Water Resources was instrumental in maintaining a broad strategic view and project goals for these

41 footnote 6.

Other Assessments 23

stakeholders. Nonetheless, given the substantial implementation delays and difficulties to address impediments in a timely manner during 2004–2008, the evaluation rates the combined performance of the borrower and the executing agencies as less than satisfactory.

CHAPTER 5

Issues, Lessons, and Follow-Up Actions

A. Issues and Lessons

66. Irrigation financing. A key goal under the PISP was to help the project areas sustain irrigation O&M. This had been a long-standing challenge for the agriculture sector. During and after the project, progress has been made to increase O&M spending for the national and provincial irrigation schemes. However, increasing the districts’ commitment to irrigation O&M (and capital investment) has been more difficult than envisaged at appraisal (paras. 38 and 44 and Appendix 4). Because of progress made in decentralization, districts are increasingly empowered to make spending decisions and may not consistently prioritize irrigation. Under its multiyear national irrigation programs and the national government provides special budget allocation transfers to the provinces and districts (para. 47).42 These transfers become part of the local budgets with earmarked restrictions for irrigation.43 However, even for this funding there is a varying degree of compliance in the use of these allocations. To help address this situation, the evaluation recommends that various options are explored, including giving local governments more incentives (e.g., matching finance) to allocate more funds to irrigation, or finding mechanisms to better enforce the earmarked use of the transfer funds (as agreed between the national and subnational governments).44 67. Project design and scope. The PISP was generally well designed. Its institutional arrangements effectively aligned the budgets and tasks of each project stakeholders, and promoted a more independent operations of local agencies. The planning agencies at the national and local levels effectively coordinated the line agencies. It is noted that the PISP scope of activities was rather narrow focusing mostly on irrigation capacity building and rehabilitation (paras. 36–37). This focused approach was helpful to allow simpler project implementation. However, experience under the PISP pointed to two lessons. First, the focused intervention in water delivery needs to be implemented in conjunction with complementary projects or in appropriate (water-short) locations to ensure that higher-level project outcomes (e.g., increased yields and improved incomes) can be better realized. Second, a rigid and narrow scope of activities may not be

42 In Indonesia these are known as DAK or Dana Alokasi Khusus (special budget allocations). These are

targeted at different sectors and programs prioritized nationally. 43 The PISP did not provide funds through the DAK special budget allocation channel since the project funded

only the national government budget. The PISP therefore became at odds with the government’s internal financing arrangements and decentralization effort. Under the PISP, national budgets were still used on the irrigation systems that since 2006 had become local responsibilities, to be funded by the district and/or provincial budgets.

44 The WISM Program and other projects prepared after the PISP are utilizing an on-granting mechanism, whereby district recipients of national transfers are obligated to finance the first part of the agreed irrigation works before receiving reimbursements. This helps to ensure the earmarked irrigation funds are appropriately used.

Issues, Lessons and Follow-Up Actions 25

optimal for projects that aim to support decentralization. Decentralization fosters more independent and bottom-up decision making by local governments and communities, and it is therefore better supported through projects that allow a broader and flexible menu of activities. 68. Staff training. In its delivery of capacity building training, the PISP design resulted in positive lessons. Staff training modules adopted under the project generally supported routine work functions of district and provincial agencies (e.g., planning, participatory consultations, construction management), rather than only project-specific tasks (e.g., loan disbursements and PISP-specific reporting). Further, there were efforts to follow up on the training sessions with hands-on activities that made use of the training materials. The only shortcoming was that quizzes or short tests should have been administered in the training sessions to help gauge if knowledge transfer had taken place (para. 24).45 69. Tracking of capacity improvement. As a part of the continuing institution building, agriculture sector indicators can be improved by including more service performance indicators for agencies. For irrigation, these can be the extent of water loss in irrigation systems and irrigation service coverage during dry seasons. For agricultural extension, there is a need for the Ministry of Agriculture to monitor to what extent farming practices are actually changed and influenced by the extension programs delivered (para. 31). 70. Presentation of risks in appraisal report. Reflecting on the project preparation process, the evaluation notes that the project team was fully aware of the risks to the PISP during its implementation. These risks emanated from the planned updating of the Water Law in 2004 and associated key irrigation regulations (paras. 10 and 20), and the impending government-wide decentralization process. The exact implications on how institution structures and procedures would change and the timing of these changes were not known. The report and recommendation of the President did not discuss the full extent of these risks, perhaps due to space limitations or a concern over further delays to the project’s approval (footnote 1). This points to a continuing need for the project review process in ADB to allow a flexible project design if warranted, for example during this uncertain period in early 2000s for Indonesia. A more forthcoming discussion on risks and a flexible design at approval would have made it easier later to change the project scope and financing approach. This would have reduced the project implementation challenges and delays. 71. Reduction of administrative burden. Government staff feedback on PISP implementation suggests a need for ADB to improve client service and reduce administrative and paperwork burden. This can be done by simplifying project disbursement (ensuring sufficiently large imprest account replenishment thresholds and simpler cost-sharing arrangements) (paras. 16 and 17). The sector project modality used for the PISP required a large volume of reports because each of the 1,614 schemes handled by the project was recognized as a subproject. Although proposed investments should be subjected to feasibility studies, efforts should be made to reduce report generation in future projects (e.g., by combining scheme assessments, simplifying the feasibility study format, and delegating feasibility review to the respective districts and provinces).

45 Certification may need to be introduced for certain tasks and positions within the irrigation sector. Staff

certification, however, is usually beyond project-level training support.


72. Monitoring and evaluation. The project effectiveness, efficiency, and impact assessments suggest a need for high-quality outcome and impact data at the project level. In addition to self-evaluation by line agencies and project offices, future projects should consider allowing more independent entities to gather key data (e.g., on yields and cropped areas, incomes, and farmers’ views on irrigation and extension service quality). This can be done by the local planning or finance agency, or an agency of the local parliament. 73. Pursing project impacts––rice self-sufficiency versus income improvement. Strategic discussions during PISP implementation among the project stakeholders often led to confusion between the goals of increasing rice production toward self-sufficiency, which is a national strategic goal, and the urgent need to improve incomes from farming. This is not a new issue, and it is one that is common in many countries. In moving forward with sector dialogue and preparation of new projects, these sometimes competing needs should be continuously discussed particularly regarding the acceptable trade-off between staple food (rice) self-sufficiency (as a part of a national security plan) and its economic costs in terms of fiscal subsidy burden on the national and local budgets, as well as in terms of lower farm incomes. 74. Adapting to new trends. The PISP household surveys point to emerging trends in the agriculture sector that future projects need to take into account. These include (i) the growth of contract farming to meet agro-processing and large wholesalers’ needs, (ii) the emergence of more commercially oriented farming, (iii) the consolidation of small land plots (especially in Java), (iv) more dynamic labor movement into and out of rural areas, and (v) the movement of youth away from agriculture. Household data at project completion showed that about 90% of labor inputs for paddy land preparation in PISP districts were hired labor and tractor services. There are also challenges from conversion of fertile and irrigable farm lands into nonagricultural uses and increasing competition for water. Discussions at the central and local levels pointed to the need to consider innovative water-saving technologies for some crops, such as using sprinkler irrigation for contoured lands, as water-saving technologies are becoming cheaper to implement. 75. Land acquisition. In relation to the PISP land acquisition assessment (para. 63), this evaluation recommends that in future projects that rely on voluntary land contributions or acquisitions from beneficiaries, a clear mechanism should be identified on how such contributions are agreed upon and compensated, if necessary, among the members of beneficiary groups (e.g., WUAs). Further, these contributions should be accounted for as counterpart funds to the project.

B. Follow-up Actions

76. Continue institutional improvement. A key sustainability issue for PISP districts was the lack of adoption of RP2Is, the rolling medium-term plans for irrigation, which are necessary for systematic sector planning and budgeting (paras. 53–54). More efforts have been made to improve the irrigation asset inventory using the software developed under the PISP. However, the RP2I document needs to be made more user-friendly and the irrigation commission more effective. The RP2I format, as completed by the PISP, was too long and not suitable to facilitate discussions with nontechnical stakeholders. There is a need to mainstream the role of the irrigation commissions, continue staff training in irrigation, and further strengthen the legal framework in

Issues, Lessons and Follow-Up Actions 27

support of continuing the irrigation reforms.46 RP2I documents and deliberations of the irrigation commissions need to be integrated into the existing district and provincial planning and budgeting cycles to garner greater attention and funding commitment to irrigation at the local levels. A more in-depth analysis is needed to pursue this. Given that institution building is a long-term undertaking, this evaluation recommends that the follow-on irrigation project in Indonesia covers the same districts covered by the PISP and WISMP to build on the work and momentum of these projects. 77. More cost-effective operation and maintenance. To help overcome budget constraints at the district level, the PISP districts should pursue cost-effective construction solutions such as engagement of the WUAFs and WUAs as contractors or subcontractors, since they have performed satisfactorily and at a lower cost. This would require a review of legal impediments (if any) that have prevented the engagement of WUAs and WUAFs without a tendering process (para. 55). Furthermore, to enhance efficiency in budget utilization—both in budget programming and in selection of contractors—citizen groups (WUAFs and farmer associations) can be supported to be involved in local democratic processes such as observing and taking part in district parliamentary hearings for annual budget allocation, and observing the bidding process for civil works.

78. Monitor operation and maintenance funding. As discussed in para. 66, there is a need for the central government (through the Ministry of Finance) to better monitor the utilization of the special irrigation budget transfers given to the districts, including segregating capital and routine budgets and expenditures. There also needs to be a mechanism to bind districts to previously agreed funding commitments for irrigation.

46 This includes the issuance of a new water law and the revision of its supporting regulations. Water Law

No. 7 of 2004 was revoked by the Constitutional Court of Indonesia in 2013, mainly due to the need to review the legal provisions for private sector participation in water projects.

Appendixes

APPENDIX 1: EVALUATION MATRIX

Criteria Evaluation Questions/Indicators Evaluation Findings

I. Project Performance

A. Relevance

Are project objectives consistent with national agriculture, water, and rural development strategies and policies; ADB’s country strategies and sector policies; and the needs of the beneficiaries?

Yes.

Assess relevance before and after the project. Are the original objectives still relevant at the time of evaluation? Have significant changes in the project context or ADB policies been retrofitted to the design

Yes. Scope change introduced to better deal with implementation challenges.

Were the project activities and outputs (i.e., component mix) appropriate for achieving the project’s core objectives? Were they supported well by other project or program activities (by the government and other development partners) to achieve the project outcomes and impacts?

Partly.

Was the scale of project inputs and outputs adequate to achieve stated outcomes? Were outputs and outcomes too ambitious given the time and resources available?

Partly. Outputs and outcomes not adequate to strengthen farmers’ bargaining power.

Were other aspects of project design done well (financial allocation and modalities, implementation arrangements, and monitoring and evaluation)?

Yes, although there were issues in disbursement design of grant.

Were project risks identified during appraisal sufficiently mitigated in the design (e.g., government/executing agency reorganization, low capacities, slow release or lack of counterpart funds, resettlement complexity, extended time and resources to meet safeguard requirements)?

Partly.

Has the project design been participatory, taking into consideration input and needs expressed by key stakeholders (executing agencies, planning and finance ministries, local governments, and target beneficiaries)?

Yes.

Has the final appraisal report integrated relevant lessons learned at the country and sector levels? Was the appraisal report consistent with the project preparatory technical assistance report? Did it gather key information on the local conditions? Has the project benefited from available knowledge (e.g., the experience of other similar projects in the area or in the country)?

Yes

Does the envisaged targeting approach facilitate access for disadvantaged groups (smallholders and women) as planned?

Yes

B. Effectiveness Envisaged outcomes were (i) decentralized management of irrigation systems on a sustainable basis, and (ii) increased yields of irrigated crops. Outputs were (i)

See design and monitoring framework in Appendix 3.

30 Appendix 1


upgraded district capacity for irrigation responsibilities, (ii) water users associations strengthened and empowered, (iii) improved irrigation performance, (iii) agricultural services rationalized and developed, (iv) farmer access to agricultural services improved, and (v) water resource management information system improved. To what extent have the project outcomes and outputs been achieved (or to what extent will they be achieved)? The evaluation will consider the design and monitoring framework indicators (marked by an asterisk) and other indicators.

Outcome 1: Decentralized and sustainable management of irrigation systems.

Were the provinces/districts given greater responsibilities in irrigation and agricultural sector management?

Yes.

Were provinces/districts given greater authority in project management (strategy formulation, planning, budgeting, activity selection, procurement, implementation involvement)?

Partly, although not directly under the PISP.

Did O&M of irrigation budgets and spending per hectare grow in real terms, and was deferred maintenance avoided?

Partly, in national irrigation and provincial schemes.

Outcome 2: Increased yields of irrigated crops. Did average paddy yields increase (3.5 tons/ha to 4.0 tons/ha)?

Did cropping area, cropping pattern and intensity change?

No. An increase of only 0.4 tons/ha during wet season.

Were there changes in total outputs?

Yes. More fallow land used in crop production.

To what extent were outputs achieved? Yes

Output 1: Upgraded capacity of districts and provinces for irrigation management.

Largely achieved. See design and monitoring framework in Appendix 3.

Output 2: Irrigation performance at scheme level improved.

Largely achieved.

Output 3: District agricultural services rationalized and developed; water users associations strengthened and empowered; farmers’ access to agricultural services and inputs improved.

Partly achieved. Extent of adoption of new technologies unknown, so as improvement in famers’ access to agricultural services and inputs.

Output 4: Water resources management information system improved.

Largely achieved.

C. Efficiency What was the recalculated project economic internal rate of return?

Were there any delays in project implementation and delivery that may affect benefits or costs in a significant

24%, indicating an efficient use of resources. Yes. About a 2.5 year delay.

Evaluation Matrix 31


manner? In addition to delays and cost overruns, were there unnecessary complications in project management that could have been avoided?

Yes. Cumbersome disbursements.

D. Sustainability How resilient are the achievements in institutional and personnel changes and capacity building?

Skills building and training benefits will continue, although there is a relatively high staff turnover.

Political and institutional sustainability: Is there clear indication of government commitment (both at the national and local levels) to continue supporting the project activities and reforms after the loan closing (e.g., provision of budget, staff, policy, and institutional mandate)?

There was an effort to maintain momentum of reforms in irrigation planning and budgeting, but the main PISP output, the district irrigation development and management plan, has not been adopted.

Beneficiary ownership: Do project activities benefit from participation and ownership of local communities, grassroots organizations, and the rural poor?

Stakeholders participated in PISP activities, but they have not developed full understanding and ownership of some activities.

Technical sustainability: Are adopted approaches technically viable? Do project users have access to adequate training for maintenance?

Yes, technical solutions were appropriate.

Financial sustainability: Are the activities promoted by the project adequate financial returns?

Adequate O&M for the national and provincial schemes but not for the district schemes. Adequate O&M.

Environmental sustainability: Are the ecosystem and environmental resources (e.g., water availability, soil fertility, vegetative cover) likely to be stable or is there depletion?

Likely to be environmentally sustainable.

II. Other Assessments

A. Impact 1.5 million farm family incomes increased by about 15%–20%. Incidence of poverty among project beneficiaries reduced by one-third. Did the composition source of household incomes change? Did farm households’ physical assets change (land holdings, livestock, trees, equipment)? Did other household assets change (houses, motorcycles, radios, telephones, etc.)? Did the natural resources status change (water, forest,

Achieved. Achieved. To be further assessed as data from the 2015 household survey is being analyzed.

32 Appendix 1


etc.)? Are there any signs of increased/decreased soil fertility?

B. ADB Performance (quality-at-entry and supervision)

Did ADB mobilize adequate technical expertise in project design?

Yes.

Was the design process participatory (with national and local agencies, and grassroots organizations)?

Yes.

Were specific efforts made to incorporate the lessons and recommendations from previous independent evaluations and project completion reports in project design?

Yes.

Did ADB (and the government) take the initiative to suitably modify project design (if required) during implementation in response to any changes in circumstance or emerging opportunities?

Yes.

How was ADB’s performance in procurement supervision and disbursements?

Adequately responsive.

Was prompt action taken to (i) ensure the timely implementation of recommendations from back-to-office reports, including the midterm review; and (ii) resolve any implementation bottlenecks?

Yes, with some delays.

What is the role and performance of the resident mission? How was the relationship between headquarters and the resident mission?

Both sides were supportive as PISP preparation was expected to lead to new projects.

Has ADB been active in creating an effective partnership and coordination among partners (i.e., the Government of Netherlands) to ensure the achievement of project objectives?

Yes.

C. Borrower Performance

Was the government actively involved in the design? Yes.

Has the government assumed ownership for the project through initiatives and actions? Did the government take the initiative to modify project design (if required) in response to any changes in circumstance or emerging opportunities?

Yes, concurrent with ADB initiatives.

Have the flow of funds and procurement procedures been suitable for ensuring timely implementation?

Yes, the procedures were suitable.

Have adequate staffing and resources been provided for project management? Have counterpart funds been provided adequately and on time?

Partly. Amounts were adequate, except for O&M. Provision of counterpart funds was delayed during 2006–2008 for the Ministry of Home Affairs

Evaluation Matrix 33


Has project management discharged its functions adequately, and has the government (through the National Development Planning Agency and Ministry of Finance) provided guidance to project management when required? Did the government ensure adequate coordination of national agencies (Ministry of Home Affairs, Ministry of Public Works, Ministry of Agriculture), and between national and subnational levels?

Project level coordination and executing agency performance was satisfactory. However, there were challenges faced by the borrower (prolonged clarifications on the water sector and irrigation legal framework, difficult decentralization adjustments, delays in counterpart funds), which caused substantial difficulties in implementation.

Have audits been undertaken in a timely manner and reports submitted as required?

Yes, and financial audit reports were submitted to ADB in a timely manner.

Has an effective monitoring and evaluation system been put in place, and does it generate information on performance and impact, which is useful for project management to take critical decisions?

A project performance management system was developed by the project and implemented at the central and local levels.

Have loan covenants and the spirit of the loan agreement been followed?

Covenants have largely been met except for delays in counterpart funds.

Has the government contributed to planning an exit strategy, and has it made arrangements for continued funding of certain activities?

Partly. Commitment to adopt RP2Is (district irrigation development and management plan) and irrigation commission has not clearly been shown; so as the lack of sustained allocation of staff and budget for some activities.

ADB = Asian Development Bank, O&M = operation and maintenance, PISP = Participatory Irrigation Sector Project. Source: Independent Evaluation Department (IED)

APPENDIX 2: COVERAGE OF PRIMARY DATA GATHERING

No. Province/District Agency Interview a Household Survey

I. Lampung Province

1. South Lampung

2. East Lampung

3. Central Lampung

4. Pesawaran

II. Banten Province

5. Lebak

6. Pandeglang

III. West Java Province

7. Garut

8. Kuningan

9. Indramayu

10. Cirebon

IV. Central Java Province

11. Banyumas

12. Purworejo

13. Brebes

14. Tegal

15. Cilacap

V. East Java Province

16. Madiun

17. Tulung Agung

18. Bojonegoro

19. Lamongan

20. Ngawi

21. Malang

VI. South Sulawesi Province

22. Sinjai

23. Maros

24. Bone

25. Bulukumba

26. Tana Toraja

27. North Toraja

a Includes two control districts of Batang (Central Java) and Blitar (East Java). Source: IED

APPENDIX 3: PROJECT PERFORMANCE AGAINST DESIGN MONITORING

FRAMEWORK

Design Summary Performance Targets Achievement at Project Completion Report

Evaluation Findings

Impact

Increased economic growth in 25 districts of rural Indonesia

1.5 million farm family incomes increase by about 15%–20% by 2012

Average net farm income increased from Rp10.2 million before the project to Rp23.4 million with the project. This is equivalent to an increase of 129.5% in nominal terms.

Income increased by 24% in real terms, still exceeding the target range.

Reduced rural poverty

Incidence of poverty among project beneficiaries reduced by one-third by 2012

Incidence of poverty fell from 24.4% in the baseline survey in 2008 to 6.8% in the 2012 survey, well in excess of the target impact of one-third reduction by 2012.

Income and poverty impacts to be further assessed as the data from the 2015 household survey are being analyzed.

Outcome

Sustained decentralized management of irrigation systems

Adequate level of routine O&M of irrigation schemes regularly undertaken for about 456,000 ha by 2012a

1,614 irrigation schemes covering 457,220 ha were rehabilitated to avoid outstanding maintenance works.

On average, there has been at least adequate funding for O&M for 101,926 ha (of the total 457,220 ha) under the national government’s responsibility. Funding was Rp180,000/ha in 2012 rising to about Rp250,000/ha in 2014.

Deferred maintenance is avoided for 456,000 ha by 2012

O&M funds of the national standard of Rp180,000/ha (as of 2012) or $19 per ha were available for all central and most provincial schemes. O&M allocation for nearly all district schemes was below this standard.

No updated O&M funding data for provincial and district schemes. Provincial schemes (totaling 81,226 ha) have received O&M support from the central

36 Appendix 3


Evaluation Findings

government. District structures visited suffered from deferred maintenance. District schemes (totaling 274,067 ha or 60% of the total) are not adequately maintained. The last available data (2012) indicates estimated district spending of Rp138,000 ($15)/ha per year

Indicator(s) are needed to monitor the progress of decentralization.

Increased yields of irrigated crops

Average paddy yields increase by 0.5 tons/ha, from 3.5 tons/ha to 4.0 tons/ha in 2012

Average rice yields in the wet season increased by 0.96 tons (17.6%), from 5.47 tons/ha to 6.43 tons/ha, and in the dry season by 0.91 tons (18.0%), from 5.08 tons/ha to 5.99 tons/ha.

Confirmed. The increase in rice yield alone is estimated at 11%, but accounting for better utilization of fallow (idle) lands, the incremental rice yield increase is estimated at 17%/ha per year.

Outputs

District capacity for management of devolved irrigation responsibilities upgraded

25 district governments strengthened and functioning effectively as focal points for irrigation management by 2012

27 districtb and six provincial governments were strengthened to effectively implement their tasks and responsibility for participatory irrigation management, in accordance with current national policies and regulations on irrigation.

Training targets were met in terms of number of personnel and types (staff, beneficiaries, women). No assessment was made on the quality of skills improvement achieved.

Project Performance Against Design Monitoring Framework 37


Evaluation Findings

5,000 district staff trained in irrigation planning and management by 2012

8,311 central, provincial, and district government staff trained in participatory irrigation management during the project

Improved irrigation planning and management cycle introduced as well as issuance of regulations to implement the Water Law No. 7 of 2004 and National Regulation No. 22 of 2006. Regulations are to be updated in view of the revocation of Law No. 7 of 2004 in 2013. There was no reform or business process improvement supported for agricultural extension.

District irrigation management plan prepared in participating districts by 2012

The content of RP2Is, which comprise a management plan for existing systems and a development plan for new systems, was endorsed by the National Steering Committee for Water Resources of the National Development Planning Agency in November 2008.

26 RP2Is were prepared, of which 18 have been endorsed by district heads. The legal basis for RP2Is in Indonesian water legislation is still being drafted.

RP2Is (first version) were produced in all except one district. However, these have not been adopted and used by the districts after project completion.

38 Appendix 3


Evaluation Findings

Irrigation performance at scheme level improved

6,250 WUAs on government- managed irrigation schemes either strengthened or established by 2012

The target was revised to 5,000 WUAs following a reduction in the coverage of irrigation schemes from 625,000 ha to 456,000 ha in 2009.

No further observations.

WUAs within a scheme or in the schemes on the same river within close proximity are federated into WUAFs

643 WUAFs formed and legalized.


Management of irrigation schemes covering about 250,000 ha is transferred to WUAs by 2010 through partnership agreements [Revised: O&M for about 456,000 ha implemented in joint management between government and WUAs as agreed in participatory irrigation management agreement arrangements]

In 2012, the average O&M allocation was Rp138,000 per ha for district schemes, and Rp180,000 per ha for provincial and central schemes. Participatory irrigation m anagement agreement arrangements for O&M were implemented in 87% of entire project area.

Cost recovery was actually an outcome 1 target (see above). Participatory irrigation management agreement arrangements vary. For the tertiary canals, the governments assist farmers as originally intended at appraisal. For the primary and secondary systems, farmers are employed by national and local governments.

Full cost of O&M of primary and secondary systems by the government and tertiary systems by the WUA assumed by 2012

O&M of primary and secondary systems are the responsibility of the government, supported by WUAs or WUAFs through the joint management participatory irrigation management agreement with the district water resources agencies.

O&M responsibility was successfully demarcated for WUAs as well as between the national, provincial, and district governments.

Project Performance Against Design Monitoring Framework 39


Evaluation Findings

Full cost of O&M of tertiary systems is already the responsibility of WUAs since issuance of Regulation 20/2006 on irrigation management

District agricultural services rationalized and developed, WUAs strengthened and empowered, farmer access to agricultural services and inputs improved

Adoption of improved irrigation agronomy techniques by members of 5,000 WUAs

Improved agriculture practices adopted; 112 agricultural training and demonstration plots established, which provided nine agricultural topics, including system rice intensification, women’s participation, water management, and access to credit.

Training provided and agricultural training and demonstration units set up as reported in the project completion report.

Adoption of improved technology packages developed under other projects are adopted by the farmers belonging to 5,000 WUAs

The Ministry of Agriculture decided to use the agricultural training and demonstration units to disseminate new technologies instead of piloting activities. Farmers were trained through these agricultural training and demonstration units, each of which serves 4,000 ha implemented in 26 districts.

The extent to which improved techniques were adopted is unknown.

11,794 WUA members received training in intensification and diversification of enterprises, and water management systems.

The successful completion of pilot study on using WUAs to improve farmers’ bargaining power and make service/input providers more accountable to farmers in 5 districts

Study was completed and extension training done, although more needs to be done to improve farmers’ bargaining power against service

40 Appendix 3


Evaluation Findings

by 2011

and input providers.

Water resources management information system improved

Water resources data management unit at Directorate General of Water Resources Central Office institutionalized by June 2010

The Water Resources Data Unit was institutionalized in 2005 and equipped in 2007.


Water resources information and data gathering, processing, and quality control facilities in project districts improved and integrated into the Directorate General of Water Resources Central Office decision support system by 2010

With reorganization of the Ministry of Public Works in 2008 and establishment of river basin organization offices, the tasks of the Directorate General of Water Resources Central Office were diluted. In the new setup, tasks for water resource data gathering for cross- boundary rivers were transferred to the provinces, and those for cross-province rivers were transferred to the river basin organization offices.

This component supported introduction of the irrigation asset management information system, which is a critical requirement for enhancing irrigation sector planning at the district and provincial levels.

In all project districts and provinces, water resource information centers were set up. A link with the Directorate General of Water Resources Central Office’s decision support system is still under construction.

ha = hectare, O&M = operation and maintenance, RP2I = Rencana Pengembangan dan Pengelolaan Irigasi (district irrigation development and management plan), WUA = water users association, WUAF = water users association federation. a The project performance target to rehabilitate 625,000 ha of irrigation systems was modified to 456,000 ha and was approved by the Asian Development Bank on 15 October 2009. At appraisal, the project was expected to (i) undertake light rehabilitation works covering about 625,000 ha, and (ii) fully rehabilitate 137,000 ha. Due to significant increases in unit costs for civil works resulting from increases in fuel costs in 2005 and 2008, the scope of the light rehabilitation works was reduced to 319,000 ha, and the scope of full rehabilitation works was revised from 137,000 ha to 456,000 ha. b Including Pesawaran District, a new district split in 2006 from Lampung Selatan District in Lampung Province, and Toraja Utara District, which was split in 2009 from Tanah Toraja District in South Sulawesi Province. Source: Project Completion Report and IED.

APPENDIX 4: INSTITUTIONAL ASSESSMENT

1. This institutional analysis of the Participatory Irrigation Sector Project (PISP) covers five areas. Section A covers capacity building of districts and provinces in irrigation. Section B discusses capacity building of agricultural agencies in delivering extension. Section C covers water users association strengthening. Section D discusses irrigation cost recovery which is related to Sections A and C. Section E is on institutional issues in project implementation.

A. Capacity Building of Districts and Provinces in Irrigation

2. The PISP was intended to establish a sustainable decentralized irrigation management (PISP Outcome 1), which entailed improved capabilities of the districts and provinces to take responsibility in managing irrigation schemes under their authority. Project activities included training of agency staff, and on-the-job exercises (with the support of consultants) to reform irrigation planning, regulations, and project management.

1. Staff Training

3. The initial part of the district and provincial capacity building was staff training in 27 districts and six provinces. The training modules delivered were intended to support the PISP component activities. The project completion report (PCR) includes a summary list of modules, which benefited 8,311 staff (971, or 11.6%, of whom were women).1 This exceeded the appraisal target of 5,000 staff to be trained. Training was targeted at the three PISP agencies (planning, irrigation, and agriculture) and was delivered by consultants. Most subjects were related to irrigation: (i) planning and coordination; (ii) design, construction, operations, and management; (iii) agriculture on irrigated land; (iv) overseas training in water management; and (v) short courses in participatory irrigation management. Other sessions included familiarization of staff with new local irrigation regulations. The evaluation generally considers the training topics appropriate. Although the training topics supported specific PISP activities, they generally have long-term relevance to the local governments’ capacity building. There were some financial training for disbursement that was only relevant to implement the PISP and meet the requirements of the Asian Development Bank (ADB); these were less useful after the project. The government PCR reported that a training needs assessment was done in the districts and provinces before the modules, and that targeted trainees were identified.2 High personnel turnover is a challenge that has been consistently mentioned in the provincial and district interviews during the evaluation as well as in the government’s PCR. Another issue with the PISP staff training was that no assessment (e.g., a quiz or test) was conducted after the training sessions. This would have helped assess the extent of knowledge transfer and the sessions’ effectiveness. Nonetheless, the training sessions were successfully followed up by hands-on staff work in relevant project activities (e.g., strategic planning, contract management and supervision, and extension delivery), which enhanced the benefits of staff training.

2. Irrigation Reforms

4. Irrigation reforms supported by the PISP centered around the adoption of the district irrigation development and management plan (Rencana Pengembangan dan Pengelolaan Irigasi or RP2I), a 5-year rolling sector plan to be developed in each district and to be supported by robust data on irrigation facility inventory (operation and damage status), projected water demand (based on future land-use plan and crop needs), and a water resource management plan. Additionally, an irrigation commission

1 Project Completion Report, Appendix 3. 2 Government of Indonesia. 2013. Participatory Irrigation Sector Project (PISP). National Project Completion Report. Jakarta.

42 Appendix 4

(comprising at least 50% citizens and farmers) was to be established in each PISP district and province to promote more participatory and transparent decision making on irrigation. The three main reform activities (asset inventory taking, operation of the irrigation commission, and drafting of the RP2I) in paras. 5–8 below. 5. Inventory of irrigation assets. Under the PISP, the upgrading of the irrigation asset inventory was to be funded and improved in all districts. By project’s end, updated inventory data were collected for 431,111 hectares (ha). Gathering of irrigation asset data had been done before the PISP using different methods to help the drafting of annual work plan proposals for irrigation. The effort under the PISP was to further standardize, modernize, and simplify asset inventory taking, using global positioning system-referenced photographs, digital maps, short descriptions, and rough cost estimates to replace the older, more numerical engineering assessment in inventory taking. The goal of the changes was to make data collection easier, cheaper, and faster so districts would have a more current and complete view of their irrigation infrastructure status, albeit with fewer details. 6. The PISP successfully introduced the needed software system called the irrigation asset management information system (SI-PAI). Staff feedback on the system has been positive. With the new software and procedure, data can be collected by less-trained personnel (e.g., younger farmers who use smartphones lent out by the district) and the data are better understood by decision makers not familiar with technical data. However, sustained adoption of the SI-PAI varies among districts. Of the nine districts visited by the evaluation, five have used the SI-PAI on a sustained basis (Table A4.1). The extent of adoption of the system is seemingly linked to the level of understanding among key staff on the usefulness of the system, and their involvement during SI-PAI development in their districts. For example, in Tulung Agung District (East Java), the head of the water resource agency and key staff members were involved in the software trials during the project; as a result, they have continued to use, and implement improvements to, the system.3 In contrast, in the nearby district of Malang (East Java), there was no staff involvement, and following the project, the agency reverted back to using a preexisting inventory system. Another challenge to sustaining the SI-PAI is a lack of budget allocation (e.g., in Lebak District in Banten Province). An encouraging finding during the evaluation was that all district irrigation staff interviewed have come to value and recognize the importance of complete and credible infrastructure status data in justifying irrigation work plan and budget requests for the irrigation sector. 7. Irrigation commission. The irrigation commissions were created as coordination bodies and a platform for dialogue among the public sector decision makers, water users associations (WUAs), and citizen stakeholders in irrigation to render decision making in irrigation more open and participatory. The irrigation commissions are to play a role in all irrigation issues at the provincial and district levels, and they were formed based on Minister of Public Works Decree No. 31 of 2007. The PISP helped form and convene an irrigation commission in each of the 27 districts. Visits to nine PISP districts showed that the commissions have remained somewhat active since project completion, and they mainly facilitate decision making on water allocation and resolve conflicts (Table A4.1). There are a few exceptions, including in Banyumas District (Central Java), where the irrigation commission ceased to exist following the repeal of Water Law No. 7 of 2004.4 The district expects its irrigation commission to be reactivated in 2016 in recognition of its important roles in water distribution and gathering data on planned crop planting. In Cilacap District (Central Java), irrigation commissions were set up at the subdistrict level to help perform the above mentioned functions. The irrigation commissions have generally become less active and have met less often since the PISP was completed, but their functions and existence have been sustained except in some districts. In one non-PISP district visited (Blitar

3 Inventory data were updated in 13 schemes in 2014 and 40 schemes in 2013. 4 The 2013 decision of the Constitutional Court of Indonesia on the annulment of Water Law No. 7 of 2004 was made because

the Court agreed with the petition that considers the law not adequately protecting the public interest in the public-private partnerships in the water sector.

Institutional Assessment 43

District, East Java), no commission was established. In East Java and South Sulawesi provinces, the irrigation commissions, which were formed in 2009 under the PISP, are still active.

Table A4.1: Irrigation Institutional Capacity Building under the Participatory Irrigation Sector Project

Province/District Inventory of Irrigation Structure

Irrigation Commission

Status of District Irrigation Development and Management Plan

Banten

1. Pandeglang New inventory software still utilized; in 2015 it was used on 10 schemes

Formed by a district decree; routinely meets to discuss water allocation

Document completed and used until 2010; not updated after PISP

2. Lebak

New software and simplified report not utilized in 2015 because of budget constraints, but plans to continue in 2016

Formed by district head and active; minimum one meeting annually


West Java

3. Kuningan New SI-PAI software used; local government allocates budget annually to carry out inventory taking

Formed by district head and active; convenes two times a year mostly to discuss water allocation


4. Cirebon New SI-PAI software and utilized after PISP; in 2015 it was used in three irrigation areas

Formed by district head and active; conducts meetings on water allocation


Central Java

5. Banyumas New software used and budget allocated to inventory updating

Not active since Water Law No. 7 of 2004 was repealed in 2013

Document completed and used only 2010; not updated after PISP

6. Cilacap New inventory software and simplified report not utilized

Formed and active; commissions were formed in subdistricts as well; conducted comparative study visit to Malang in 2014

Document completed but not used anymore; database of irrigation system based on geographic information system will be built

East Java

7. Tulung Agung New software and simplified report format utilized

Formed and active Document completed, but not updated or used anymore

8. Malang

New SI-PAI software and simplified report not utilized

Formed but not active; irrigation agency not sure of the benefit and functions of the commission

Document completed, but not updated or used anymore

S. Sulawesi

9. Bone New software and simplified reporting not utilized; reverted to old system

Formed, but has convened infrequently after PISP closing

Document completed, but not updated or used anymore

PISP = Participatory Irrigation Sector Project, SI-PAI = Sistim Informasi Pengelolaan Aset Irigasi (irrigation asset management information system). Source: IED

44 Appendix 4

8. Preparation of the district irrigation development and management plan. The RP2I document was introduced and promoted under the PISP scheme to improve irrigation sector planning over the medium term (5-year rolling), particularly to strengthen needs analysis and identify gaps in investment and operation and maintenance (O&M). By project completion, 26 of the 27 PISP districts were reported to have completed preparing first versions of the RP2I that were to be updated annually or periodically. The documents were prepared with varying levels of support from the project consultants and varying involvement by district staff. Tulung Agung District (East Java) and Bone District (South Sulawesi) reported that their staff involvement was constrained by the limited number of agency staff and the urgent need of the project to complete the RP2Is; as such the RP2I drafting was largely led by the consultants. The evaluation interviews found that the RP2I preparations have not been mainstreamed into routine district activities after the project completion. Aside from a lack of staff time and fund allocation, the main reasons for the discontinuation of RP2I updating was that the document format was considered too complicated and long with unclear added value compared to that of the existing planning tools. Districts have devoted more funds and attention to the development of irrigation databases, including infrastructure inventory status. There has been a continuing interest to make documents and data more useful to facilitate discussions to garner more support for irrigation in annual district budget discussions and the investment prioritization process.

3. Staff Retention

9. One key human resources challenge in irrigation has been high staff turnover and a decline in the number of field workers, mainly irrigation field officers. The provincial government of East Java and the government of Bone District (South Sulawesi), for example, reported that young and adequately skilled staff members are being recruited to replace retiring staff. However, newly hired staff are often transferred to non-agriculture sectors. The decline in youth interest in agriculture in Indonesia poses a challenge over the medium and long term. As a result, alternative arrangements must be considered, such as engagement of honorarium staff and the delegation of public service functions to beneficiaries including farmers associations and WUAs and/or water users association federations (WUAFs).

B. Agricultural Extension

10. The institutional arrangement to deliver agricultural extension in Indonesia involves collaboration between the technical line units and a central extension unit, namely the Agency for Agricultural Extension and Human Resources Development, under the Ministry of Agriculture.5 Extension activities under the PISP involved a collaboration between the Directorate General of Agricultural Infrastructure and Facilities—the main PISP counterpart in the Ministry of Agriculture—and the Agency for Agricultural Extension and Human Resources Development. PISP budgets for extension support were channeled through to the Directorate General of Agricultural Infrastructure and Facilities, and this arrangement was replicated down to the subnational levels. With this arrangement, the staff training and extension module development supported by the PISP had been tailored more to support the work program of the Directorate General of Agricultural Infrastructure and Facilities (e.g., on-farm irrigation, tertiary canal maintenance, and WUA formation and support). Other topics, such as the introduction of high-yielding varieties, income enhancement, alternative livelihood, and cooperative development were also supported by the PISP. However, PISP only supported the dissemination of these modules that had existed before the PISP. 11. Agricultural extension continues to face challenges after PISP completion. The number of extension officers was reported to be inadequate. In some districts visited by the evaluation team the

5 Badan Penyuluhan dan Pengembangan Sumber Daya Manusia Pertanian or BPPSDMP. At the district level, the extension

agencies have different names. In Banyumas District (Central Java), it is called the Agency for Agricultural Extension; in Malang District (East Java), it is called the Agency for Food Security and Extension; and in Bone District (South Sulawesi), it is called the Agency for Agriculture, Fishery, and Forestry Extension.


average was one extension officer per two to three villages and they have secondary education level qualifications. There is an ongoing difficulty to replace retiring officers. Civil service staff are being supplemented increasingly by honorarium extension staff, who are not permanent staff. As a result, the envisaged role of extension officers to coordinate dialogue and training (including facilitating visits by expert resource persons) could not be exercised well because of the limited knowledge of the extension workers in technical subjects and in learning facilitation. Extension workers’ routine activities are often occupied by data gathering required by the government (farmers’ aspirations, crop outputs, and inputs use). 12. There has been an ongoing effort to improve the skills of extension workers through a certification program. Feedback received during the evaluation pointed to generally inadequate incentives for existing workers to be certified. Certified staff do not receive additional rewards or compensation, and most extension officers therefore still remain uncertified.

C. Capacity Building of Water Users Associations

13. Institution building in the farmers’ organization in the form of water users associations is critical to sustain the irrigation subsector. WUAs were strengthened through support of their formation as legal entities, seminar and hands-on training, and provision of facilities and equipment to improve WUA administration. As formal institutions, WUAs can cooperate better with the district agencies in implementing work- and cost-sharing agreements, and in executing work contracts. 14. A key PISP feature that helped empowers the WUAs and WUAFs was the priority given to the WUAFs to be contractors in the rehabilitation of irrigation networks for the primary and secondary canal facilities. Two mechanisms were allowed: (i) direct contracting for light rehabilitation works with contract values of less than Rp100 million, and (ii) operational cooperation or subcontracting with commercial contractors on more complex rehabilitation works (valued at more than Rp100 million). During the project, about 2,142 contracts were signed with the WUAFs for light rehabilitation, benefiting 314,777 ha. 15. Feedback from the district governments was very positive on the work of the WUAFs (Table A4.2). The arrangement was more cost effective and produced better quality structures than the average commercial contracts. Site visits to the rehabilitated structures found all of them are still functioning, although there have been some deterioration because a lack of maintenance. 16. However, the involvement of WUAFs in construction work was confined to the PISP implementation period. After the PISP ended, WUAFs’ direct engagement was discontinued on the grounds of (i) concerns over the legality of direct (untendered) contracts being awarded to the WUAFs, which was considered as a special arrangement under the PISP; and (ii) the presumed lack of capability of WUAFs to meet the technical qualification requirements to take part in bidding (e.g., availability of qualified staff, equipment, and financial assets as well as different business permits). Following the project, farmers’ involvement in the rehabilitation work of the public irrigation assets (primary and secondary canals) has reverted back to an individual arrangement rather than through engagement of WUAFs as organizations. This situation has stifled WUAF development and contradicts the government’s intent to (i) make WUAFs self-reliant financially, and (ii) reduce their dependence on government handouts. Following the project completion, WUAs and WUAFs continue to coordinate management of tertiary canals. This task has been aided by social assistance grants provided by the government (Section D). However, in receiving the social assistance, the WUAs and WUAFs become dependent on the governments (central and local) which have made them less independent and critical of the government policies and service quality.

46 Appendix 4

Table A4.2: Local Government Views on the Role of Water Users Associations and Water Users Associations Federations in Participatory Irrigation Sector Project Rehabilitation

No. District Feedback Banten 1 Pandeglang Capability of WUAs and WUAFs to rehabilitate irrigation networks was higher

than that of contractors 2 Lebak Contracts with WUAs and WUAFs for irrigation rehabilitation were cheaper

than those with commercial contractors West Java 3 Cirebon Almost all the irrigation networks construction work carried out by WUAs and

WUAFs produced satisfactory results 4 Kuningan The quality of work done by WUAs and WUAFs to rehabilitate irrigation

networks during the PISP was satisfactory Central Java 5 Banyumas The quality of rehabilitation works by WUAs during the PISP was meeting

standards 6

Cilacap Besides the quality of structures being satisfactory, the completed canals were longer than what was planned and agreed in the contract

East Java 7 Tulung Agung The quality of the work done by the WUAFs was good. However, further

involving the WUAFs in irrigation construction using purely domestic budgets may require a regulatory change because WUAFs are not construction service companies

South Sulawesi 8 Bone If WUAFs are to be used to carry out construction work going forth, there

needs to be policy clarification from the central government so district staff will not be held legally accountable for misprocurement due to direct contracting

9

Malang The WUAs have the capability to carry out construction work, but, administratively, they are not allowed to do that work

PISP = Participatory Irrigation Sector Project, WUA = water users association, WUAF = water users association federation. Source: IED

D. Funding of Irrigation

17. Per National Regulation No. 20 of 2006, the division of responsibilities among governments for irrigation management is segregated into three levels: (i) the district governments are responsible for irrigation schemes (i.e., primary and secondary canals and structures) with command areas smaller than 1,000 ha, (ii) the provincial governments are responsible for irrigation schemes that range from 1,000 ha to 3,000 ha in size or that cross district boundaries, and (iii) the national government is responsible for schemes that are larger than 3,000 ha or that cross provincial boundaries.

1. District-Level Funding (Primary and Secondary Canals) 18. As discussed in Chapter 3 (paras. 43–46), O&M funding has generally been adequate for schemes under the responsibility of the national and provincial governments, although there is a need for the provinces to improve accuracy of their budget and spending data. In contrast, O&M funding for the district schemes have generally been found to be inadequate. Further, it is interesting to note that in three of the nine districts visited during the evaluation (Lebak, Kuningan, and Banyumas), there was a year-on-year decline in the nominal O&M budgets in 2015 (Table A4.3). Evaluation of the performance of the O&M and rehabilitation of irrigation networks conducted by the Center for the Agricultural and Socioeconomic and Policy Studies at the Agency for Agricultural Research and Development of the Ministry of Agriculture confirmed that the decline in irrigation system performance


was attributed mainly to the physical deterioration of irrigation infrastructure which, in turn, was attributed to low O&M performance.6

Table A4.3: Irrigation Operation & Maintenance in Selected Districts

No. Province/District O&M Funds and Spending Banten 1 Pandeglang In 2015, Rp1 billion was allocated, which was used for O&M in 20 schemes

(Rp250,000 per hectare). 2 Lebak In 2014, Rp525 million was allocated to 175 schemes, while in 2015, Rp480 million

was allocated to 160 schemes. West Java 3 Cirebon In 2015, Rp6 billion in O&M funds was allocated. 4 Kuningan In 2014, O&M funds totaled Rp800 million, and in 2015, they were reduced to

Rp680 million. They were used for the cleaning of canals and honoraria of 23 workers.

Central Java 5 Banyumas In 2014, the amount allocated for irrigation O&M was Rp700 million, and in 2015,

it dropped to Rp500 million. These funds were used partly for damage repairs which may be classified as capital spending.

6 Cilacap Funds for irrigation O&M amounted to Rp3.5 billion for 2015, which were allocated for the cleaning of irrigation networks, and partly for the rehabilitation of canals.

East Java 7 Tulung Agung Funds for irrigation in the district’s budget is about Rp800 million in 2015, which

was for irrigation O&M and light rehabilitation, including for irrigation commissions and the irrigation asset management information system.

8 Malang In the past 3 years, the district government allocated budget for irrigation O&M that amounted to Rp1.5 billion–Rp2.0 billion, depending on the assessed condition of the irrigation networks. The funds were used for maintenance work and periodic rehabilitation.

South Sulawesi 9 Bone The amount of funds allocated from the district’s annual revenue and expenditure

plan for irrigation O&M was about Rp500 million per year in the recent years, a relatively small amount.

O&M = operation and maintenance. Source: IED interviews

19. With the clearer demarcation of responsibility, the WUAFs’ willingness to contribute to the maintenance of the government’s irrigation facilities may have declined. Spot checks in Tulung Agung District (East Java) and Bone Distict (South Sulawesi) showed that farmers are not active in clearing overgrown vegetation and sedimentation of PISP-rehabilitated primary and secondary canals. Farmers become involved only if they are paid for their labor. 20. The Government of Indonesia and ADB are planning to continue PISP work with a follow-on project, and a key issue to determine is how to improve project fund transfer to the provinces and districts. During the PISP, the national government had more authority to manage PISP funds, as they were entirely part of the national budget. The district governments submitted proposals against these budgets. As there has been progress in decentralization, more budgetary control is being given to the district and provincial governments. Budgets allocated to irrigation (i.e., funds transferred from the national to local budgets and earmarked for irrigation) must also be better monitored to ensure they are properly used following the agreed criteria and conditions.

6 Government of Indonesia, Agency for Agricultural Research and Development. 2006. Evaluation of the Operational

Performance and the Maintenance of Irrigation Networks and Rehabilitation Efforts. Jakarta.

48 Appendix 4

2. Water Users Associations and Tertiary Canal Funding 21. As tertiary canals were designated as assets to be managed by the WUAs, their main funding comes from farmers’ dues. However, these dues have been difficult to gather. Their collection is commonly done only once a year and not all farmers pay. Table A4.4 summarizes the results of interviews at the districts with agency staff and the WUAs on dues collection. There has been a substantial variation in WUAs’ success in dues collection, and most districts have collection rates of below 50%.

22. Dues and contributions are more easily collected in-kind (in the form of unhusked rice and labor) rather than in cash. Efforts and programs to make WUAFs undertake other business ventures (e.g., rice milling and seed production) have been difficult to sustain, as these have added complications to WUAF operations. Furthermore, these business development efforts have not been consistently supported by line technical agencies for public works, water resource management, and agriculture. The most promising additional sources of income for WUAFs are to become contractors or subcontractors to commercial contractors for rehabilitation and maintenance works funded by the national and local governments (for the primary and secondary canals). These arrangements need to be continued and expanded.

Table A4.4: Farmers’ Payment of Irrigation Dues

No. Province/District Irrigation Dues Collection and Payment

Banten 1. Pandeglang Dues payment is collected only in a small number of WUAs and payment is made

once a year. Historically, farmers have paid their when they need rehabilitation work done and there are no other sources of financing.

2 Lebak Only about 20% of farmers paid for water with dues, and there was no specific penalty for not paying dues. Therefore, the funds gathered by WUAs are very small. Farmers indicated that they are willing to be self-reliant in rehabilitating damaged networks and are prepared to provide labor and in-kind materials.

West Java 3 Cirebon Dues payment is made usually in the dry season.

Farmers who are rice field owners paid their dues every year on average, while only a small proportion of tenant farmers pay their dues.

Dues are directly paid through the village office and such payment is related to the payment of land and building taxes.

4 Kuningan More than 50% of farmers pay their dues every year in the form of 20 kilograms of unhusked rice, and dues are used to pay operation and maintenance workers and honoraria of WUA personnel.

Central Java 5 Banyumas Dues payment is collected only in certain locations and only in about 10% of total

WUAs; in those WUAs, however, most farmers pay their dues. 6 Cilacap In more than 45% of WUAs, members pay irrigation dues, especially in the dry

season, and payment is made in two forms: unhusked rice and money. East Java 7 Tulung Agung Dues are paid only in the dry season, mostly in the form of unhusked rice; it is

estimated that fewer than 50% of farmers pay. 8 Malang It is difficult to gather dues; farmers still view water as a free public good and a

responsibility of the government to provide. South Sulawesi

9 Bone About 60% of WUA members pay dues.

WUA = water users association. Source: IED interviews.


23. Social Assistance. Government assistance for the tertiary (on-farm) canal maintenance has come mostly as a part of various social assistance programs for the poor rural communities, and it can be funded by the central or subnational budgets and provided as grants directly to the WUAFs, WUAs or other associations. These grants are justified as subsidies to the poor or as one-time assistance for communities cope with costly damages to their tertiary canal system. The social assistance mechanism is prone to fund misuse and corruption (on the part of district officials, village heads, and WUA officers), and its challenges are weak monitoring of fund utilization and verification of work quality. Resources allocated for monitoring and audit have been limited.

E. Project Management

24. The following are additional observations relating to the PISP management: i. The work to enhance irrigation planning took place at the same time as the civil works

rehabilitation during 2008–2010. This was not ideal since the civil works did not fully benefit from the improved sector assessment and plan prepared under the project.

ii. In many cases, the bottom-up requests submitted by communities and WUAs—as outcomes of participatory discussions--were too large or expensive for the available PISP budget per scheme (e.g., rehabilitation of a dams or large structures). This caused disappointments and undermined the participatory consultation process. Attention should be given in future projects to better guide community consultations with clearer parameter limits on issue and activity scope to be discussed.

iii. The exact plan to empower WUAs was frequently changed for various reasons during PISP implementation. This influenced and delayed the bottom-up drafting of the PISP’s annual work plan.

iv. Project reports that had to be made by the district agencies were reported to be too complicated and numerous and thus beyond the capacities of local staff to prepare.

v. Loan and grant disbursements involved many steps and they are time consuming. As an example, more than 5 months were required to process a WUAF contract worth below Rp100 million. Disbursement methods need to be re-examined and streamlined.

vi. District agricultural staff suggested that their offices could have done more under the PISP beyond irrigation to help increase agricultural outputs.

vii. Most of the project consultants were based in Jakarta while only a few were based in the provinces and districts. Staff suggested that more consultants should have been placed outside Jakarta during project implementation.

viii. Project staff in the district was chosen by the national ministries in some districts. As a result the district implementation units developed close direct links with the ministries in Jakarta, which controlled PISP budgets. This arrangement caused a lack of involvement of the district’s line agencies and leadership in the day to day running of the project.

ix. The PISP did not prepare an exit strategy, thus making it difficult for the district governments to propose routine budget requests to continue the PISP activities.

APPENDIX 5: ECONOMIC ASSESSMENT

A. Introduction

1. The economic analysis for this evaluation centers on the updating of the project’s economic internal rates of return (EIRRs), which contribute to the assessment of how efficiently the Participatory Irrigation Sector Project (PISP) utilized its resources. This updating involves a reestimation of project costs and benefits and discussions on relevant factors that affect these costs and benefits.

B. Sample Subprojects

2. The EIRR analysis was done on 15 sample irrigation schemes or subprojects, which are the same as those selected during the project completion review in 2013–2014.1 These were selected from the 1,614 total schemes completed under the PISP, with a total irrigation command area of 457,220 hectares (ha). Of the total, 468 schemes (covering 315,000 ha) that had benefited from only light rehabilitation (i.e., sustainable replacement of rehabilitation) were excluded from target sampling. They were excluded because it was difficult to credibly attribute observed improvements in crop yields with the low-cost work carried out (less than $80 per ha). The initial EIRR calculations showed these schemes to generate very high EIRRs (above 50% per annum); there was therefore a concern that these would diminish the accuracy of the project’s EIRR estimate. 3. Excluding the sustainable replacement of rehabilitation schemes, there were still 1,145 schemes (71% of the total) that benefited from full rehabilitation works covering a command area of about 141,700 ha. These are mostly small schemes under the responsibility of the districts, which were the main targets of the PISP. The evaluation found the sampling process of the project completion report (PCR) to be reasonable for the purpose of economic analysis. In each of the PISP provinces (Lampung, Banten, West Java, Central Java, East Java, and South Sulawesi), a district was selected (Table A5.1). In each selected district, two or three sample schemes were selected.2 Screening was done primarily based on the irrigation scheme profile data compiled by the project offices to capture the typical range of water resource scarcity, topography, and cropping practices within the PISP area. The EIRR samples covered six districts of the total 27 (in six provinces). 4. As discussed in Section C, farm system data and economic benefits were estimated from a survey of 630 farm households done in 2013 in the representative irrigation command areas and districts. These were supplemented with more in-depth data from a farm household survey conducted in 2015 in conjunction with this evaluation, which covered 900 households in eight PISP districts (see Appendix 2), seven districts under the Water Resources and Irrigation Sector Management (WISM) Phase 1 Program, and one non-project districts (para. 9).

C. Subproject Costs

5. Investment costs. The main investment cost for the schemes was the civil work rehabilitation cost, which included cofinancing from the Government of Indonesia, the Asian Development Bank, and the Government of the Netherlands, as well as beneficiary contributions. Government administrative overhead costs, taxes, custom duties and interest costs were excluded. Data for the investment costs were from the completed civil works contract records. This evaluation improves the estimates made at project completion by including the estimated values of farmer labor contributed to construction, which were not recorded in the PCR analysis. Similarly, non-civil work investment costs (e.g., farmer

1 ADB. 2014. Project Completion Report. Participatory Irrigation Sector Project (Indonesia) (Loans 2064-INO. Manila.

2 The sample of 15 schemes was finalized after excluding some schemes that reported outlier data.

Economic Assessment 51

training, water users association development, agricultural support services, and direct project management) were estimated more accurately at 95% of the civil works expenditures. These were estimated from the final PISP investment data in Appendix 7 of the PCR.3 Rehabilitation was completed within a 1-year period, so rehabilitation costs for each scheme are confined to 1 year for the purpose of EIRR calculation. Table A5.1 presents the estimated economic investment and routine operation and maintenance (O&M) costs.

Table A5.1: Economic Investment and Operation and Maintenance Costs (Million Rp per ha, 2013 prices)

Province (District)a

Irrigation Scheme (Size)

Total Investment Costsb

Total annual O&M cost

Lampung (South Lampung)

Way Pisang 1 (972 ha) Way Made (674 ha)

5.61 0.32

Banten (Lebak)

Cipeuncang Pare (620 ha) Cisiih (850 ha)

3.86 0.32

West Java (Garut)

Baranangsiang (882 ha) Cirompang (847 ha) Ciroyom (762 ha)

5.97 0.43

Central Java (Banyumas)

Kediri (731 ha) Pandak Raden (463 ha) Kaliomas (331 ha)

3.67 0.32

East Java (Madiun)

Sewu (1,332 ha) Sareng 866 ha)

3.94 0.32

South Sulawesi (Sinjai)

Balakia I & II (851 ha) Arango I (518 ha) Lompoa (500 ha)

5.68 0.32

Overall 4.79 0.34 O&M = operation and maintenance. a To simplify presentation, the table is broken down by district instead of scheme. b For civil works, it was estimated that 75% of the total cost was in local currency and 25% was in foreign currency. Local currency costs were adjusted using the standard conversion factor (0.9 for Indonesia). Source: PCR Economic Analysis Data.

6. The following are general approaches used in the costs and benefits valuation: (i) the local currency share of project costs is adjusted by a standard conversion factor, which is 0.9 for Indonesia; (ii) all cost and revenue streams are in constant 2013 prices in rupiah; and (iii) the EIRRs are calculated over a 20-year period. 7. The O&M cost requirement was estimated at Rp200,000 per ha per annum, which was assessed as adequate for 2013. This was for the primary and secondary canal facilities that were under the responsibility of the national and subnational governments. O&M costs for the tertiary (on-farm) canals were estimated at Rp125,000 per ha, mainly comprising labor costs. Some schemes in West Java Province required higher O&M resources.

3 Direct civil works costs (including farmer’s contributions) totaled $57.7 million, while other costs plus overhead totaled $55.0

million. These brought the total investment to $112.7 million (Footnote 1, Appendix 7).

52 Appendix 5

D. Subproject Benefits

8. Economic benefits of the PISP were derived from the estimation of the increase in crop incomes (revenues less expenditures) as a result of improved irrigation and extension services provided under the project. This estimation took account of changes in the cropping areas, yields per hectare, and input quantities, as well as economic prices of concerned farm outputs and inputs. 9. Farming systems data (input and output quantities) were gathered from farmers through two surveys. One was done at project completion in 2013 and the other in conjunction with this evaluation in 2015 (para. 4).4 There were other sources considered for comparison: (i) the schemes’ socioeconomic and technical profiles prepared before the rehabilitation work during 2006–2009, and (ii) routinely collected district and subdistrict data on farm production. The 2013 survey covered 15 sample subprojects in six districts (Table A5.1),5 while the 2015 survey covered 22 subprojects in eight PISP districts (Appendix 2).6 The two surveys overlapped in four districts.7 10. Cropping areas. In the PISP provinces, annual crops can be planted up to three times a year. These are in the wet season, which lasts from November to March, the first dry season (DS1) from April to July, and the second dry season (DS2) from August to October. The 2013 survey (during the PCR) found modest differences in cropping intensity or the proportions of the irrigation command areas planted with rice and secondary annual crops in DS1 and DS2 (Table A5.2).8 These differences were for before- and after-project comparisons based on farmers’ recollections at project closing. A significant share of available land was planted with rice during the wet season and DS1 before the project in 2005.9 During DS2, secondary annual crops are more commonly grown, especially in Java.10 Complementing these data, the 2015 survey (done as a part of the evaluation) found larger differences in the cropping intensity for rice in DS1 and DS2 between the PISP and comparable control irrigation schemes (Table A5.2). Based on the data review and field discussions, the evaluation considers that the PCR underestimated the cropping intensity benefits of the project by comparing the before- and after-project data. Combining the available survey data, the evaluation considers that the PISP rice areas at the start of the project would have diminished during the project period, especially during both dry seasons, had the PISP repairs not taken place. These decreases would be comparable, therefore, to the differences in cropping intensities observed in 2015 between the PISP and non-PISP (control) schemes.11

4 The 2015 survey was part of a study financed by the Government of the Netherlands, which cofinanced the PISP, and was

undertaken by the Policy and Operations Evaluation Department of the Government of the Netherlands’ Ministry of Foreign Affairs. The survey was a part of an evaluation of the Government of the Netherlands’ support for the water sector worldwide.

5 This evaluation carried out a close review of the 2013 PCR data, including a data clean up prior to utilization of these data. 6 The 2015 survey covered 22 PISP schemes (in eight districts), 17 schemes (in seven districts) under the World Bank’s Water

Resources and Irrigation Sector Management Program phase 1, and 15 control schemes not supported by any externally-funded projects. Fifteen households were interviewed per scheme.

7 Lebak (in Banten Province), Banyumas (in Central Java Province), Madiun (in East Java Province), and Sinjai (in South Sulawesi Province).

8 Cropping intensity indicates the ratio between the harvested area and the total farm area, and is usually expressed as a percentage. This can be for a single crop or combination of crops and for one or more growing seasons. For a PISP scheme (which has a fixed potential irrigated or command area before and after rehabilitation work), the maximum cropping intensity in one growing season is 100%, and for the three growing seasons per year, the maximum cropping intensity is 300%.

9 Although high by Asian standards, double cropping of rice is common in Indonesia. The Food and Agriculture Organization of the United Nations reported the overall cropping intensity of rice on Indonesia’s irrigated land to be 195% in 2005. Food and Agriculture Organization of the United Nations. 2012. Indonesia irrigated crop calendar. http://www.fao.org/nr/aquastat (accessed 3 August 2015).

10 These are called palawija in Indonesia. The 2013 household data survey showed that 61% of the household respondents planted maize, 17% planted string beans, 10% planted peanuts, and 6% planted soybeans. The remaining households planted other legumes and vegetables.

11 The 2015 copping intensity and yield gains were estimated using linear regressions and propensity score matching, thus correcting for scheme characteristics not attributable to the PISP. Analysis is ongoing on the 2015 household data and more refined figures will be provided in the IOB impact report.


11. Secondary annual crops (maize, soybean, peanuts, other legumes, and vegetables) are usually planted together (intercropped) at the same time and on the same plots. As such, their cropping area data are reported in an aggregated manner, instead of by individual crop. Per the 2013 survey, secondary crops were grown mostly in DS2 and there was a modest increase in cropping intensity when comparing the before- and after-project scenarios (from 15% to 20%). As for the 2015 survey, the data on secondary crop areas are still not finalized, and there is a lack of corresponding yield and inputs data (para. 15). Therefore, the EIRR calculation utilizes mainly the 2013 data to estimate the secondary crop benefits, where the cropping intensity is assumed to remain stable at 15% in the before- and without-project scenarios during DS2, and to rise to 20% in the with-project scenario (Table A5.2).12 There are year-to-year variations in secondary crop areas and yields depending on variation in rainfall and irrigation supply. Even when fields are not fully irrigated in DS2, improved irrigation during the wet season and DS1 provides more residual soil moisture for these crops.

Table A5.2: Change in Rice Cropping Intensity

(% of irrigated land)

Season (crop) Before Project

Without Project

With Project

Difference Between

With- and Without-Project

Scenarios

2013 Survey

Wet season (rice) 100% 100% DS1 (rice) 92% 97% DS2 (rice) 22% 29% DS2 (secondary crops) 15% 20%

2015 Survey Wet season (rice) 97% 97% 0%

DS1 (rice) 70% 100% 30% DS2 (rice) 24% 36% 12%

EIRR Wet season (rice) 100% 100% 100% 0%

DS1 (rice) 92% 67% 97% 30%

DS2 (rice) 22% 17% 29% 12% DS2 (secondary crops) 15% 15% 20% 5%

DS1 = first dry season, DS2 = second dry season, EIRR = economic internal rate of return. Sources: Surveys as part the project completion report preparation and project evaluation.

12. Data breakdown by district and province in cropping intensity is less reliable due to the generally small number of sample observations. The schemes in South Lampung District suffered more from water shortage, so improved irrigation increased rice areas during DS1. Rice areas expanded in DS2 in South Lampung, but most land is still left fallow in DS2. In West Java Province (Garut District), the project allowed triple cropping of rice, while in Central Java Province (Banyumas District), DS2 rice and secondary crop areas both increased. There was no major change in cropping intensity in Banten Province (Lebak District), East Java Province (Madiun District), and South Sulawesi Province (Sinjai District). Data gathered in 2013 were also stratified by farm location in the irrigation system (at the head, middle, and tail). The analysis of the survey data showed there was no clear pattern of change in cropping intensity gains by location. Cropping intensity increased most in the upper reaches (head) of the Lampung and West Java systems, but in Central Java most of the increase was observed at the tail end.

12 The 2015 survey indicates that in the control (without-project) schemes, during DS1 there was a higher proportion of land

(higher cropping intensity) growing secondary crops as compared to that in the with-project schemes. This plausible because secondary crops were replaced by rice as water supply became more predictable due to the project. This reduction in secondary crop area during DS1 is not fully captured in the EIRR benefit estimation due to paucity of data, but is addressed in the sensitivity analysis (i.e., by assuming a smaller increase in area growing secondary crops).

54 Appendix 5

13. Yields and crop inputs. Data from the 2013 survey showed the average rice yields rose by 18%–19% (from around 4 tons/ha to 5 tons/ha) during the project across the three growing seasons (Table A5.3). Based on the evaluation’s data examination and discussions with staff and consultants, the evaluation found these results problematic since these yield increases were not accompanied in the recorded data by substantial increases in crop inputs (seeds, fertilizers, labor, and equipment services) (para. 14). Moreover, the baselines (estimated from farmers’ recollection of yields in 2006–2007), which ranged from 5.0 tons to 5.5 tons,13 were rather high to increase by another 1.0 ton/ha. As a comparison, the appraisal report in the report and recommendation of the President expected an average increase of just 0.25–0.30 tons per ha in the wet season and 0.50–0.65 tons per ha in the dry season, from an average baseline of 3.5 tons/ha. Furthermore, subdistrict statistics gathered separately (covering both PISP and non-PISP schemes) recorded an increase of 0.25 tons/ha in 2007 to 0.38 tons/ha in 2014. In contrast, the 2015 survey found an average difference of 0.4 tons in the wet season (from 4.4 tons/ha to 4.8 tons/ha) and an 11% increase and no significant yield increase in DS1 and DS2. These were more consistent with the input increases and other yield data. The 2015 survey approach, which compares control and with-project locations, adjusts incremental benefits for trends that occurred either with or without the project (e.g., adoption of high-yielding varieties, improved transport, and changes to cropping techniques).

Table A5.3: Rice Yields and Inputs (tons/ha)

Season Without Project

With Project

Yield Gain

Changes in Inputsa

2013 Survey Wet season

5,425 6,418 18% Seed (-17%) & use of high-yielding varieties, pesticides & herbicides (+18%)

DS1 5,049 6,023 19% (similar to above)

DS2 5,490 6,481 18% (similar to above) 2015 Survey Wet

season 4,352 4,824 11% Lesser seed, use of high-

yielding varieties. Modest increase in inorganic fertilizers

DS1 4,123 4,123 0% No changeb DS2 4,224 4,224 0% No changeb

Baseline for Economic Analysis

Wet season

5,425 6,040 11% Seed (-17%) & use of high-yielding varieties, pesticides & herbicides (+11%)

DS1 5,049 5,049 0% No change DS2 5,490 5,490 0% No change

DS1 = first dry season, DS2 = second dry season, ha = hectare. a Excluding improved irrigation and drainage. b The regression actually showed different figures for the with- and without-project scenarios, but these differences are not statistically significant. Source: Surveys as part the project completion report preparation and project evaluation.

14. Examining the rice cultivation inputs, there was higher use of manure fertilizers before and after the project, and a modest increase in the use of pesticides and herbicides (from 1.8 kilograms/ha to 2.2 kilograms/ha). The average amount of seeds used fell, presumably due to the adoption of high-yielding varieties. However, other key inputs (inorganic fertilizers, labor, and mechanized tractor utilization) remained largely the same. The input changes were too low to have brought about

13 This was in the form of dried rolled unhusked rice, (gabah giling kering), which is consistent with other yield data. The 2013

household survey had limitations; it was conducted in early 2013 and asked farmers about before- and after-project conditions, which required a long recall period of 4–7 years.


significant yield increases.14 The field interviews and data from the 2015 household survey suggest that adoption of high-yielding varieties and modest changes in fertilizer and chemical applications took place as well in the without-project (control) districts. This partly explains the lack of difference in rice yields between the with- and without-project scenarios in the 2015 data. Another trend shared between control and with-project districts was the adoption of the System of Rice Intensification (SRI) technique.15 Given the aforementioned considerations, the evaluation considers the yield gains estimated in 2015 more appropriate to adopt as a base case for the economic analysis (Table A5.3).

15. As for the secondary annual crops, the 2013 survey gathered yields data in terms of aggregate sales (i.e., in rupiah) per ha rather than individual crop yields. This is because of the intercropping or mixed crop planting in one location at a time (para. 11). Revenues from secondary crops per ha rose 7.0% in constant terms over the course of the project and accounted for about 26% of the total crop revenues in the sample farms in 2013. Because of the more complex crop and input combinations for secondary crops, the evaluation refrains from triangulating yield and inputs information from different sources and relies mostly on the original PCR’s 2013 survey data for estimating economic returns. 16. Farm prices. This evaluation utilized updated world price data for the economic analysis. For rice, long-run price projections of the World Bank were used. The long-term average forecast prices were lower than the average prices for the 2008–2012 period previously used in the PCR’s economic analysis. The on-farm rice price was lower by 29%. Regarding farm inputs, fertilizer prices were adjusted to eliminate the effect of subsidies. Inorganic fertilizer prices have been heavily subsidized in Indonesia. No adjustments were made for other tradable agricultural inputs (pesticides and herbicides) since their contributions to the total production costs are small. Non-tradable input costs were valued at the local market prices prevailing in 2013. The main non-tradable input is labor.16 Table A5.4 shows the economic and financial prices comparison for key tradable outputs and inputs used in the analysis.

17. Economic benefits. Based on the analyses in paras. 10–15, the project’s incremental benefits are mainly derived from (i) expansion in the rice and secondary annual crop areas in DS1 and DS2, and (ii) increase in paddy yields in the wet season. These benefits were attributed to more reliable irrigation and improved drainage and, to a lesser extent, changes in inputs used and technologies.17 In estimating the EIRRs, the following assumptions were made: (i) since rehabilitation works were done fairly quickly within 1 year, it was assumed that there was no significant disruption to crop production because of the civil works; and (ii) the build-up of crop production and incremental benefits was estimated to take 3 years from the completion of rehabilitation (which was the period needed for farmers to adapt to more reliable irrigation water availability and other project interventions).

14 The 2013 survey reported a rise in the use of high-yielding varieties from 73% to 97%. The adoption of high-yielding varieties

in West Java should not suggest that the farmers sampled were less progressive prior to the PISP. Many of the traditional seed varieties produced in West Java command higher market prices due to their superior eating quality, although their yield potentials are lower than those of the newer high-yielding varieties.

15 The System of Rice Intensification technique can raise paddy yields and conserve water, which makes it particularly well suited for dry-season cultivation. Compared with normal rice cultivation, System of Rice Intensification involves planting single seedlings instead of multiple seedlings in a clump, and not flooding the paddy fields during the rice vegetative growth stage. Planting is in a square pattern with more space, rather than in rows or at random, and is followed by more intensive weed control. The rice field should be kept moist rather than continuously saturated with water in order to improve root growth and support the growth of aerobic soil organisms. The wider spacing of rice plants enables more vigorous growth of roots and canopy. Higher yield is achieved with 80%–90% reductions in seed requirements and 25%–50% less irrigation water, but the method can be more intensive in its demand for labor or mechanical inputs. Fertilizers and pesticides can be used.

16 Studies of rural Asia have traditionally imputed an economic value of labor below the market rate, assuming that there is surplus labor that would otherwise be unemployed. The PCR valued labor at 85.5% of the market wage. However, recent evidence from Indonesia and other Asian countries indicates that real rural wages have risen significantly over the past decade, reflecting increasingly integrated national labor markets and decline of labor participation in agriculture. For this reason, the economic value of labor, including family labor, is set at the market wage rates of the 2012 crop year.

17 There was a decline in the frequency of reported drainage problems during the wet season (from 60% to 2% overall) in the provinces of South Sulawesi, West Java, and Lampung.

56 Appendix 5

Table A5.4: Key Economic and Financial Prices (Rp per kilogram)

Commodity Economic

(a) Financial

(b) (a)/(b)

Paddy/rice 2,265 3,787 60%

Maize 2,410 1,963 123%

Urea 4,403 1,926 229%

Triple super phosphate 4,588 2,323 198%

Potassium chloride 4,565 2,035 224%

Diammonium phosphate 5,605 3,156 178% Source: PISP project completion report

E. Economic Returns

18. The updated EIRR for the PISP is estimated at 24% (Table A5.5), above the threshold cost of capital of 12%, which suggests the project is likely to provide a robust economic return on its investment and O&M expenditures. The full economic analysis model is provided as Supplementary Appendix A. This is compared to the EIRR of 29% estimated at appraisal and the EIRR of 54% estimated at project completion. The base case EIRR is based on conservative assumptions, particularly in the estimated yield gains (paras. 13–15). The EIRR estimates vary widely by district, ranging from 7% to 52%. The variation is due to differences in yields and inputs across locations, but also because of data biases (larger inconsistencies) when data are isolated in smaller samples. Crop benefits are more accurately assessed on an aggregate basis capturing data from all the sample schemes. The sensitivity analysis shows that the economic efficiency of the PISP is robust generally. However, the project EIRR is susceptible to unfavorable changes in rice yield and a drop in rice price and lower revenues of secondary annual crops, as well as combination of these adverse circumstances (Table A5.5).

Table A5.5: Economic Internal Rates of Return and Sensitivity Analyses

EIRR Baseline 24%

a. No change in rice yield (-11% for wet season yield), no change in technology and inputs

11%

b. Secondary crops area -10% 21% c. Rice price -10% 15%

d. Secondary crops revenue/ha -10% 19% e. Fertilizer prices +10% 22% f. Labor prices cost +10% 20%

EIRR = economic internal rate of return, ha = hectare. Source: IED estimates.

19. Adequate operation and maintenance. Sustaining an adequate level of O&M financing will likely determine whether the PISP subprojects can deliver their projected benefits over the 20-year time frame used to calculate the EIRRs. Adequacy of O&M is a particular concern for the small schemes (smaller than 1,000 ha), since a substantial share of the O&M financing is to be provided by the districts. O&M funding for the national irrigation schemes (larger than 3,000 ha) was adequate on average at Rp180,000 ($19) per ha per year in 2012 and about Rp250,000 ($21) per ha per year in 2014. The provincial schemes (ranging in size from 1,000 ha to 3,000 ha) are likely to benefit from the same level of funding. However, O&M funding has not been adequate for the district schemes (Chapter 3, paras. 43–46). Of the 15 schemes in this EIRR analysis, 14 are below 1,000 ha in size and are therefore under the management of the districts.

Documents

Indoneisa, Participatory Irrigation Sector Project