Indonesia: Decentralized Basic Education Project

8/10/2019 Indonesia: Decentralized Basic Education Project

1/71

PerformanceEvaluation Report

Indonesia: Decentralized Basic

Education Project

EvaluationIndependent


2/71


3/71

Reference Number: PPE:INO 2014-15Loan and Grant Numbers: 186-INO and 004!-INO

Inde"endent E#a$uat%on: PE-!!4

Performance E#a$uat%on Re"ortPerformance E#a$uat%on Re"ortPerformance E#a$uat%on Re"ortPerformance E#a$uat%on Re"ortNo#emberNo#emberNo#emberNo#ember 2014201420142014

Indones%aIndones%aIndones%aIndones%a:::: &ecentra$%'ed (as%c Educat%on Pro)ect&ecentra$%'ed (as%c Educat%on Pro)ect&ecentra$%'ed (as%c Educat%on Pro)ect&ecentra$%'ed (as%c Educat%on Pro)ect

*+%s document %s be%n, d%sc$osed to t+e "ub$%c %n accordance %t+ .&(/s Pub$%c ommun%cat%ons Po$%c 2011


4/71

NOTES

(i) The fiscal year of the government ends on 31 December.(ii) In this report, $ refers to US dollars.(iii) For an explanation of rating descriptions used in ADB evaluation reports, see

Independent Evaluation Department. 2006. Guidelines for Preparing

Performance Evaluation Reports for Public Sector Operations.Manila: ADB (aswell as its amendment effective from March 2013).

Director General V. Thomas, Independent Evaluation Department (IED)Director W. Kolkma, Independent Evaluation Division 1, IED

Team leader H. Son, Principal Evaluation Specialist, IEDTeam member

S. Labayen, Associate Evaluation Analyst, IED

The guidelines formally adopted by the Independent Evaluation Department on avoiding conflict ofinterest in its independent evaluations were observed in the preparation of this report. To theknowledge of the management of the Independent Evaluation Department, there were no conflicts ofinterest of the persons preparing, reviewing, or approving this report.

In preparing any evaluation report, or by making any designation of or reference to a particularterritory or geographic area in this document, the Independent Evaluation Department does not intendto make any judgment as to the legal or other status of any territory or area.


5/71

Abbreviations

ADB Asian Development BankBOS Bantuan Operational Sekolah

(School Operational Assistance Program)DBEP Decentralized Basic Education ProjectMONE Ministry of National EducationMORA Ministry of Religious AffairsNTB Nusa Tenggara Barat (West Nusa Tenggara)NTT Nusa Tenggara Timur (East Nusa Tenggara)PCR project completion reportTA technical assistance

Currency Equivalents

Currency Unit rupiah (Rp)

At Appraisal

At Project

Completion At Independent Evaluation

(15 October 2001) (15 June 2012) (31 August 2014)Rp1.00 = $0.0001 $0.0002 $0.0001

$1.00 = Rp9,935 Rp9,420 Rp11,705


6/71


7/71

Contents

Acknowledgments vii

Basic Data ix

Executive Summary xi

Chapter 1: Introduction 1

A. Evaluation Purpose and Process 1B. Expected Results 2C. Evaluation Method 2

Chapter 2: Design and Implementation 4

A. Formulation 4B. Rationale 5C. Cost, Financing, and Executing Arrangements 6D. Procurement, Construction, and Scheduling 8

E.

Outputs 8F. Consultants 9

G. Covenants 10H. Policy Framework 10

Chapter 3: Performance Assessment 12

A. Overall Assessment 12B. Relevance 12C. Effectiveness 19D. Efficiency 27E. Sustainability 28

Chapter 4: Other Assessment 30

A.

Impact 30B. ADB and Executing Agency Performance 30

Chapter 5: Issues, Lessons, and Follow-up Actions 32

A. Issues 32B. Lessons 33C. Follow-up Actions 34

APPENDIXES

1 Summary Design and Monitoring Framework 36

2 List of Interviewees 41

3 Questionnaire for Schools 444 Number and Type of Project Schools 50

5 Rating Matrix for Core Evaluation Criteria 51

6 Education Statistics 53

7 Poverty Statistics 55


8/71


9/71

Acknowledgments

A team of staff and consultants from the Independent Evaluation Department (IED)contributed to this study by conducting data gathering and analysis, desk reviews,interviews, research, and surveys. Hyun H. Son (team leader) prepared this report andStella Labayen (associate evaluation analyst) formatted it. The report benefited fromthe overall guidance of Vinod Thomas and Walter Kolkma. The team acknowledges thevaluable inputs of Rizza Leonzon and Ekki Syamsulhakim, the consultants who workedon this evaluation.

The team also thanks Wolfgang Kubitzki, Ferny Suhandi, and Imelda Marquez for theirassistance in organizing meetings with central and district government officials,development partners, and project schools. The team is grateful to Asian DevelopmentBank (ADB) staff, school representatives, and the Government of Indonesia for theirhospitality, as well as assistance and participation in the interviews. Their insights intothe decentralized basic education sector were most helpful in writing the report.

The report was peer reviewed by Tu Chi Nguyen of the World Bank and Farzana Ahmedof IED. Valuable comments were received on an earlier draft from ADBs Southeast AsiaDepartment and Indonesia Resident Mission. The report was also shared with theGovernment of Indonesia, which welcomed the findings and recommendations of thestudy.


10/71


11/71

Basic Data

Decentralized Basic Education Project (Loan 1863-INO; Grant 0047-INO)

Key Project Data

($ million)As per ADB Loan/Grant

Documents Actual

Total Loan Cost 125.0 138.4

Foreign Exchange Cost 21.1 26.3Local Currency Cost 103.9 112.1

ADB Financed 100.0 112.1Borrower Financed 25.0 26.3

Total Grant Cost 36.3 29.5

Foreign Exchange Cost 36.3 29.5

Borrower Financed 8.3 1.5Government of the Netherlands 28.0 28.0

Key Dates Expected ActualAppraisal 626 July 2001Loan Negotiations 310 October 2001Board Approval

- Loan- Grant

29 November 200114 March 2006

Loan Agreement 20 February 2002Loan Effectiveness 20 May 2002 20 May 2002

Closing Date- Loan- Grant

31 December 200831 December 2008

30 June 200930 September 2011

Closing Extensions 2 (Grant)Loan Effectiveness to Closing (months) 81 87Project Completion June 2012

Borrower Government of IndonesiaExecuting Agency Ministry of National Education

Mission Data

Type of Mission No. of Missions No. of Person-Days

Inception 1 138Handover 1 14Review 12 309Special project administration 2 82Midterm 1 75Final 1 10Project completion review 1 20Independent evaluation 1 9


12/71


13/71

Executive Summary

Decentralization is a policy option that has been adopted by many countries to improveaccess to basic services, including education. Since 2001, the Government of Indonesiahas embarked on a program to decentralize delivery of basic education. By making theeducation system more attuned to local needs, decentralizationthe process ofdelegating authority or functions from central government to local government unitswill be able to play a crucial role in improving the access to and quality of basiceducation in Indonesia.

To help Indonesia roll out its decentralized basic education program, the AsianDevelopment Bank (ADB) supported a project to improve the participation in andcompletion of 9 years of basic education, particularly among the poor. The project

included primary and junior secondary levels in the provinces of Bali, East NusaTenggara (NTT), and West Nusa Tenggara (NTB). It supported three outcomes: (i)improved participation, transition, completion, and performance in basic educationamong poor children in Bali, NTB, and NTT; (ii) implementation of school-basedmanagement in project schools; and (iii) effective district education management in thethree provinces. To achieve these objectives, the project had three components: schooldevelopment; district basic education development; and monitoring, evaluation, andreporting.

The project was financed by a combination of loan and grant. Of the $138.4 millionloan component, $112.1 million was financed by ADB and $26.3 million by theGovernment of Indonesia. The grant component totaled $29.5 million, of which $28.0million was funded by the Government of the Netherlands and $1.5 million by theGovernment of Indonesia.

This report presents performance evaluation findings of the Decentralized BasicEducation Project based on four core evaluation criteria (relevance, effectiveness,efficiency, and sustainability) and two additional criteria (institutional development andimpact). Overall, the project performance is rated successful.

The project is rated highly relevant. The timing of its implementation was responsive toaddressing insufficient capacity during Indonesias transition to a decentralized basiceducation system. The project is also aligned with the education policies and strategiesof both the government and ADB. Changes in the project coverage involvingwithdrawal of five Jakarta districts and redirection of this funding to two districts in

impoverished NTT improved its relevance. Modifications in project design during theearly phase of implementation, as well as weak coordination between the Ministry ofNational Education (MONE) and the Ministry of Religious Affairs (MORA), the projectsexecuting and co-executing agencies, respectively, posed challenges to implementationbut did not compromise relevance either in terms of targeting schools and districts orin achieving the objective of improving access of the poor to basic education.

The project is effective. Analysis of primary time-series data obtained during theevaluation indicates that statistically significant improvements in education outcomes


14/71

xii Indonesia: Decentralized Basic Education Project

including enrollment, completion, transition, and performance were observed inselected project districts after the project was implemented. The project also helped toimprove school capacity by promoting an evidence-based approach to formulatingschool development plans; encouraging community involvement in education matters;and improving district capacity for basic education, particularly in teacher developmentand the evaluation of school development plans.

The project is efficient. Despite the limited size of grants, the project supported notableachievements in improving school and district capacity for managing basic educationand enhancing school infrastructure. The progress of its loan component was initiallyslow, but this picked up throughout project execution. The grant component was twiceextended, as the implementation period of only 2.5 years proved too short for severelypoor and often inaccessible districts.

The project is likely sustainable. The formulation of school development plans, whichwas pioneered by the project, has become standard practice among schools. Schoolcommittees continue to participate in creating school strategies and budget plans.School and district officials who received training under the project continue to impartknowledge to their counterparts who were not part of the project. The central and

district governments provide the primary source of funding to help project schoolsdeliver continued support for poor students through scholarships, transportallowances, remedial and catch-up programs, and tutoring for students missing school.

The project is rated as having a significantimpact on enhancing local capacity for basiceducation management, improving education outcomes, and shaping governmentpolicies related to decentralized basic education services. ADBs performance is ratedless than satisfactory, given that high turnover of its project officers constrainedeffective and continued supervision and management of the project. MONEsperformance is rated satisfactory, since it eventually gained sufficient capacity afterinitial difficulties.

The evaluation draws the following lessons:

i. Decentralization of basic education can be reinforced through strongcollaboration between MONE and MORA. Structural misalignment hampers thecoordination of these ministries in managing basic education because MORAremains centralized while MONE has been decentralized. Although bothministries may have misaligned structures and varying levels of resources andexpertise, they are nevertheless jointly tasked with providing basic education. Amore collaborative effort is needed for its successful delivery and management.

ii. Good technological infrastructure at the school and district levels is imperativefor data management. Although there is an online database for schoolindicators, called Dapodik, schools have yet to fully maximize this medium

because many lack internet access. Capacity building of school staff is alsonecessary for the online system to work effectively.

iii. High staff turnover impedes capacity building and disrupts efforts to promotedecentralized basic education management. Frequent staff changes amongADBs project officers and district governments hampered the transfer andsharing of knowledge, as it did communication and rapport-building amongstakeholders.


15/71

Executive Summary xiii

Given these lessons, the project performance evaluation report makes theserecommendations:

i. Coordination between MORA and MONE, which is crucial for decentralizedbasic education, requires improvement. Best practices in educationmanagement need to be shared with officials of both ministries. Moreover, the

role of provincial governments in decentralized basic education should bestrengthened, particularly in monitoring and evaluation. This is becauseMORAs provincial offices are primarily tasked with providing madrasaheducation, while responsibility for general education lies with districtgovernments and district MONE offices.

ii. Endeavors to enhance the delivery of basic education in Indonesia should betied in with continued investments in enhancing transport and technologicalinfrastructure and school facilities. Improving the access of the poor totransport is a priority everywhere, but particularly in remote areas. In theseareas, improving access to junior secondary schools, which are normally locatedin subdistrict capital cities, is especially needed. Improvements in technologicalinfrastructure in schools and districts are necessary to facilitate continuous data

updates on the Dapodik system. Continued rehabilitation of classrooms inremote areas in Bali, NTB, and NTT is also necessary.

iii. More than a decade since it was first implemented, decentralized basiceducation in Indonesia remains beset with challenges. These include its impacton teacher policy, given the considerable district authority in staffing (includingover transfer of teachers) and the perceived politicization of educationappointments made by district governments. Should ADB choose to continueproviding support in this area, it should consider how such challenges could bebest addressed.


16/71


17/71

CHAPTER 1

Introduction

A. Evaluation Purpose and Process

1. This project performance evaluation report assesses the Decentralized BasicEducation Project, which was designed to support decentralized planning,management, and delivery of basic education services in Indonesia. The project aimedto enhance participation in and completion of 9 years of basic education, particularlyamong the poor in Bali, West Nusa Tenggara (NTB), and East Nusa Tenggara (NTT). Theproject was evaluated based on four core criteria (relevance, effectiveness, efficiency,

and sustainability) and two additional criteria (institutional development and impact),as specified under Asian Development Bank (ADB) guidelines. 1 The evaluation alsoassesses the projects impact on student performance in beneficiary districts, analyzesthe roles of its stakeholders, and reviews other factors that may have affected projectperformance. This evaluations findings will feed into the forthcoming validation ofIndonesias County Partnership Strategy 20122014 final review.

2. The project completion report (PCR) was prepared in June 2012. Overall, theproject was rated successful based on the four core evaluation criteria. It was ratedhighly relevant to the poverty reduction priorities of ADB and the Government ofIndonesia. Changes in the projects scopewhich included adding districts inimpoverished NTTfurther improved its relevance. The project was rated effective,based on (i) improvements in net enrollment, completion, and transition rates andexamination scores (although no targets had been specified); (ii) implementation ofschool-based management in 3,000 schools and the establishment of effectivecommunity-school partnerships; (iii) increased district education funding over 2001levels; (iv) development of data-based district education development plans; and (v)implementation of plans for quality and equity improvement. It was rated efficientbecause progress under the loan component picked up during implementation despiteinitial delays. It was rated partly sustainable. The project helped to mainstream schooland district planning and school-based management through national policies. Asnoted in the PCR, however, some stakeholders took the view that activities financedunder the project would not be sustainable without project support.

3. The project was assessed to have a likely significantimpact on project districts

and provinces given improved school management that involved both school officialsand communities. The PCR noted, however, that it was difficult to assess the projectsimpact on district capacity.

1 Independent Evaluation Department. 2006. Guidelines for Preparing Performance Evaluation Reports forPublic Sector Operations.Manila: ADB (as well as its amendment effective from March 2013).


18/71

2 Indonesia: Decentralized Basic Education Project

B. Expected Results

4. The project sought to support the decentralized planning, management, anddelivery of basic education services to improve participation in and completion of 9years of basic education. It was implemented in 26 districts across 3 provinces (Bali,NTB, and NTT).2 The project targeted 5,238 schools in poor villages, of which 4,329

were primary and 909 were junior secondary schools. It supported three outcomes: (i)improved participation, transition, completion, and performance in basic educationamong poor children in the three provinces; (ii) implementation of school-basedmanagement in project schools; and (iii) effective district education management in thethree provinces.

5. To achieve its objectives, the project had three components: schooldevelopment; district basic education development; and monitoring, evaluation, andreporting. It supported the following major outputs: (i) implementation of school-based management, (ii) improved school quality with School Development Fundassistance, (iii) improved capacity of districts to plan and manage decentralized basiceducation as reflected in their 5-year district education development plans, (iv)improved district education with District Education Development Fund assistance, and(v) effective program and financial monitoring compliance. The projects design andmonitoring framework is presented in Appendix 1.

C. Evaluation Method

6. The evaluation used qualitative and quantitative techniques in examining theprojects performance. The qualitative approach was used for in-depth interviews withofficers from the Ministry of National Education and district education offices, as wellas school principals, teachers, and staff involved in the project. Appendix 2 provides acomplete list of government and school officials interviewed during the evaluationmission. Appendix 3 presents the questionnaire for school officials.

7. The quantitative approach was particularly useful for examining educationoutcomes and poverty status. Various secondary data were used in analyzing educationoutcomes, including those for enrollment, transition, and completion rates; nationalexam scores; and poverty status in selected provinces and districts. The evaluation alsoused the findings of a survey conducted during the mission that provided primary time-series data on education indicators and other information related to the project. Surveyquestionnaires were distributed to selected project schools. The quantitative approachused in this evaluation study makes before-and-after comparisons. For somequantitative outcomes, such as enrollment, completion, and transition rates as well asexam scores, the analysis involved mean-difference testing to check robustness of thecomparison analysis.

8. The surveys sample of project schools was selected using a mixture ofprobabilistic and non-probabilistic approaches to ensure that each school in eachprovince had an equal opportunity to be selected as part of the sample. The sample forthis survey covered 56 schools, which are representative in terms of districts and typesof schools.

2 Jakarta was initially included as one of the three target provinces. However, it was withdrawn from theproject because its income was sufficient to achieve objectives similar to those of the project. The fundspreviously allocated to Jakarta were then redirected to NTT.


19/71

Introduction 3

9. Although not the case of Bali, schools in NTB and NTT are scattered on variousislands within the two provinces. Hence, the study purposively selected the islandswithin these provinces. Lombok Island in NTB and Timor Island in NTT were chosen. Thenumbers of districts within the islands was then calculated proportionately, resulting intwo districts each for Bali and NTT and three districts for NTB. A simple randomsampling mechanism was used to select the districts on each of the islands. On that

basis, the selected districts were Bangli and Buleleng in Bali; Lombok Barat, LombokTimur, and Mataram in NTB; and Timor Tengah and Timor Tengah Utara in NTT. Takinginto account differences in the local education policies, as well as socioeconomiccharacteristics in each of the provinces and districts, this study calculatedproportionately the number of project schools to be surveyed for each district. Thiscalculation was based on the number and types of schools covered by the project aspresented in the PCR (Appendix 4). A stratified random sampling mechanism based onthe level of school was then used to decide which project schools were selected as partof the sample.


20/71

CHAPTER 2

Design and Implementation

A. Formulation

10. The Government of Indonesia has implemented numerous measures to improvethe access to and quality of basic education, including decentralizing the managementand delivery of basic education following the passage of Law No. 22/1999 on RegionalGovernance and Law No. 25/1999 on Fiscal Decentralization. To this end, districtgovernments have been tasked with providing financing for basic education.

11. In 1997, the government requested ADB support for improving the delivery ofbasic education in the provinces of Bali and West Nusa Tenggara (NTB) and the SpecialCapital Region Jakarta (footnote 2). The government selected these provinces becauseof their gaps in basic education and the regional demarcation between ADB and theWorld Bank.3 Jakarta withdrew from the project, however, and was replaced by EastNusa Tenggara (NTT).

12. ADB in 1998 approved small-scale project preparatory technical assistance (TA),but the Asian financial crisis caused project preparation to be suspended.4This wasresumed in 2000. The government requested that the project support thedecentralization of basic education management to build on achievements of the ADB-funded Social Protection Sector Development Program. ADB subsequently approvedsmall-scale TA to redesign the project.5A loan appraisal mission was fielded during July2001.

13. The project had no attached TA, but two related TA projects and a small projectfunded by the Japan Fund for Poverty Reduction complemented projectimplementation. Bridging TA providing support for decentralized educationmanagement commenced implementation in March 2002 and supported the initialsteps in decentralizing the management of basic education to improve service delivery.This included capacity building for a special decentralization unit in the Ministry ofNational Education (MONE).6Additional TA on integrating poverty considerations intodecentralized education management began implementation in April 2003, shortlyafter project consultants were fielded. The TA supported poverty mapping for 16districts covered by the project and 50 participatory poverty studies.7 The project,

3 ADB. 2001. Report and Recommendation of the President to the Board of Directors on a Proposed Loan tothe Republic of Indonesia for the Decentralized Basic Education Project. Manila.

4 ADB. 1998. Technical Assistance to the Republic of Indonesia for Basic Education in Bali and Nusa TengaraBarat Project.Manila (TA 3007-INO, $150,000, approved 16 April 1998).

5 ADB. 2000. Technical Assistance to the Republic of Indonesia for the Preparation of Decentralized BasicEducation Project. Manila. (TA 3456-INO, $150,000, approved 14 June 2000).

6 ADB. 2006. Technical Assistance Completion Report: Support for Decentralized Education Management inIndonesia.Manila (TA 3701-INO).

7 ADB. 2007. Technical Assistance Completion Report: Integration of Poverty Considerations in DecentralizedEducation Management in Indonesia.Manila (TA 3957-INO).


21/71

Design and Implementation 5

backed by Japan Fund for Poverty Reduction, was approved in May 2002 andconsultants were fielded 12 months later.8 It pilot tested school-based managementand community-based education as a pro-poor service delivery scheme in particularlyunderserved schools.

14. The project was funded by a $100.0 million loan from ADB and grant

cofinancing of $28.0 million from the Government of the Netherlands. The loan wasapproved on 29 November 2001 and closed on 31 December 2008. The grant wasapproved on 14 March 2006 and extended for 1 year, to 31 December 2009, to allowfor project completion in NTT, and later to 31 December 2010 to cover emergencyreconstruction work in Sumatra. Some $10.0 million was reallocated from the loan torebuild schools after an earthquake and tsunami struck Aceh and North Sumatra inDecember 2004. Similarly, grant savings of $1.3 million were used in reconstructing 9schools in West Sumatra after a 2009 earthquake.

B. Rationale

15. The project provided crucial support during Indonesias transition to adecentralized system of basic education. Its commencement coincided with the initialimplementation of Law No. 22/1999 on Regional Governance and Law No. 25/1999 onFiscal Decentralization in 2001, during which a lack of technical guidance for provincialand district governments in executing the laws, especially for the education sector,posed a challenge. The project therefore supported pioneering efforts in implementinga decentralized education system.9

16. Per the governments request, the project supported the process ofdecentralizing basic education management, in accordance with the laws on regionalgovernance and fiscal decentralization (footnote 3). Prior to implementation of theselaws, the education system was highly centralized. Basic education was managedcentrally through complex and compartmentalized structures in three ministries: theMinistry of Home Affairs, MONE, and the Ministry of Religious Affairs (MORA). MONE

was responsible for overall planning, coordination, and regulation for education at alllevels and it directly administered and financed public junior secondary education. TheMinistry of Home Affairs administered and delivered public primary education,overseeing teacher management, establishment and maintenance of primary schools,and provision of subsidies to private primary schools. MORA administered and financedpublic primary and junior secondary madrasahschooling and supervised and providedfinancial assistance to private madrasahs. This system had been replicated at provincialand district levels, where all three ministries were represented in parallel by localoffices. Similar to this complex management structure, budgeting and financing forbasic education involved five ministries: the Ministry of Finance, Ministry of HomeAffairs, MONE, MORA, and the National Development Planning Agency, with a mix offinancing arrangements and funds sources for each school type and level.

17. Under Indonesias decentralized basic education system, district governmentsare now responsible for planning, financing, managing, and delivering basic education.Their responsibilities also cover school staff appointments and transfers and schoollocation planning. The central government sets policies, the national curriculum, and anational examination and assessment system. It also sets standards for (i) achievementat each age; (ii) learning materials; (iii) acceptance, transfer, and certification of

8 ADB. 2007. Implementation Completion Memorandum for INO: Supporting Community-Based BasicEducation (JFPR 9016).Manila.

9 Study teams interviews with stakeholders during evaluation mission.

The project

provided crucia

support during

Indonesias

transition to a

decentralized

system of basic

education


22/71


students; and (iv) learning times for basic education and fixing the educationalcalendar. The central government establishes procedures and standards for staffappointments and dismissals; determines pensions, salaries, benefits, and rights andobligations; and establishes the legal status of civil servants and regional civil servants,including teachers and principals.

18. Provincial governments (i) define policy on student selection and acceptance,including equity for minority students and students from poor families and remoteareas; (ii) provide instructional materials for basic education; (iii) assist in highereducation management, curriculum production, accreditation, staff appointments, andopening and closing of colleges; and (iv) manage special institutions, including trainingcenters, teacher training institutions, and special education. Provincial governmentsmay also decide on the transfer between districts of civil servants, including schoolprincipals and teachers, and their transfer between cities and provinces (footnote 3).

19. By supporting the decentralized system, the project also responded tochallenges in improving the equity in and quality of delivering basic educationservices.10The proportion of primary school-age children enrolled at this levelthe netenrollment ratewas 93.0% at the start of project implementation in 2001, while the

ratio of female-to-male primary enrollment was 98.0%. Nearly 660,000 primary school-aged children were out of school in that year, more than 60.0% of them girls.11

20. ADBs development agenda for Indonesia also prioritizes basic education. Until1990, ADB support for Indonesias education sector was channeled mainly to thevocational, technical, and higher education subsectors. Since then, however, ADB hasincreased support for basic education to improve poor childrens access to basiceducation and enhance learning outcomes at this level. Among ADBs basic educationinvestments in Indonesia is the Social Protection Sector Development Program, whichsupported a scholarship and school block grant program. The Social Protection SectorDevelopment Program was introduced in the aftermath of the Asian financial crisis. Ithelped maintain stability in primary and junior secondary school enrollment andsupported policy dialogue for improving social safety nets mechanisms.

C.

Cost Financing and Executing Arrangements

21. At appraisal, the total cost of the loan component was $125.0 million. At loanclosing, the actual project cost was $138.4 million, of which $112.1 million wasfinanced by ADB and $26.3 million by the government. The appreciation of specialdrawing rights versus the dollar caused the 11.0% increase in the loan amount. Therewere no cost overruns or underruns since project expenditures were linked to schooland district development plans.

22. In terms of cost categories, the PCR noted that expenditure on research,surveys, and studies (42.0% of the appraisal estimate) was low because the project had

not carried out baseline and midterm evaluation surveys and other studies. 12 Theevaluation team reviewed two impact evaluation documents produced under theproject: (i) Decentralized Basic Education Project: Consulting Services for Baseline andBenefit Impact/Evaluation Studies, Initial Findings (2006); and (ii) Benefit, Monitoring,

10Indonesias education system consists of preschool, primary education (grades 1 to 6), junior secondaryeducation (grades 79), senior secondary education (grades 1012), and tertiary education. Basiceducation covers 9 years of compulsory primary and junior secondary education.

11ADB. 2013. Key Indicators for Asia and the Pacific.Manila.12ADB. 2012. Completion Report: Indonesia: Decentralized Basic Education Project. Manila (p. 7).


23/71


and Evaluation Studies of the Decentralized Basic Education Project (Baseline andMidterm Evaluation Studies): Final Report (2008).13However, these impact evaluationstudies did not clearly identify the type of impact measurement being calculated.Moreover, only baseline and midline evaluations were conducted. The baseline studydid not incorporate the appropriate control group as the respondents at the schoollevel only included principals, teachers, students, and school committees from 55

project schools. While the midterm report stated that it included non-project schools, itdid not sufficiently discuss how the non-project schools were selected. Moreover, thebaseline report used the district minimum service standard as the comparisonbenchmark, which may result in misleading findings since the minimum servicestandard may also include project schools. Given the different sampling methods andquestionnaires used by the baseline and midterm evaluation studies, the study teamassumed there to be no connection between the two. The study team also had noaccess to the data used in the studies. As the PCR had noted, due to the absence ofdata, information for measuring project benefits also relied on the memories of thestakeholders.14

23. The estimated cost of the grant component was $36.3 million, of which $28.0million was to be funded by the Government of the Netherlands and $8.3 million by

the Government of Indonesia. The actual cost was $29.5 million, of which theNetherlands funded $28.0 million and Indonesia $1.5 million.

24. At loan closing, the project had disbursed $112.1 million, about 112% of theoriginal loan amount and 97.0% of the current loan value, which included an initialadvance of $2.6 million (25 June 2002) to finance the cost of various small contracts.The project disbursed $28.0 million in grant funds, including the ADB administrationcost of $0.5 million. The loan account was financially closed on 30 June 2009 and thegrant on 30 September 2011.

25. MONE was the projects executing agency. MORA was the coexecuting agencyand assigned staff to project management offices at the different levels of government.At national level, a project steering committee provided policy guidance. Districtcoordination committees, education boards, and school committees assisted thesteering committee. Meanwhile, the central technical committee assumed technical andprogrammatic functions and the central facilitation unit managed overall financial andadministrative coordination. The facilitation unit was supported by provincial anddistrict project management units. Although complex, this implementation structurewas appropriate given the projects large size and constant changes in administrativecontext (footnote 3).

13 The projects baseline and midterm evaluations can be accessed in ADBs Archives Section. See InsanHitawasana Sejahtera. 2008. Decentralized Basic Education Project: Consulting Services for Baseline andBenefit Impact/Evaluation Studies, Initial Findings. Jakarta; and Indonesia Ministry of National Educationand Indonesia University of Education. 2006. Benefit, Monitoring and Evaluation Studies of theDecentralized Basic Education Project (Baseline and Midterm Evaluation Studies) Final Report. Jakarta.

14 Proper impact evaluations should include appropriate design and identification of control and treatmentgroups. The design of an impact evaluation should be specified at the beginning of a program and includethe type of measurement impact (e.g., average treatment effect, average treatment on the treated,spillover), sampling method of both treatment and control groups, and the evaluation method. Theselection of treatment and control groups should also be adequately discussed. Impact evaluations shouldbe done also at the baseline and endline of a project.


24/71


D. Procurement Construction and Scheduling

26. The loan component was approved in 2001, took effect in 2002, and closed onschedule in 2008. Although there were initial delays, the pace of implementationpicked up. Some variations in project implementation procedures were adopted.Phased implementation was originally identified at the planning stage, whereby the

batches were based on schools within districts rather than on the districts themselves.Phased implementation was abandoned, however, because districts wanted tocommence project activities immediately. Given this change, all original districtscommenced project activities in mid-2003 and all districts continued to receive supportover the next 5 years. Within these districts, three different batches of schools alsoreceived support for 3 years. Challenges were also encountered in use of the SchoolDevelopment Fund, which was released a year before consultants were mobilized andbefore schools had formulated their school development plans.

27. The grant was approved in 2006 and extended twice before it was closed in2010 due to (i) a 1-year extension to make sure that the programs benefited NTT, and(ii) a second 1-year extension to allow grant savings to be used for reconstructingschools in Padang Pariaman.

28. Civil works involved rehabilitation of existing facilities as well as construction ofearthquake-resistant schools on the remote island of Simeulue and rehabilitation ofearthquake-ravaged schools in Padang Pariaman.

29. Schools and districts carried out most of the procurement for items eligibleunder the District Education Development Fund and the School Development Fund.District project management units procured eligible items under the district educationfund through direct purchases using limited local tenders. On the other hand, schoolsprocured eligible items under the School Development Fund through direct purchaseand school committees managed procurements for the rehabilitation of schools. Cashexpenditures used direct purchase and were supplemented by such in-kind

contributions from the community as materials and labor. Training and workshopswere provided by facilitators and trainers according to ADB and government guidelines.

30. Several procurement problems were uncovered by the external auditor, most ofwhich were related to overpayment for workshops and training. As noted in the PCR,the findings of the external audit indicate that community monitoring of schoolactivities and finances is a viable method of ensuring compliance.

E. Outputs

31. The project supported three major components: (i) school development; (ii)district basic education development; and (iii) monitoring, evaluation, and reporting.These components were supported by the following outputs.

32. Implementation of school-based management. The project improved theschools capacity in developing and implementing school development plans. Projectschools were required to develop these plans to access project funds for scholarshipsfor the poor, rehabilitation of classrooms and other facilities, and other activitiesidentified in their plans. About 97% of the 4,244 project schools drew up these plans,


25/71


according to the baseline study.15However, project schools were not required to reporttheir achievements against their plans. The project also helped to strengthencommunity involvement in schools through the formation of school committees.

33. Improved school quality with School Development Fund support. This was theprojects funding mechanism to support activities identified in school development

plans. To ensure transparency and accountability of the School Development Fund, itsaccounts were audited annually by external auditors. Public accountability systemswere also established to monitor the projects operation. As noted in the PCR, 95% ofschools reported on the School Development Fund, program and budget plans, andprogram and budget implementation. Moreover, 85% of schools were reported to haveestablished complaint mechanisms, though only 10% of school principals were seen aswilling to receive complaints.

34. Improved capacity of districts as reflected in a 5-year district educationdevelopment plan. The project established district education boards in compliance withMinisterial Regulation No. 044/2002. These were tasked with monitoring districteducation programs, as well as school development plans. The boards were providedtraining on education management, including budget analysis. The legal framework for

the boards was revised in 2010 by a new regulation on the management of education(PP 17/2010). As a result, funding of the boards by districts became discretionary andthe number of board members was reduced. Although board members work in avoluntary capacity, they nevertheless require funding for their operations. The projectalso required all districts to publish their education statistics annually and to completetheir district education development plans. During the project, all districts publishedstatistics annually and completed their district education development plans. Somedistricts continue to update their plans and publish their statistics, but weaknesses ininformation systems make it difficult to keep the data current and for them to be ableto publish the statistics on time.

35. Improved district education with District Education Development Fund support.To improve district education, the project supported the preparation, implementation,and monitoring of strategic 5-year district education development plans and 5-yearannual action plans by all districts. Under the plans, districts developed systems forplanning, managing, financing, staffing, and monitoring teacher development; schoolsupervision; recruitment, training, and performance review of school principals;facilities expansion; and district financing of education. The fund supported theimplementation of priority district education development programs and capacitybuilding activities.

36. Effective program and financial monitoring compliance. The project helpedpromote transparency and accountability as part of school-based management. Schoolsinform communities of their school development plans, programs, and budget plans,as well as their School Development Fund and District Education Development Fund

accounts.

F. Consultants

37. Under the loan component, three packages of consulting services wereenvisaged during appraisal: (i) school-based management; (ii) district capacity building

15Insan Hitawasana Sejahtera. 2008. Decentralized Basic Education Project: Consulting Services for Baselineand Benefit Impact/Evaluation Studies, Initial Findings. Jakarta.


26/71


and project implementation support; and (iii) independent monitoring, evaluation, andreporting. Some 82 person-months of international consultant inputs (compared with78 person-months at appraisal) and 8,403 person-months of national consulting inputs(compared with 7,782 person-months at appraisal) were used. The consultants for theloan component were mobilized a year after project activities commenced.

38. Under the grant component, two additional packages were planned: (i) schooland district building and project implementation support, and (ii) independentmonitoring and evaluation. The grant packages mobilized only national consultants,using 1,288 person-months of national consulting inputs (against the planned 960person-months). All consultants were recruited in accordance with ADBs Guidelines onthe Use of Consultants. Several changes requested by the consulting firm led toconsiderably greater person-months and cost. A university research institute wascontracted in 2006 to conduct the baseline and midterm evaluation study, but itperformed poorly. The institute was replaced by a social survey firm in 2007, whichconducted the required baseline and impact studies.

39. District project and school personnel considered package (i) and (ii) consultantsunder the loan component helpful or very helpful, but they found package (iii)

consultants too focused on monitoring schools utilization of the School DevelopmentFund. The consultants for package (iii) only monitored financial compliance in theschools and districts. They did not conduct program monitoring and evaluation afterthis activity was removed from their contract. The monitoring and evaluation activitiesfor the loan project did not refer to the design and monitoring framework indicators.Meanwhile, as noted by the PCR, the consultants under the grant componentperformed well and achieved good results.

G. Covenants

40. Twenty-five of 27 covenants were complied with. The loan covenants on therole of the provincial project monitoring units and gender were only partially complied

with. These units were envisaged to monitor or evaluate project activities and impactbut failed to do so. As a result, only progress and performance were monitored, notthe achievement of indicators and targets. The provinces were originally assigned toconduct performance monitoring and evaluation, but this was not carried out. Anindependent contractor was hired to conduct an impact study on beneficiary schoolsunder the loan project and baseline and impact studies on beneficiary schools in NTTunder the grant. Meanwhile, the covenant on gender required two women membersper school committee. In NTT, almost 70% reached this target, but results in Bali andNTB were poor.

H. Policy Framework

41. The project influenced national government policy and procedures, and mostnotably the School Operational Assistance Program (BOS), which was launched in 2005.BOS partly adopted the projects block grant system, and it provides block grants to allschools to enhance the implementation of school-based management and lower thecost of providing basic education. While compulsory fees from parents may not beretained, the central government permits local governments and parents to financially


27/71


support schools if BOS funds are insufficient.16 The disbursement of BOS funding isdecided by school committees in a similar procedure established by the project.

42. The project also influenced legally binding national education standardsintroduced from 2006 to 2008 and minimum service standards for districts introducedin 2010. Moreover, school-based management is now official policy and implemented

in all schools nationwide. However, the school-managed model for rehabilitation underthe project has been discontinued under new legislation requiring a return to districtcontracting. Moreover, many districts now restrict parental and community financialsupport for schools, which was one of the projects achievements.

16 J. Tobias et al. 2014. Towards Better Education Quality: Indonesias Promising Path. London: OverseasDevelopment Institute.


28/71


29/71

Performance Assessment 13

94.3

96.6

93.6

95.5

85

90

95

100

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

P

c

B a l i Nusa Tenggara Barat

Nusa Tenggara Timur Indonesia

System Law in 2004. Between 2002 and 2004, it passed other laws relevant todecentralization, such as those on regional government and centralregional financialbalance, the national planning and financial system (budgeting), and the nationaltreasury (payments mechanism).

46. The projects objective of improving basic education outcomes, including

participation, transition, completion, and performance, was also relevant to Indonesiaseducation challenges at the time of project implementation. Although Indonesia hasnearly achieved universal access to primary education, participation in secondaryeducation remains a concern. The average net enrollment rate of children aged 715years was 94% in primary school and 59% at the secondary level at around the time ofproject implementation in 19982002.17Figures 1 and 2 indicate that average netenrollment in Nusa Tenggara Timur (NTT) was generally lower than the nationalaverage for both primary and secondary levels during 20032013, while net enrollmentin Bali and Nusa Tenggara Barat (NTB) was slightly higher for much of the period. Lowaverage enrollment in NTT may be attributed to the poor quality of basic infrastructureand school facilities, which restrict childrens access to education services. The slump inIndonesias net enrollment rate from 2010 to 2011 coincides with a decrease in publicspending on education. Public spending on education reached 3.5% of GDP in 2009

but decreased to 3.0% and 2.8% of GDP in 2010 and 2011, respectively.18As discussed,the government selected the provinces based on education gaps and the regionaldemarcation between the Asian Development Bank (ADB) and the World Bank.19

Figure 1: Primary Net Enrollment in Selected Provinces, 20032013

Note: Since 2007, the net enrollment rate includes nonformal education (Package A, whichis primary school, and Package B, which is junior secondary).Source: Study team calculations based on Indonesian Statistics Office data.

17 National Development Planning Agency. 2002. National Program for Indonesian Children: Education.Jakarta.

18 Figures are World Bank estimates accessed at data.worldbank.org on 6 October 2014.19 The World Bank at the time supported basic education projects in the provinces of Central Java, East Java,

Kalimantan, Nusa Tenggara Timur, Sulawesi, Sumatra, and West Java, and was preparing a new basiceducation project covering East Java and South and West Sumatra. See ADB. 2001. Report andRecommendation of the President to the Board of Directors on a Proposed Loan to the Republic ofIndonesia for the Decentralized Basic Education Project. Manila.


30/71


80.780.2

59.2

73.7

0

30

60

90

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

B a l i Nusa Tenggara Barat

Nusa Tenggara Timur Indonesia

47. Improving enrollment at junior secondary level is also an area of concern inIndonesia (Figure 2). Several factors influence low enrollment at the junior secondarylevel, including household welfare, gender, religious background, childs performance,supply-side readiness, and employment opportunity. Children from poor householdsare less likely to enroll in junior secondary school, as are girls and children from Muslimbackgrounds. Children with higher national exam scores in primary school are more

likely to enroll in junior secondary school. Enrollment at junior secondary level is alsofound to be positively affected by the number of schools built in those areas where thechildren live. Finally, children tend to continue their school through junior secondarylevel if there are few employment opportunities in the community.20

Figure 2: Junior Secondary Net Enrollment in Selected Provinces,

20032013

Note: Since 2007, the net enrollment rate includes nonformal education (Package A, whichis primary school, and Package B, which is junior secondary).Source: Study team calculations based on Indonesian Statistics Office data.

48. Children dropping out of school also undermined efforts to improveIndonesias basic education system. About 600,000 Indonesian children dropped out ofprimary school in 2001, at the start of project implementation, putting the dropoutrate at 2.7%.21Among junior secondary school students in 2001, the dropout rate was3.5%, meaning that about 300,000 students left junior secondary school in that year.NTB and NTT had higher dropout rates in 2001 than the national average, at 3.2% and5.4%, respectively, while Bali had a lower rate of 2.1% (Table 2). In Indonesia, poorchildren tend to give up schooling for working. Some evidence nevertheless suggeststhat banning children from working could worsen their school attendance since theywill not have sufficient financial resources to continue studying.22

20 Suryadarma et al. 2006. Causes of Low Secondary School Enrollment in Indonesia.SMERU.21 National Office for Research and Development. 2002. Education Statistics. Ministry of National Education.

Jakarta.22 A. Suryahadi, A Priyambada, and S. Sumarto. 2005. Poverty, School, and Work: Children during the

Economic Crisis in Indonesia. Development and Change. 36 (2). pp. 351373.

P

c


31/71


Table 2: Dropout rate of Primary and Junior Secondary Students

in Selected Provinces, 2001/2002 ( )

Province Primary Junior Secondary

Bali 2.1 1.9Nusa Tenggara Barat 3.2 2.9Nusa Tenggara Timur 5.4 3.4Indonesia 2.7 3.5

Source: National Office for Research and Development. 2002.

49. The project was particularly relevant to improving the transition rate of juniorsecondary school-aged children in NTT. Some of the poorest villages are located nearthe border of Timor-Leste. According to interviews with stakeholders, the nearest juniorsecondary school was about 15 kilometers from these villages. In addition, the roadsleading to the subdistrict capital, where junior secondary schools are normally located,were in very poor condition, making parents reluctant to send their children to juniorsecondary schools. Through the project, these districts received assistance to buildjunior secondary schools, such as those in Biboki Utara and Bikome Utara, allowingchildren to continue their studies at the junior secondary level.

50. The project was also relevant to the need at the time of implementation forrehabilitating school infrastructure. During 2001/2002, 24% of primary school buildingsin Indonesia, including classrooms, showed signs of severe damage and 33% of slightdamage (Appendix 6, Table A6.3). About 4% of junior secondary buildings were badlydamaged and 10% slightly damaged (footnote 17). In NTT, Bali, and NTB, respectively,66%, 55%, and 50% of primary classrooms were damaged. It was difficult to ascertainthe projects contribution to improving school facilities at the time of the evaluationmission because other sources of funding were made available for school buildingrehabilitation after the projects life. Interviews with Ministry of National Education(MONE), school, and district officials revealed that the government has beenrehabilitating schools since 2012, allocating as much as Rp17.5 trillion (equivalent to$1.5 billion) for this endeavor.

51. The project was also relevant to government and ADB education policies andstrategies. As discussed earlier, it was highly aligned with the countrys decentralizationstrategy, which serves as a means for improving the quality of and access to basiceducation by building a more integrated education system at local level that caneffectively respond to local needs and conditions. ADBs Country Assistance Plan, 20012003 for Indonesia acknowledged decentralization of basic education as a major areaof ADB support, particularly in addressing the need to develop local managementcapacity and to ensure better provision of basic education for the poor. The ADBstrategy also recognized that although decentralization could improve transparency,accountability, and community participation, there were risks, particularly during thetransition period. These included inadequate technical capacity and human andfinancial resource constraints that could hamper the efficient delivery of basic services,including education. The strategy stated that ADB will help redefine the role and

support the reorganization of MONE in line with the decentralization law that wasbeing newly implemented at that time.23

52. Decentralization in Indonesia has attracted support from many donors. Theseinclude, among others, ADB, United Nations agencies, and the World Bank. Alongsidethe government, these three launched a partnership in April 2000 to supportgovernance reforms, including decentralization. ADB assumed a major role in

23ADB. 2000. Country Assistance Plan: Indonesia, 20012003. Manila.


32/71


decentralization and various anticorruption initiatives. The project has been the modelafter which other donor-funded projects on decentralized basic education in Indonesiawere patterned. The United States Agency for International Development implementeda similar project which was patterned after the project. ADBs Country Assistance Plan,20012003 for Indonesia also stated that ADB, UNICEF, and the World Bank sharedsector support in education on a geographic basis (footnote 3). For instance, the

project excluded Flores Island in NTT because the Australian government was alreadyimplementing a primary education project in that area at the time.

53. The projects scope was relevant to the objective of improving the quality ofand access to basic education, particularly among the poor. The changes in scope,which saw five districts in Jakarta withdraw and the corresponding funding reallocatedto two districts in NTT, made the project even more relevant. NTT and NTB are amongthe poorest provinces in Indonesia (Table 3).

Table 3: Poverty Incidence and Annual Changes in Poverty Rates in Indonesia ( )

Province

Poverty Incidence Average Annual

Change (2006

2011)996 2006 2011

( ) Rank ( ) Rank ( ) Rank ( ) Rank

Jakarta 2.4 1 4.6 1 3.8 1 3.6 32Bali

7.8 2 7.1 2 4.2 2 8.1 3SouthKalimantan 8.5 3 8.3 3 5.3 3 7.3 8BangkaBelitung 10.9 5 5.8 4 9.5 1Banten 9.8 4 6.3 5 7.1 10CentralKalimantan 13.5 4 11.0 6 6.6 6 8.1 5EastKalimantan 9.7 6 11.4 8 6.8 7 8.1 4RiauIslands 12.2 11 7.4 8 7.8 6Riau 12.6 8 11.9 10 8.5 9 5.7 19NorthSulawesi 17.9 7 11.5 9 8.5 10 5.3 25WestKalimantan 24.2 13 15.2 17 8.6 11 8.7 2Jambi 14.6 5 11.4 7 8.7 12 4.8 28WestSumatra 9.8 9 12.5 12 9.0 13 5.5 21

NorthMaluku 12.7 13 9.2 14 5.6 20SouthSulawesi 16.7 11 14.6 15 10.3 15 5.9 17WestJava 11.1 10 14.5 14 10.7 16 5.3 24NorthSumatra 13.2 12 15.0 16 11.3 17 4.9 27Indonesia

17.6 17.8 12.5 5.9

WestSulawesi 20.7 19 13.9 18 6.6 11EastJava 22.1 16 21.1 21 14.2 19 6.5 13SouthSumatra 15.9 15 21.0 20 14.2 20 6.4 14SoutheastSulawesi 29.2 20 23.4 25 14.6 21 7.5 7CentralJava 21.6 17 22.2 22 15.8 22 5.8 18CentralSulawesi 22.3 21 23.6 26 15.8 23 6.6 12DIYogyakarta 18.4 14 19.2 18 16.1 24 3.2 33Lampung 25.6 18 22.8 23 16.9 25 5.1 26Bengkulu 16.7 19 23.0 24 17.5 26 4.8 29

Gorontalo 29.1 29 18.8 27 7.1 9Aceh 12.7 23 28.3 28 19.6 28 6.2 15Nusa

Tenggara

Barat 32.0 22 27.2 27 19.7 29 5.5 23Nusa

Tenggara

Timur 38.9 24 29.3 30 21.2 30 5.5 22Maluku 44.6 25 33.0 31 23.0 31 6.1 16WestPapua 41.3 32 31.9 32 4.6 31Papua 42.3 26 41.5 33 32.0 33 4.6 30Sources: United States Agency for International Development and DAI/Nathan Group. 2013. Provincial PovertyRates in Indonesia, 20062011. Jakarta.

The projects

scope was

relevant to the

objective of

improving the

quality of and

access to basic

education

particularly

among the poor


33/71


34/71


Figure 5: Percentage of Poor

in Nusa Tenggara Timur Districts, 2010

Source: Secretariat of the National Team for the Acceleration of Poverty Reduction, Office of the VicePresident of the Republic of Indonesia. 2011.

55. Although the projects design was modified in the early stages ofimplementation, it remained relevant to the projects objective of improving poorpeoples access to basic education. The four-tier criteria for school selection (forprimary and junior secondary schools and madrasah) were simplified to three tiers, inwhich subdistricts, villages, and schools were ranked. The targeting criteria wereeventually simplified further for both the loan and grant to a one-tier procedure basedon the poverty ranking of schools within each district. Similarly, demand-led financing

was modified, and all schools were given standard amounts of block grant forrehabilitation and quality improvement.

56. These changes in design did not appear to compromise project relevance,particularly in targeting. MONE first selected the poorest districts and then the poorestsubdistricts within these impoverished districts. In selecting schools, district educationofficials conducted site visits to check the actual conditions on the ground and the levelof community involvement. Interviews with MONE, school, and district officialsindicated that schools in areas where community involvement needs improvement wereoften considered candidates as project schools. MONE pointed out that the Ministry ofReligious Affairs (MORAs) weak regulation of madrasah posed challenges duringproject implementation, noting MORAs lack of clear guidelines for the establishment

of new madrasah.

25

According to interviews with stakeholders, since the projectprovided grants that a school could use depending on its need, some new schools were

25During the study teams interviews, MONE said it follows certain criteria in establishing public schools,examining first the number of prospective students and the number of students already absorbed byexisting schools. For example, there should be at least three primary schools in an area to allow for theconstruction of a junior secondary school. Moreover, the distance between the new and existing juniorsecondary school should be 5 kilometers. Unless existing schools do not meet minimum service standardsor cannot accommodate the number of students, the request for a new school is rejected. In contrast,according to interviews with MONE officials, Islamic schools were built even if they were just 100 metersaway from an existing one.

41.1

34.032.8 32.4 31.7

29.928.7

26.7 25.9

22.9 22.721.6 21.2 20.8 20.4

15.513.4 12.7 12.1

10.69.6

23.5

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0


35/71


built in order to access the block grant. In Lombok, anecdotal evidence reveals that oneprimary Islamic school split into two. To address this problem, MONE issued a policythat only schools already existing for 5 years can access the block grant.

57. In selecting poor students, schools usually obtained student socioeconomicdata during enrollment, including household income and parents employment status.

Students who are orphans are often accorded priority, according to interviews withstakeholders. School officials also conducted house visits to verify students livingconditions. Since the grants provided under the project are relatively small, it wasimperative to be highly selective of the project districts and schools, as the interviewsreveal.

58. The projects design was also able to accommodate emergency needs forschool infrastructure rehabilitation in the aftermath of the earthquake and tsunamithat hit Aceh and North Sumatra in December 2004. As discussed in Chapter 2, loanand grant savings were used to rebuild damaged schools. About 42 schools werereconstructed in the affected areas and a model for earthquake-resistant schoolbuildings was developed.

C.

Effectiveness

59. The project is assessed as effective. This evaluation found evidence thateducation outcomes, including enrollment, completion, transition, and performance inselected beneficiary districts, improved after project implementation. Interviewedstakeholders deemed enhanced school capacity to formulate school development plansand greater community involvement in education concerns to be the projects mostimportant contributions.

60. Improved education outcomes. Access to basic education improved in thebeneficiary provinces (Table 4). Between 2001 and 2011, net enrollment rates atprimary level increased in NTB and NTT but slightly decreased in Bali. Primary netenrollment in NTT and Bali remained slightly below the national average in 2011. Net

enrollment at the secondary level increased in all three provinces during 20012011.NTTs secondary net enrollment increased from 38.0% in 2001 to 60.9% in 2011 butremained below the national average over the last decade.

61. Completion rates have also improved. NTT saw a notable increase in thesecondary completion rate from 92.8% in 2001 to 97.7% in 2011. However, bothprimary and secondary completion rates in NTT and NTB remained below the nationalaverage in 2011, while Balis were higher than the national average in that same year(Table 5).


36/71

Table 4: Net Enrollment Rates in Selected Provinces, 20012011

(%)Item 2001 2002 2003 2004 2005 2006 2007 2008

Bali

Primary 93.02 92.68 91.73 93.67 93.36 93.33 94.02 94.78Secondary 69.19 67.97 69.12 70.75 69.75 70.15 73.97 67.63Nusa Tenggara Barat

Primary 91.74 92.65 92.20 91.58 93.32 94.50 95.47 94.50Secondary 56.95 60.28 61.21 61.08 69.38 69.62 67.00 71.66Nusa Tenggara Timur

Primary 88.52 87.44 88.88 90.68 92.16 91.58 91.57 92.01Secondary 38.00 40.89 41.16 44.22 43.30 47.23 38.68 49.71Indonesia

Primary 92.88 92.70 92.55 93.04 93.25 93.54 93.66 94.15Secondary 60.47 61.69 63.49 65.24 65.37 66.52 66.13 67.26

Source: Study teams calculations based on national socioeconomic survey data.

Table 5: Completion Rates in Selected Provinces, 20012011

(%)Item 2001 2002 2003 2004 2005 2006 2007 2008

Bali

Primary 98.61 98.56 98.77 98.94 98.94 98.83 99.12 99.28Secondary 97.83 97.57 98.16 97.77 97.32 96.90 99.14 93.12Nusa Tenggara Barat

Primary 98.35 98.26 97.59 97.90 98.77 98.57 98.76 99.16Secondary 92.62 94.79 94.89 95.48 97.01 97.22 97.29 95.60Nusa Tenggara Timur

Primary 97.70 97.52 97.74 98.19 98.62 98.38 97.82 98.88Secondary 92.77 97.11 96.40 97.30 97.31 97.17 97.88 95.37Indonesia

Primary 98.54 98.27 98.54 98.68 98.92 98.70 99.04 99.18Secondary 96.16 97.09 97.18 97.50 96.92 97.11 97.08 95.66Source: Study teams calculations based on national socioeconomic survey data.


37/71


62. National exam scores also improved, indicating better student performance.26Although average scores in Bali and NTB were lower than the national average in 2001,they were higher than that average in 2012. NTTs average exam scores were lowerthan the national average in 2012 (Table 6).

Table 6: National Exam Scores in Selected Provinces

Province

Average

(2001)

Average

(2012)

Percentage

Increase

Bali 6.48 8.06 24.38

Nusa Tenggara Barat 6.32 7.87 24.52

Nusa Tenggara Timur 6.83

Indonesia 6.54 7.48 14.37

Sources: Study teams calculations based on data from Indonesian Familyand Life Survey and Ministry of National Education.

63. Improvements in enrollment, primary completion, junior secondary transition,and exam scores were observed in selected project districts after the project wasimplemented. Tables 7 and 8 present the education outcomes in 56 surveyed projectschools located in six districts before and after project implementation. Theseimprovements were observed both for poor students as a group and for all students.According to stakeholder interviews, the project addressed constraints on access tobasic education, such as the poor condition of basic infrastructure and school facilitiesthat largely prevented children from going to school or held down their national examscores.

26For primary school, the national examination consists of 3 subjects: mathematics, Indonesian language,and science. Each has a maximum score of 10, hence a students maximum score on the nationalexamination will be 30. For junior secondary school, there are 4 subjects, namely mathematics, Indonesianlanguage, science, and English (maximum of 40). Prior to 2003, the subjects examined at national levelwere 5 for primary school (moral education, Indonesian language, mathematics, social science, andscience), and 6 for secondary school (moral education, Indonesian language, mathematics, social science,science, and English).


38/71


39/71


Specifically the

is sufficient

evidence that

average

education

outcomes in the

selected district

were significant

higher after the

project was

implemented

than before it

was carried out

64. A paired t-test analysis was made to determine whether the changes in theeducation outcomes of the surveyed districts were statistically significant. The statisticalanalysis was used to determine whether the mean education outcomes improved afterthe project was implemented. It was carried out using data from 56 project schoolsselected using both probabilistic and non-probabilistic approaches to ensure that eachschool in each province had an equal opportunity to be selected as part of the sample.

The impact of the project on education outcomes was examined in the selectedschools, doing so for all students and for poor students as a group.27

65. The findings suggest that the project had an impact on education outcomes forall students in those districts examined (Table 9). Specifically, there is sufficientevidence that average education outcomes in the selected districts were significantlyhigher after the project was implemented than before it was carried out.28Increasedenrollment indicates that students had greater access to basic education, whileincreased transition and completion indicate greater opportunities for these students tofinish basic education. Improved performance can also be observed in a rise in nationalexam scores. Among school participation outcomes, the highest average difference wasobserved on completion, followed by transition.

Table 9: Paired t-test for All Students at District Level

Variable

Average

Before

DBEP

Average

After

DBEP Difference

Standard

Error

t-

statistic

Probability

(T


40/71


67. School development. The project developed the schools capacity indetermining priority education challenges and identifying corresponding strategies toaddress them. It provided school officials with training on how to formulate schooldevelopment plans, involve communities in preparation of the plans, and monitorprogress of the schools development. The interviewed principals noted that some ADBconsultants were easily accessible for consultation and queries even beyond working

hours. Given these contributions, the project effectively paved the way forimplementation of school-based management in Bali, NTB, and NTT. Just 2 years intoproject implementation, such management was made official national policy in 2003.

68. The project has also effectively changed the mind-set of both school- anddistrict-level officials in implementing school-based management. The projectscondition requiring schools to develop school development plans before they couldaccess block grants bolstered the capacity and decision-making skills at school level.School development plans have become the most important consideration whenselecting priority school programs, according to the interviews. School developmentplans examined by the evaluation team were deemed comprehensive. Their contentcovered the identified education problems, corresponding strategies to address theseproblems, work schedule, budget, and monitoring schedule.

69. The flexibility in the use of the block grants helped enhance the capacity ofschools in basic education management. Such flexibility allowed project schools todetermine priority needs and design innovative activities to address them, as interviewswith stakeholders revealed. In contrast, such other transfers from the centralgovernment to schools as School Operational Assistance Program (BOS) grants employmore rigid regulations regarding those activities they can be used to fund.

70. While the projects block grants were mostly used for rehabilitation of schoolfacilities, some schools adopted innovative approaches for school improvement.Anecdotal evidence gathered by the evaluation team reveals that a school in Anjani,East Lombok used the block grant to address malnutrition among its students, 30.0%of whom were found to be malnourished following medical checkups. The schooltherefore used the block grant in conducting a lunch-feeding program to address thisproblem. Some schools in Bangli, Bali used their block grants to send teachers toremote communities. Since these communities had no schools and children had to walkseveral miles to reach the nearest school, these schools used the grants to dispatchteachers who went to and taught children in these remote areas. According to theWorld Banks school-based management national survey in Indonesia from 2010,budget allocation is one of those areas where school principals perceived they havehigh autonomy in decision making. On average, 90.0% of principals said they haddecision-making authority in allocating the school budget, recruiting teachers, settingthe school vision and curriculum, and selecting textbooks and teaching materials.29

71. The project was also effective in inducing communities to participate actively in

school development through the formation of school committees. School committeesusually are comprised of community leaders, local entrepreneurs, teachers, parents,and heads of villages or districts. According to the World Bank survey, parents accountfor roughly 75.0% of school committee members, followed by council representativesat 20.0% and teachers at 4.0%. The representation of the stakeholders is even more

29The World Bank school-based management national survey used data from a nationally representativesurvey of about 400 public primary schools in Indonesia. The World Bank commissioned the RANDCorporation to conduct the survey. See Karam, K., Marshall, J., and Vernez, G. 2012. Implementation ofSchool-based Management in Indonesia. RAND Corporation.

While the

projects block

grants were

mostly used for

rehabilitation of

school facilities

some schools

adopted

innovative

approaches for

school

improvement


41/71


varied, as parents also serve in such other functions as religious leaders orrepresentatives of nongovernment organizations (footnote 29).

72. School committees were actively engaged in the formulation and subsequentrevisions, if necessary, of school development plans. Although schools were notrequired to report on achievements against their school development plans, according

to interviews with stakeholders, school committees, along with principals, localgovernment officials, and ADB staff and consultants, contributed to the formulationand subsequent revisions, if necessary, of the plans.

73. Local ownership of the project as fostered by community participation in schooldevelopment also increased the communities commitment to contribute to the projectvoluntarily. Through the school committees, community members contributed laborand other in-kind assistance to support school projects. Stakeholders also deemedcommunity involvement as having a significant effect on education quality. With theintroduction of BOS, however, schools have been expected to eliminate tuition fees orother compulsory payments from parents.30To some extent, this might have inhibitedsome parents who would like to assist the school financially. Nevertheless, the centralgovernment still permits schools to receive voluntary contributions from community

members for school development.

74. District basic education development. Interviewed stakeholders reported thatdistrict education offices primary role under the project dealt with the provision oftraining, particularly in teacher development and evaluation of school developmentplans. The district education offices were much involved in evaluating schooldevelopment plans and accordingly provided feedback to schools. Monitoring duties bysupervisors and district staff also included providing feedback on principal and teacherperformance, checking on the conditions of school facilities, monitoring or observingclassrooms and instruction, assessing teacher training needs, and reviewing andapproving the school budget (footnote 29).

75. The District Education Development Fund had been used to finance activitiesrelated to the district education officers capacity building such as trainings orseminars. The fund had also been used to assist teachers and school principals inimproving the quality of education. Its support for training teachers, school principals,and supervisors included contextual teaching and learning, in-service training,competency-based training, mastery of teaching and learning materials, evaluation,and classroom management, among others. The training was provided to both projectand non-project schools in a district, expanding capacity in both. Interviews withstakeholders revealed that district education boards are still in place as they aremandated under Ministerial Regulation No. 044/2002. As noted in the PCR, however, anew regulation on the management of education (PP 17/2010) made funding of thedistrict education boards by districts discretionary and reduced the number of boardmembers.

76. The projects impact on capacitating district education offices to produce 5-year district education development plans and 5-year annual action plans was lessnotable than that of other outputs. As noted in the PCR, the district educationdevelopment plans were supposed to be revised annually, but this was not done in alldistricts. Instead, districts base their annual implementation on the governmentsannual work plan, in line with government procedures. Moreover, only a few districts

30 World Bank. 2010. Supporting the BOS Program. Education Update. Issue No. 01. Jakarta


42/71


integrate school development plans into their annual work plans. In 2012, just 12.0%of districts consolidated school development plans for use in their district educationplanning process.31 Despite this, those district officials interviewed said the trainingthey received in formulating development plans proved useful, as district educationoffices are required by law since 2004 to create development plans.

77. Challenges at the district level on decentralizing basic education have beenobserved. One of thesehigh staff turnover at the district levelhas hampered theprojects capacity building efforts. Local district leaders, including mayors or heads ofdistrict education offices, have full authority to move officers in various units ordepartments, including principals and district education staff. In some districts, therewas insufficient knowledge transfer. However, in some districts in which the projectwas supported by a team of district education staff, the remaining staff members wereable to keep up with progress.

78. Second, some districts have too many teachers and others too few sinceteacher deployment and mobilization have also come under the authority of everydistrict government. Moreover, education appointments are perceived to be politicizedin that they are made by district heads. According to interviewees, some staff

appointments are based on political ties.

79. Third, district governments have yet to maximize their education budgets and ahuge portion is spent on personnel. District governments allocate some 30%40% oftheir budgets to education. Because around 85% of their education budgets is spent onpersonnel costs, limited funds are left for education development.32

80. Fourth, there are flaws in information systems that project districts use to trackand publish statistics. Data-keeping has been a problem, as documents are discarded ordeleted after 5 years due to storage issues. Stakeholders reported that no soft copies ofdata are available.

31The figure is based on a World Bank survey of 50 Indonesian districts on the quality of local educationgovernance. See World Bank. 2013. Local Governance and Education Performance: A Survey of the Qualityof Local Education Governance in 50 Indonesian Districts. Jakarta.

32 Ministry of Education and Culture. 2013. Overview of the Education Sector in Indonesia 2012:Achievements and Challenges.Jakarta.


43/71


D. Efficiency

81. The project is rated efficient. Despite the limited amount of grants, the projectachieved notable results in enhancing local capacity for education governance andimproving school infrastructure. Especially considering that the grants provided underthe project were spread over a 3-year period, some stakeholders noted these to be

quite small. Interviews with Ministry of National Education, school, and district officialsrevealed that some Rp20 million to Rp25 million is usually required to rehabilitate oneclassroom in an elementary school (measuring 8 x 7 meters) and Rp30 million to Rp35million for one classroom in a junior secondary school (measuring 9 x 7 meters). Since ablock grant amounted to around Rp90 million rupiah, three to four classrooms couldbe rehabilitated. The PCR nevertheless pointed out that junior secondary schoolfacilities improved, in particular within NTT, with 80% of junior secondary schoolshaving administrative offices and 60% having libraries and science laboratories.Moreover, community support helped lower the costs of rehabilitating facilities, assome parents or school committee members offered in-kind assistance such as freelabor. In addition to contributing to school infrastructure, the project made hugecontributions in improving schools and districts capacity for local education delivery.

82. Indeed, it is impossible for the project to address all the education-relatedinfrastructure requirements in Bali, NTB, and NTT. Continued rehabilitation of schoolfacilities is imperative, and particularly for those in impoverished areas such as a schoolin the city of Kefamenanu, district of Timor Tenga Utara, in NTT, which the evaluationteam visited and found to have some classrooms that were badly damaged (Figure 6).As stated, the government has recently scaled up its efforts to rehabilitate schools inthe country.

Figure 6: Condition of Classrooms in Timor Tengah Utara,

Nusa Tenggara Timur, 2014

Source: Study team, taken during the evaluation mission.

Despite the

limited amount

of grants the

project achieved

notable results i

enhancing local

capacity for

education

governance and

improving schoo

infrastructure


44/71


83. The PCR pointed out that progress under the loan was initially slow, but thepace did pick up throughout execution. On the other hand, the grant project requiredtwo extensions to achieve project objectives. The grants shorter implementation periodof only 2.5 years proved inadequate for severely poor and often inaccessible districts.As noted in the PCR, funds were generally disbursed on time and the full budgetallocation, as well as most of the additional funds made available by the rising

exchange rate, was used each year. There had been an 11.0% increase from theappraisal amount of the project.33The increase was actually 29.6% of the base cost ofthe project, the largest share of which was attributed to the payment for internationalconsultants that was 127.0% of the appraisal amount. The person-months of theconsultants, on the other hand, increased by only 12.8% (from 78 person-months atappraisal to an actual 88 person-months).

84. The external auditor found a few problems with procurement. Most of thoseproblems identified related to overpayment for workshops and training. Under the loancomponent, 57 audit casesrelating mostly to fund misuse due to overpayment,unclaimed penalties, and undeposited bank interestwere identified. Under the grantcomponent, 18 cases were identified. All 75 identified cases (involving less than 1.5%of schools) were resolved.

E. Sustainability

85. The project is rated likely sustainable. Its contributions to improving capacityfor local education governance are reinforced by continued transfer of knowledgebetween school and district officials who received training and their