Evaluating a test-based subsidy program for low-cost private schools:

Regression-discontinuity evidence from Pakistan

Felipe Barrera-OsorioDhushyanth Raju

(The World Bank)

Overview of the study:Strategy and Results

• Strategy of estimation– Since entry into the program is a function of a minimum student pass

rates in the test (average at the school level), we used a Regression Discontinuity approach in two entry school waves (Phases 3 and 4)

– We used both administrative and own collected data via phone survey to schools near the discontinuity point

• Results– For Phase-4 applicants (sharp design), we find large positive impacts on

the number of students, teachers, classrooms and blackboards. Students to teacher ratio remains constant. Impact emerging within a short period

– Using Phase-3 (fuzzy design), we did not find statistically significant impacts.

2

Program setting

• In general, Pakistan has poor performance in educational indicators– Participation rate in formal school (grade 1+) of children ages 6-15 was 65.7%– While 91.8% of individuals ages 12-16 completed primary school (grade 5), only 40.7%

completed secondary school (grade 10) – Mean scores in national assessments less than 50%

• Public sector the dominant provider of education with weak accountability and weak incentive systems.

• Dramatic growth since mid 90’s of the private school sector (Andrabi et al 2007; Andrabi et al 2006; Das, 2006)– 25% of school-goers (6-17 year olds) in private school– Specially for the poor and in rural areas– Fees in private school are generally low: median annual fees per student in 2000 were

960 (rural area) and 751 (urban) rupees

• The government formed an partnership with the private sector in order to increase enrollment and quality of education, specially for disadvantaged households 3

Foundation-Assisted Schools (FAS) program structure and coverage

• Administration: Designed and administered by the Punjab Education Foundation, a semi-autonomous organization

• Objectives: Increase school participation and achievement levels among children from disadvantaged households

• Timeline: Program established in 2005 and expanded in phases (4 phases completed).

• Coverage: – Districts: 18 out of the 35 districts in Punjab; 87% of schools in 7

districts;– Schools: 1,082 low-cost private schools; – Students: 474,000

4

FAS program: eligibility conditions

1. Initial benefit eligibility conditionsa. Minimum enrollment size of 100 students.b. Pass a qualitative inspection conducted by PEF.c. At least 67% of students tested have to score 33% or higher on the test.

2. Continued benefit eligibility conditionsa. Maintain minimum enrollment size of 100 students.b. Eliminate all tuition and fees for all students: signboard outside school

gate which announces tuition-free schooling and provides PEF contact information.

c. Report enrollment figures to PEF every month.d. At least 67% of students tested have to score 40% or higher on the test.

First violation: Penalties (e.g., frozen enrollment). Second violation: Permanent program disqualification.

5

FAS program: Benefits

• Subsidy: Rs. 300 (US$4.3) per student per month. • Subsidy level set at upper-end of price range for low-cost sector. • Use of subsidy largely unfettered.

• Teacher bonus: Rs. 10,000 (US$143) per teacher per year for 5 teachers in schools in which at least 90% of students in tested classes obtain a score of 40% or higher in the Quality Assurance Test. • 370% of mean monthly teacher salary at baseline.

• School bonus: Rs. 50,000 (US$714) to the school in each district

with the highest pass rate in the QAT per year. • 76% of mean monthly subsidy payment to schools given mean

enrollment size at baseline.6

FAS program: Tests

• The entry test: Short-listing Quality Assurance Test (SLQAT): – To be part of the program, at least 67% of students tested have to

score 33% or higher in the SLQAT– Pared-down version of QAT– Developed and administered by PEF.– Offered to two to three grades in a unannounced visit to the

school

• Continuation test: Quality Assurance Test (QAT):– For continuation in the program, at least 67% of students in

tested grades have to score 40% or higher in the QAT – Based on samples of syllabi and textbooks used in low-cost

private schools.– Selected subjects: English, Urdu, Mathematics, and Science.– Offered twice a year in two or three grades 7

FAS Program: Application process and Number of schools

8

Number of schools PHASE 3

PHASE 4

Identification• Entry process:

– Schools apply when a call for applications is issued. – Schools with qualifying applications are subject to a physical

inspection – In phase-3 and phase-4, schools that pass the physical inspection

are offered the SLQAT

• Assignment to FAS program (treatment) based ultimately on school’s pass rate on SLQAT– To be part of the program, at least 67% of students tested have

to score 33% or higher in the SLQAT– Virtually all schools that attained at least the cutoff, accepted

treatment (take-up rate of 94% in Phase 3 and 98%, in Phase 4)– No treatment dropouts to date

9

Local smoothness in baseline mean outcomes

• Comparison of means for those that marginally fail the SLQAT and for those that marginally pass: no statistically-significant differences, except for mean SLQAT score (scores positive correlated with pass rates).

• LLR estimates: Some statistically-significant breaks but nothing systematic to suggest a breakdown of the conditional smoothness condition for RD identification.

10

Data used for the evaluation• Data before the program

– Source: Administrative data.– Data on school characteristics and non-learning outcomes

obtained from applications maintained electronically by PEF.– Data on SLQAT pass rates and average school test performance

obtained from individual student test score data maintained electronically by PEF

– Number of schools: 747 (Phase 3) and 830 (Phase 4)

• Data after the program– Source: Phone interviews (schools phone numbers in

applications)– Data collected on school characteristics– Number of schools (just passers and just failers): Phase 3: 192.

Phase 4: 236 – Data collected 14 and 10 months after first subsidy payment to phase-

3 and phase-4 entrants (partially spans two academic years)11

Main findings

• Large positive effects on enrollment, teachers, classrooms, and blackboards; within short treatment period of 10 months

• Conservative estimates: +85 children (37% relative to baseline mean); +3.4 teachers (37%); +4 classrooms (47%); and +2.8 blackboards (27%)

• The annual cost per student of increasing enrollment by 1% is 97 rupees (US$1.4)– Among the lowest cost-effectiveness ratios estimated

Ongoing studies• More evaluation research is planned with several data sources

– Annual public school censuses– The Punjab Examination Commission test score data – Additional FAS program administrative data from PEF– Primary survey data from samples of low-cost private schools

• Some example of key questions– What are the impacts of the FAS program on student learning– Does the FAS program have any impacts on the socioeconomic

composition of their students (particularly in terms of gender and economic background)

– Does the FAS program induce the displacement of students from neighboring public schools and to what extent