Impact evaluations in education - World Bank

EDUCATION STAFF DEVELOPMENT PROGRAM

Impact evaluations in education

Evaluating effectiveness in promoting Learning for All

Deon Filmer, Lead Economist

AFTHD

The World Bank

HD Learning Week 2013


• Education is well suited to impact evaluation

– Intervention can be applied at school, classroom, or student level

– Clear, easy-to-grasp outcome measures

– Contrast with trade or macro policy

• Result

– Pioneering IE work among academics in education

– Most active World Bank sector for IEs


HOW DO WE KNOW WHICH INTERVENTIONS ARE INCREASING ENROLMENT AND COMPLETION?


Inequalities related to poverty: Grade 6 completion of 15-19 year olds in the richest and poorest quintiles.

Poorest quintile East Asia/Pacific Europe/Central Asia Latin America/Caribbean Middle East/North Africa South Asia Sub-Saharan Africa

Within-country inequalities are as big—if not bigger—than cross-country inequalities

Source: Filmer (2010)


Why are participation and completion low?

• Costs of schooling

– Poverty and cash/credit constraints

– Opportunity costs

– Exclusion

• Benefits of schooling

– Quality

– Perceived returns


Example #1:

Progresa/Oportunidades CCT in Mexico – RCT

How much do conditional cash transfers increase enrolment?


Progresa/Oportunidades CCT program in Mexico

• Large-scale program, with one of first major rigorous impact evaluations

• Multisectoral

– Cash to families based on both schooling and health conditions

• Carefully designed for rigorous impact evaluation:

– Phased in with a randomized control trial (RCT) design

– Can compare outcomes in villages with the program to villages without the program


• Small impact at primary grades– Baseline enrollment grades 0-5: 94%

– Impact on enrollment: 1.9

• Large impact at transition to secondary school– Baseline enrollment grade 6: 45%

– Impact on subsequent enrollment: 8.7

• Small impact at secondary level:– Baseline enrollment grades 7-9: 43%

– Impact on subsequent enrollment: 0.6

Source: Schultz (2004)

Impact evaluation evidence from a CCT program: Mexico


Example #2:

Cambodia scholarship program – Discontinuity design

How effective are poverty-targeted scholarships at increasing enrolment?


Cambodian Secondary School Scholarship Program background

• Targeted 100 lower secondary schools

• Payments conditioned on school attendance and progress

• Implementation design lends itself to rigorous impact evaluation:– Students in feeder schools ranked by a “dropout-

risk” score

– Secondary school applicants offered $60 or $45 scholarship (or none), depending on risk

– Enabled RDD design


Results from first follow-up: Impact of scholarships on attendance and enrollment

• Attendance from school visits • Enrollment from household survey

0

0.2

0.4

0.6

0.8

1

-25 -15 -5 5 15 25

Relative ranking

Pro

ba

bili

ty

Recipients Non-recipients

0

0.2

0.4

0.6

0.8

1

-25 -15 -5 5 15 25

Relative ranking

Pro

ba

bili

ty


Estimate of impact

Estimate of impact

Large positive impact of program on attendance and enrollment (about 25 percentage point increase)

Source: Filmer and Schady (2009)


0

0.2

0.4

0.6

0.8

1

-25 -15 -5 5 15 25

Relative ranking

Pro

ba

bili

ty


Comparing program alternatives: Impact of different scholarships amounts

• No scholarship versus $45 • $60 versus $45 scholarship

0

0.2

0.4

0.6

0.8

1

-25 -15 -5 5 15 25

Relative ranking

Pro

ba

bili

ty


Estimate of impact

Estimate of impact

First $45 is much more cost effective (almost nodiscernable additional impact of $60 over $45)


Results from second follow-up:Girls

Enro

llme

nt

Impact estimate

Non-recipient mean

Boys

Enro

llme

nt

Impact estimate

Non-recipient mean

Sizeable impacts beyond the scholarship period itself


Accumulation of IE evidence:Summary of CCT/Scholarship programs

Country Age/Grade/Gender Baseline enrollment

Impact

Chile Ages 6-15 60.7 7.5***

Colombia Ages 8-13Ages 14-17

91.763.2

2.1**5.6***

Ecuador Ages 6-17 75.2 10.3**

Honduras Ages 6-13 66.4 3.3***

Jamaica Ages 7-17 18 days out of 20

0.5**

Mexico Grades 0-5Grade 6

Grades 7-9

94.045.042.5

1.98.7***

0.6

Nicaragua (1) Ages 7-15 90.5 6.6***

Nicaragua (2) Ages 7-13 72.0 12.8***

Bangladesh Ages 11-18 (Girls) 44.1 12.0**

Cambodia (1) Grades 7-9 (Girls) 65.0 31.3***

Cambodia (2) Grades 7-9 65.0 21.4***

Pakistan Ages 10-14 (Girls) 29.0 11.1***

Turkey Primary schoolSecondary school

87.939.2

-3.0*5.2

Source: Fiszbein and Schady (2009)

• Accumulation of evidence allows us to recognize patterns

• Impacts tend to …

– Be larger when baseline enrollments are lower

– Be larger at transition points

– Be larger for poorer families


IMPACT EVALUATIONS FOCUSED ON HOW TO IMPROVE QUALITY AND LEARNING


Improving learning outcomes through both demand- and supply-side programs

• Increasing the benefits of schooling through:– Targeted scholarship programs

– Increases in teacher pay

– Provision of information to parents

Increasing use of rigorously designed studies to evaluate impact on learning; e.g., these RCT examples:Cambodia targeted primary-school scholarships

India learning-based teacher bonuses

Pakistan school report cards for parents


Example # 1

Primary-school scholarships in Cambodia: RCT

What type of scholarship targeting best promotes learning by recipients?


Primary Scholarships Program

• Built on secondary school scholarships program

• 209 schools in 3 Provinces

• Two forms of targeting within schools:

– Poverty: select the poorest applicants (similar to CESSP)

– Merit: select the applicants with the best test scores

• Focus on learning


Evaluation design

• Schools randomly assigned to

– Treatment (poverty or merit) cohort of students that began receiving scholarships in 2008/2009 school year

– Control same cohort did not receive scholarships

– (In the subsequent school year, the next cohort in both sets of schools began receiving scholarships)

Randomization means that we can easily compare outcomes between groups and interpret difference as impact


Impact results of primary scholarships (1)

Impact on … Poverty targeted scholarships

Merit targeted scholarships

… probability of completing grade 6

0.170 0.120

… years of schooling achieved

0.332 0.182

Primary school scholarships increased schooling attainment

Source: Barrera-Osorio and Filmer (2012), “Incentivizing schooling for learning: Evidence from Cambodia on the impact of alternative targeting approaches”


(Preliminary) Impact results of primary scholarships (2)

• Impacts on assessments of cognitive skills– Reminder: Applications December 08/January 2009 (Grade 4)

– Follow-up: May-July 2011 (Grade 6)

Math test Digitspan

Merit scholarships +0.170 sd +0.149 sd

Poverty scholarships Not sig. Not sig.

“Merit” recipients performed better on tests, “poverty recipients” did not


Example #2

Teacher bonus pay based on student learning in India: RCT

Do learning-based teacher bonuses improve student learning?


APRest teacher incentives experiment: Context and rationale

• Context: poor service delivery quality and learning outcomes in AP schools

• Opportunity: AP government willing to experiment with innovative potential solutions

• Theory of change: Teachers motivated to work harder and focus on student learning results

EDUCATION STAFF DEVELOPMENT PROGRAMEDUCATION STAFF DEVELOPMENT PROGRAM


India AP: Summary of results

• Incentive schools performed significantly better (0.22 SD)– Improvements for all grades, districts, baseline scores– Improvements on mechanical and conceptual

components of test, and also on non-incentive subjects

• Individual incentives had more impact than group incentives after first year

• Channels: – No reduction in teacher absence, but:– Higher levels of teaching activity among teachers at

school


Example #3Information for accountability through report cards in Pakistan: RCT

Does providing information on student and school performance to parents improve student learning?


Information for accountability: Pakistan report cards

• Context: poor and varied learning results, in an active education market

• Intervention: provide report cards to parents giving information on child’s and school’s performance

• Theory of change: competitive pressure from informed parents can lead to improved quality and/or reduced tuitions in private schools


Pakistan: Report Card DesignChild information Village Schools Information

Three subjects (Math; Urdu; English)

- Child score and quintile

- Child’s School score and quintile

- Child’s village score and quintile

Quintile described as “needing a lot of work” to “very good”

For all Primary schools in villages:

- School name

- Number of tested children

- School scores and quintiles in all 3 subjects

Source: Andrabi, Das, and Khwaja, “Report Cards: The Impact of Providing School and Child Test-scores on Educational Markets” (2009).


Pakistan: Summary of Results

• Initially low-quality private schools:– Increase in learning outcomes (by 0.15 SD)

• Initially high-quality private schools:– Decrease in school fees (by 21 percent)

• Public schools:– Increase in learning outcomes (by 0.10 SD)

• Intermediate impacts:– more likely to have textbooks – devoted around 30 more minutes per day to teaching and

learning activities

• Importance of thinking about education market


Some other areas for IE in education

• School inputs (textbooks, uniforms)

• Contract teachers

• Teacher training


Summary

• Education – rich area for impact evaluation

– Possible to explore program effects on key outcomes of interest

– Large inventory of completed IEs in education, more underway

– But gaps remain (e.g., teacher training)

Documents

Impact evaluations in education - World Bank