39
1 Incentivizing schooling for learning: Evidence on the impact of alternative targeting approaches 1 Felipe Barrera-Osorio Harvard Graduate School of Education and Deon Filmer World Bank This version: October, 2012 Preliminary version. Please do not circulate. Abstract. Demand-side incentive programs such as scholarships or Conditional Cash Transfer programs have been shown to increase measures of school participation in a number of countries, although impacts on learning outcomes have been harder to identify. We evaluate the impact of a large scale primary school scholarship pilot program in Cambodia and show that the program increases school participation (enrollment and attendance)despite being targeted to some of the poorest and most remote areas. The program was designed to evaluate the equity and effectiveness implications of two alternative targeting approaches: in some randomly selected schools recipients were targeted on the basis of poverty, in others recipients were targeted on the basis of merit. While we show positive impacts on enrollment and school progression emerging from both targeting approaches, learning impacts are only detectable among merit- based recipients. We present evidence on student effort and household education investment compatible with the asymmetry in learning impacts. While there are some equity implications of a merit-based approach to targeting (the poverty-based approach unsurprisingly identifies a poorer group of recipients), the tradeoff is not particularly stark. Scaling up an approach that targets students with high academic potentialwhile ensuring that the poorest student are among that setis likely to be the approach that maximizes both equity and effectiveness objectives. JEL classification codes: I21; I24; I28; O10 Keywords: education; Cambodia; randomization; scholarships; merit-based targeting; poverty-based targeting. 1 We thank Luis Benveniste, Norbert Schady, Beng Simeth, and Tsuyoshi Fukoaka and the members of Primary School Scholarship Team of the Royal Government of Cambodia’s Ministry of Education for valuable input and assistance in carrying out this work. Adela Soliz provided able research assistance. The paper has also benefitted from comments by Muna Meky, Halsey Rogers, and Shwetlena Sabarwal. The authors are, of course, responsible for any errors. This work benefited from funding from the World Bank as well as through the EPDF Trust Fund (TF095245). The findings, interpretations, and conclusions expressed in this paper are those of the authors and do not necessarily represent the views of the World Bank, its Executive Directors, or the governments they represent.

Incentivizing schooling for learning: Evidence on the ... · Incentivizing schooling for learning: Evidence on the impact of alternative targeting approaches1 ... from comments by

Embed Size (px)

Citation preview

1

Incentivizing schooling for learning:

Evidence on the impact of alternative targeting approaches1

Felipe Barrera-Osorio

Harvard Graduate School of Education

and

Deon Filmer

World Bank

This version: October, 2012

Preliminary version. Please do not circulate.

Abstract. Demand-side incentive programs such as scholarships or Conditional Cash Transfer programs

have been shown to increase measures of school participation in a number of countries, although impacts

on learning outcomes have been harder to identify. We evaluate the impact of a large scale primary

school scholarship pilot program in Cambodia and show that the program increases school participation

(enrollment and attendance)—despite being targeted to some of the poorest and most remote areas. The

program was designed to evaluate the equity and effectiveness implications of two alternative targeting

approaches: in some randomly selected schools recipients were targeted on the basis of poverty, in others

recipients were targeted on the basis of merit. While we show positive impacts on enrollment and school

progression emerging from both targeting approaches, learning impacts are only detectable among merit-

based recipients. We present evidence on student effort and household education investment compatible

with the asymmetry in learning impacts. While there are some equity implications of a merit-based

approach to targeting (the poverty-based approach unsurprisingly identifies a poorer group of recipients),

the tradeoff is not particularly stark. Scaling up an approach that targets students with high academic

potential—while ensuring that the poorest student are among that set—is likely to be the approach that

maximizes both equity and effectiveness objectives.

JEL classification codes: I21; I24; I28; O10

Keywords: education; Cambodia; randomization; scholarships; merit-based targeting; poverty-based

targeting.

1 We thank Luis Benveniste, Norbert Schady, Beng Simeth, and Tsuyoshi Fukoaka and the members of Primary

School Scholarship Team of the Royal Government of Cambodia’s Ministry of Education for valuable input and

assistance in carrying out this work. Adela Soliz provided able research assistance. The paper has also benefitted

from comments by Muna Meky, Halsey Rogers, and Shwetlena Sabarwal. The authors are, of course, responsible

for any errors. This work benefited from funding from the World Bank as well as through the EPDF Trust Fund

(TF095245). The findings, interpretations, and conclusions expressed in this paper are those of the authors and do

not necessarily represent the views of the World Bank, its Executive Directors, or the governments they represent.

2

1. Introduction

There is a steadily growing international evidence-base on the impact of targeted cash transfers on a

range of outcomes. Conditional Cash Transfer (CCT) programs have gained popularity in much of the

developing world and in several countries are now the largest form of social assistance (see the review in

Fiszbein and Schady 2009). These programs, in which cash is transferred to families on the condition that

they comply with a set of conditions (typically that children enroll in—and regularly attend—school,

pregnant women make regular prenatal visits, young children are taken for regular health monitoring and

checkups visits), have been rigorously evaluated in many countries. Scholarship programs, such as the

one we evaluate here, can be thought of as single-child (i.e. individual recipient), single-sector (i.e.

education) CCTs.

Much of the rigorous evidence on the impact of CCTs have been middle-income and in the Latin

America region; the Cambodian evidence joins a much smaller set of evidence focused on low-income

countries, such as Bangladesh and Malawi.2 Moreover, the setting is one in which the primary education

survival rate is low due to high a dropout rate. Indeed, much of the evidence of the impact of CCTs at the

primary level is from countries where baseline enrollments are high, and impacts subsequently small (in

part because there is little room for increase). Establishing the extent to which cash transfers are effective

at this school level in this type of setting is an important contribution of this evaluation.

Despite this large body of evidence of the impact of CCTs on enrollment and attendance, there

has been limited evidence to date on the impact of these programs on test scores, and the few available

studies show mixed results: in the case of Cambodia (Filmer and Schady 2009) and Mexico (Behrman,

Parker and Todd 2005; Behrman, Sengupta, and Todd 2000), no results on achievement tests; in the case

of Kenya (Kremer, Miguel and Thornton 2009) and Malawi (Baird, McIntosh and Özler, 2011), positive

effects. The question of how to turn incentives for schooling into learning remains an open one.

An important feature of the program we evaluate here is that students were selected on the basis

of two alternative approaches. In some schools recipients were selected purely on the basis of poverty

(actually, a proxy thereof), but in other schools recipients were applicants who scored well on an

assessment test—i.e. a merit-based targeting approach. Kremer, Miguel and Thornton (2009) describe a

merit-based approach to targeting scholarships in a secondary school program in Kenya—and show

substantial impacts on both attendance and learning. Targeting students with high academic potential,

through the merit-based selection approach, might be a good approach if it maximizes the probability of

2 See Chaudhury and Parajuli (2008) and Baird, McIntosh and Ozler (2009).

3

having an impact on learning as well as enrollment—incentivizing schooling for learning. But, targeting

high performing students may come at the cost of reaching the poorest, since there is likely to be a

positive relationship between academic success and household economic status. Moreover, some authors

will argue strongly against merit scholarships based on equity considerations (see Orfield, 2002). The

question of whether the efficiency gain (defined narrowly as “getting more learning and enrollment per

dollar transferred”) comes at too great a cost in terms of not reaching the poor is one that this evaluation is

able to investigate.

This evaluation aims at addressing two key questions associated with this pilot program: first,

what is the measurable impact of these primary school scholarships on measures of school participation

and learning? And second, given a choice between targeting recipients based on poverty versus merit,

what are the potential tradeoffs in terms of impacts versus reaching the poorest? The evaluation shows

two main results. First, both targeting approaches cause higher enrollment and attendance rates, but only

the merit-based scholarship shows positive impact on learning, measured as result in test scores. Second,

while there are some equity implications of a merit-based approach to targeting (the poverty-based

approach unsurprisingly identifies a poorer group of recipients), the tradeoff is not particularly stark.

Before addressing the evaluation it is important to emphasize that we focus on a narrow set of

objectives, schooling and learning as measured by test scores. The program aimed to transfer cash to

poor households, which in itself is potentially welfare enhancing—and we do not address that directly.

Moreover, learning is but one objective of schooling—albeit an important one. There are additional

social as well as personal (for example, better health, delayed marriage) impacts of more schooling that

we are not addressing or evaluating here.3

In the next section we describe the setting, program and the evaluation. In section 3 we present

the empirical strategy and results. Finally, in section 4 we discuss the results and we present main

conclusions.

2. Country setting, program design, evaluation design, and data

Country setting

3 As discussed below, we focus below on the fact that the merit scholarship showed impacts on learning, while the

poverty scholarships did not. It is possible that poverty scholarships nevertheless have salutary impacts on other

outcomes through their impact on schooling.

4

Cambodia has a tradition of demand-side incentives intended to raise school enrollment and

attendance rates. While some of these operate at the primary level—such as school feeding programs, or

small scale programs that incentivize attendance for primary school children—the bulk of the programs

are targeted at the lower secondary school level. The largest of these programs have been the so call PB

Scholarships.4 The programs do not operate as simple “fee-waivers”; rather, the families of children

selected for a “scholarship” receive a small cash transfer, conditional on school enrollment, regular

attendance, and satisfactory grade progress.

Two rigorous evaluations of the impact of these programs have shown substantial increase in school

enrollment and attendance as a direct consequence of the programs.5 Recipients are on the order of 20 to

30 percentage points more likely to be enrolled and attending school as a result of the scholarships. The

evaluation of scholarships offered through the CESSP also showed that the scholarships targeted to lower

secondary school students led to more expenditures on education, and to less work for pay among

recipients. There were no negative spillover effects—either to non-recipients in schools or to ineligible

siblings in households. Impacts on learning outcomes were limited, pointing to issues of quality and the

match between students’ skill levels and the instruction they are receiving.

One important finding from previous programs, however, was that their targeting was only mildly

pro-poor. For example for CESSP scholarships, despite the fact that the program was able to reach the

poorest children who applied for the scholarships, the poorest of the poor have already dropped out of

school before grade 6—the point at which they would apply for secondary school scholarships. Figure 1

shows the proportion of children 15 to 19 who have completed each grade, based on nation-wide data:

clearly children from the poorest quintiles are much less likely to make it to 6th grade. This suggests that a

program that targets children at the end of grade 6 is not likely to be pro-poor—and that a program

targeted at poor students, earlier in the schooling cycle—is needed if the goal is to reach the poorest of the

poor.

Based in part on these findings, and on a desire to assess the viability, effectiveness and optimal

design of such a program, the Royal Government of Cambodia included a pilot primary school

4 This program, formerly called PAP12, is operated from the government’s Program Budget, the Japan Fund for

Poverty Reduction (JFPR) Scholarships funded by the Asian Development Bank and UNICEF, the Belgian

Education and Training Trust (BETT) Scholarships, and the Cambodia Education Sector Support Project (CESSP)

Scholarship Program which is funded through a World Bank project. Students who were receiving JFPR and BETT

scholarships, but who were threatened by a cessation of these scholarships because of lack of funds in the projects,

were ultimately covered by the CESSP program. 5 For the JFPR evaluation see Filmer and Schady (2008); for the CESSP evaluation see Filmer and Schady (2009)

and Ferreira, Filmer and Schady (2009).

5

scholarship program as a component of the activities funded by the Fast Track Initiative-Catalytic Fund

(FTI-CF) Grant that it received. The stated goal of the program was to increase schooling by offsetting

the direct and opportunity costs.6 Implicitly, the goal was also to improve learning outcomes through that

additional schooling. This paper reports the results of the impact evaluation of that pilot program.

Program and evaluation design

The basic design of the primary scholarship pilot was to select participating schools; and then

within schools identify scholarship recipients according to a clear and transparent criterion. Once

selected, recipients needed to stay enrolled, attend school regularly, and maintain passing grades in order

to keep the scholarship until they graduate from primary school.7 The program targeted students in

selected schools entering the upper-primary level (Grades 4, 5 and 6). The scholarship amount was set at

US$20 per student, per year.8 The scholarships were intended to be disbursed in two tranches of US$10

over the school year: once towards the beginning of the year, and once towards the middle. In the first

year of the program, scholarships were distributed in one lump sum due to delays in implementation.9

The pilot program was targeted to the three Provinces where average dropout rates between

grades 3 and 6 were highest, as determined by an analysis of Cambodia’s Education Information

Management System (EMIS). These Provinces were Mondulkiri, Ratanakiri and Preah Vihear. In order

to narrow the geographic scope of the program, 7 (of 9) districts in Ratanakiri with the highest dropout

rates were selected for participation, and all districts in the other Provinces were included. Within these

selected districts, all primary schools which offered classes through to grade 6 participated in the

program.

In order to evaluate program impact, 209 schools were randomly assigned to join the program in

its first year, 2008-09 (referred to as Phase 1 schools—104 schools), or in 2009-10, its second year (Phase

2—105 schools), as depicted in Figure 2. The identification of impact is based on the fact that among the

cohort of students studied, Grade 4 —at baseline— students in randomly selected Phase 2 schools were

6 Primary schools are officially non-fee based. Opportunity costs include various forms of child labor which are

relatively common in the areas under study—although typically labor is combined with schooling at the primary

school ages. 7 There is moderate enforcement of the conditionality. Students absent for many days are followed up by school

officials and if they return to school would remain eligible for the scholarship. After a student is absent for too

many days they would be classified as having dropped out and no longer be eligible for the scholarship. 8 CESSP lower-secondary scholarships were in the amounts of $45 and $60—however the evaluation found little

impact on enrollment and attendance of $60 over and above $45 (Filmer and Schady 2011) 9 Scholarships distributions for the cohort of recipients in Phase 1 schools analyzed here took place in July 2009

(US$20), November 2009 (US$10); April 2010 (US$10); November 2010 (US$10); and April 2011 (US$10).

6

not eligible for scholarships. These students therefore serve as a valid counterfactual group—a group that

differs, on average, from the treatment group only in that it did not receive the scholarships. Since these

students in control schools were never exposed to the program (even after the subsequent cohort became

eligible when scholarships were implemented in Phase 2 schools), the two groups of students can be

tracked over time and enrollment, attendance, and other outcomes compared.10

Schools were further randomly allocated to one of two groups in order to evaluate the

effectiveness of the alternative ways of targeting: “poverty-based” and “merit-based” targeting. The first

group of schools (“poverty-based” targeting) used a score, similar to that currently in use in the secondary

school programs (52 Phase 1 and 53 Phase 2 schools) —see Figure 2. All targeted students filled out a

simple form with questions relating to their household and family socio-economic characteristics.11

These forms were scored according to a strict formula based on weights derived from an analysis of

household survey data.12

Scoring of the actual application forms was carried out centrally by a firm

contracted specifically for this purpose, thereby reducing the ability to manipulate the program. Within

each school, the applicants with the highest scores (i.e. “the highest poverty”) were selected to be offered

a scholarship. In a second group of schools (“merit-based” targeting), applicants were ranked based on

scores on a test of learning achievement (52 Phase 1 and 52 Phase 2 schools). The test was adapted from

the Grade 3 National Learning Assessment which was developed under the Cambodia Education Sector

Support Project (CESSP) project.13

All eligible students took the test and, within each school, the

applicants with the highest test scores were selected to be offered a scholarship. Again, scoring the tests

was done centrally in order to minimize the risk of program manipulation. The number of students in each

school type was fixed exogenously, and set to half the number of registered students in the year prior to

the program (as determined by an analysis of EMIS data).14

10

Because scholarship offers are made according to a strict criterion within schools, applicants “just above” and

“just below” the cutoff for eligibility could be studied using a regression discontinuity design (RDD) approach to

evaluate impact. Future work based on this program will exploit that approach, and will be able to use data from the

first and second cohorts of students who applied to the program. 11

Table 1 reports the full set of variables included in the calculation of the score. 12

The weights were determined by estimating a model predicting the probability that a student would drop out of

school during grades 4 to 6—since addressing this dropout was the stated goal of the program. Strictly speaking,

therefore, the score should be referred to a “dropout-risk score”. However, the risk is essentially a set of household

characteristics that capture the socioeconomic status of a household—weighted to capture those elements that

predict dropout best. For convenience and ease of exposition, the score is referred to in this paper, as well as in

program documents, as a “poverty” score. 13

The National Assessment was implemented nationwide in Grade 3 in a sample of schools during the 2005/06

school year (Royal Government of Cambodia 2006). 14

The number of scholarships is not equal to half the number of applicants because (1) the rule was to allocate

scholarships to all applicants who had the cutoff score and ties mean that more applicants would receive a

scholarship offer, or (2) because of changes in enrollment numbers from year-to year.

7

Implementation and data

This study evaluates the impact of the program on first cohort of students—who filled out

application forms when the program began implementation in December 2008/January 2009. At the time,

these students were in the 4th grade.

15 All 4

th grade students in program schools filled out the application

forms as well as took the assessment test—both students in Phase 1 and Phase 2 schools, as well as

students in poverty-targeting and merit-targeting schools (see Figure 2). Recipients received scholarships

disbursements (on condition of remaining in school, attending regularly, and maintaining passing grades)

during the 2008/09, 2009/2010 and 2010/11 school years.

We use three main data sources to evaluate program impact. First, we use the full set of data

collected at the time students applied for the scholarships. That is, we have information on baseline

household characteristics as well as baseline math and Khmer language test scores for all applicants.

Second, we use the official list of students who were offered a scholarship. Third, we use endline data

that were collected specifically for this evaluation. These data are derived from a survey of a random

subsample of students from each program school that was administered at the end of the 2010/11 school

year, when a student who would have stayed in school in the correct grade would have been finishing (or

just finished) grade 6. The survey was administered in households (i.e. not in schools) to the child who

applied for the scholarship, and included a household module administered to their mother, father or other

caregiver. In total 3,618 applicants were interviewed.16

For the bulk of the analysis we use data from

1,377 students in grade 4 at baseline who were offered a poverty- or merit-based scholarship, or would

have been offered a poverty- or merit-based scholarship had they been going to a program school.

The survey asks about a broad range of issues, both in terms of school participation (for example

the “intensity” of school participation though questions relating to time spent in school), other activities

such as labor market participation, as well as various measures of cognitive development and learning

achievement. Importantly, given that these data are collected at the household level the questions can be

asked of both recipients and non-recipients whether they are in school or not. As such, another

contribution of this study is that it avoids the problem of selecting only children who are enrolled and

attending school when analyzing learning achievement data collected at the school (Kremer, Miguel and

15

These students were “supposed” to have filled out application forms prior to the beginning of the school year, i.e.

when they were still in 3rd

grade. Because of delays in effectiveness and implementation of the overall project, the

application process could only be implemented once the students had begun 4th

grade. 16

Attrition was 15%. Analysis of attrition patterns show that the share of attritors is not different by Phase 1/Phase

2 schools, nor is it related to poverty/merit status of the school.

8

Thornton, 2009). Three learning achievement tests were administered: a mathematics test, a Digitspan

test and Ravens Progressive Matrices test. 17

The items on the Mathematics test were drawn from a variety of sources including the baseline

mathematics test; questions from the national grade 6 assessment; questions drawn from publicly released

items from the Trends in International Maths and Science (TIMSS) Grade 4 Assessment. Items were

tested during a pretest and only items with adequate properties were retained for the final test. It is a

multiple choice test, measuring both knowledge and capacity to use this knowledge to solve specific

problems. Presumably, this is the measure of the most immediate academic impact of the intervention,

since exposure of the program can directly affect the ability to solve mathematical problems.

The Digitspan test is a test in which a series of numbers are read to a respondent who is then

asked to repeat the numbers back to the enumerator. The series increases from 2 numbers to a larger and

larger number, until 9 digits. Respondents are also asked to repeat the numbers back in reverse order.

The test is typically interpreted as a measure of short term memory and working memory capacity. In the

Ravens Progressive Matrices test respondents are shown a set of three images each with a pattern that

links to the others. They are then shown a set of potential images, one of which link to the three original

images, and are instructed to tell the enumerator which one “completes” the first three. The test is

typically interpreted as a measure of logical reasoning.

3. Empirical strategy

Empirical Strategy

We estimate the reduced-form of the program impact on enrollment and attendance outcomes; on

test scores; and on potential mechanisms of transmission (school / teacher effort and student / household

effort). The estimation is based on the equation

(1)

denotes the outcome variable for individual i at follow-up (t1); is in indicative of

treatment status; is a vector of controls measured at baseline; and captures unobserved students

17

A vocabulary test was also included. The results of this test were in general terms very imprecise and

tending towards zero—possibly because of problems in translating words and concepts into Khmer. For

the sake of simplicity, we do not report these results.

9

characteristics and idiosyncratic shocks. The controls includes baseline values for gender, number of

minors in the household, indicators for whether the household owns a motorcycle, a car/truck, an

oxen/buffalo, a pig, an ox or buffalo cart; indicators for whether the house has a hard roof, a hard wall, a

hard floor, an automatic toilet, a pit toilet, electricity, piped water; as well as the overall poverty index and

test scores at baseline.

The estimation is done separate for each targeting mechanism. Accordingly, for students in merit-

based targeting schools, is equal one if offered the merit-based scholarship, and zero for untreated

students in the control schools who would have been eligible for merit-based scholarships —based on

their test scores— had they attended a treatment school. An analogous treatment indicator is built for the

poverty-based targeting mechanism. Given that the treatment variable identifies at baseline all individuals

offered the scholarship, the estimation is in practice an intention to treat estimator (ITT). Errors are

clustered at school level, and each estimation includes a district level fixed effects. In order to gain

efficiency, we run seemly unrelated regressions (SUR) in estimating the impact on test scores.

The design of the intervention also allows for directly estimating the effects of the program on

non-treated students. In each treatment school, approximately half of the students were not treated.

Therefore, given the random assignment of schools into treatment, we can compare the non-poor (and

non-treated) students in the poverty treatment schools with non-poor (and non-treated) students in control

schools. Similarly, we can compare students who did not received scholarships in merit-treated schools

with control school students who would not have received merit-scholarships in control schools. In order

to estimate this, we used Equation (1), but replacing with the appropriate group of students. In this

program the scholarship is offered after the baseline test is done, and during the duration of the program,

non-scholarship recipients cannot change their status. Therefore, any effects on non-treated students

emanate from complementarities and interactions between treated and non-treated students during the

academic year.18

Baseline balance and characterization of the study sample

This section presents the general characteristics of the study sample and the validation of the

random assignment by comparing treatment and control students at the baseline. In Table 1, columns (1)

and (2) are based on all students in the control and the treatment schools, whereas columns (3)-(6) only

18

These peer effects are different in nature with the externality effects estimated in Kremer et al (2009).

The authors of that paper estimate the effect of the “promise” of scholarship on students with low scores

pre intervention. Those effects presumably emanate from the effort that all students may exercise in order

to get the scholarship.

10

use information of the treated students in the treatment schools and untreated students in the control

schools who would have been eligible for treatment —based on their poverty index score or on the

baseline test score—had they attended a treatment school. The sample is restricted to Grade 4 students at

baseline, and as described above, the control group of students was not part of the phase-in expansion of

the program. Columns (1), (3), (5) show the means and standard deviations of household and individual

characteristics prior to the intervention. Columns (2), (4) and (6) show the differences and standard errors

at baseline between the treatment and control groups. That is, the values are the coefficient estimates from

a regression of each characteristic on a dummy variable equal to 1 for a treatment school.

Two main features emerge from Table 1. First, treatment and control groups are similar in

observed characteristics. Only a few coefficients in columns (2), (4) and (6) are statistically significant (of

the 48 differences reported, 5 are significant at the 10% level, and among them, 1 is significant at the 1%

level). The results in Table 1 confirm the validity of the random assignment since both control and

treatment groups are similar in their observed characteristics. More important, in means, the poverty

index and the test score are equal between treatment and control groups. Figures 3 and 4 present the

density of poverty index at baseline for treatment and control schools. There is a clear overlap along the

whole distribution. We cannot reject equality of both distributions using a Kolmogorov-Smirnov test.

Second, on average, the recipients who are offered scholarships on the basis of the merit-based

targeting have more assets, a lower poverty index score and better performance on the baseline test than

students in the poverty-based treatment. For instance, the poverty index, which ranges from 0 (wealthiest

family in the sample) to 292 (poorest family in the sample), has a mean of 245.13 for poverty based

students, and 218.2 for merit-based students. Likewise, the baseline test (ranging from 0 to 25) has a

mean of 19.77 for merit-based students and 17.74 for poverty-based students. We return to this issue in

the discussion below where we discuss tradeoffs between the two targeting approaches.

The main finding from Table 1, however, is that it shows a balanced sample between treatment

and control groups at baseline, which is a key determinant of the random assignment approach being a

valid identification strategy.

4. Results

Impacts on enrollment and attendance

The intervention is aimed directly at incentivizing higher enrollment and attendance. In order to

keep the scholarship, the selected students must stay enrolled, attend school regularly, and maintain

11

passing grades until they graduate from primary school (sixth grade). We focus on three enrollment and

attendance proxies: the proportion of students reaching 6th grade, the highest grade completed, and the

hours of school attended in the past seven days.

In order to provide a baseline for assessing the relative magnitude of impacts, Table 2 reports

outcome variables of the students at the follow up in the control group. Between 61% and 64% of the

students reported reaching at least sixth grade and the average grade completion is around 5.4.. The third

outcome variable was constructed from a question that asked students how many hours they attended

school the past seven days, conditional on being enrolled. Depending on the control group, students

reported an average having attended school for about 8.83 hours (poverty) and 9.27 hours (merit) in the

past week.

Table 3 reports the program impacts on the enrollment and attendance proxies. Columns (1) and

(2) present the effects of treatment of the poverty and merit-based interventions, respectively, when

controlling for baseline student characteristics (i.e. the variables through the application forms, reported in

Table 1), poverty index and test scores at baseline, and province fixed effects. As discussed above, the

randomized assignment was successful in that it produced a balanced sample. Controlling for additional

variables in the program impact regressions should therefore only affect the precision of the estimates, not

the magnitude of the estimated effects. Appendix 1 presents the results without controls; as expected, the

results are very similar to the results in Table 3.

Overall, Table 3 shows consistent evidence of positive impacts from the interventions on

enrollment and attendance. The proportion of students reaching grade 6 increased with both treatments,

and the effects are similar in magnitude. The estimated impacts range between 12% point and 17% point

increase from a counterfactual of around 61%-64%. Similarly, the intervention increased the average

highest grade completed in both treatment samples, with effects ranging from 0.332 –in poverty-based

intervention—to 0.187 grades—merit-based—, from a counterfactual of about 5.4 years. These impacts

are similar than those found in the context of the Secondary School scholarships program (where impacts

on enrollment were on the order of 20-25% point increase). These impacts are larger than most

documented in countries elsewhere in the world (Fiszbein and Schady 2011), and should be assessed

against the very small size of the transfer considered (i.e. $US20 per year).

The measure of attendance (number of hours in school in the past seven days) shows positive

estimates, but none of these are statistically significant, with the exception of the estimate without

controls for poverty treatment. Nevertheless, taking the point estimates at face value, the results suggest

12

that the intervention increases attendance by on the order of 2.9 hours per week for poverty-based

treatment and 0.64 for merit-based.

In sum, there is strong evidence that the program increased enrollment and suggestive evidence

that it also improved attendance rates, regardless if the scholarship is based on merit or on poverty

status—the targeting approach did therefore not affect the extent to which the program increased

measured school participation.

Results on test scores

There are two main channels through which the program could impact test scores. First, by

incentivizing enrollment and attendance, students are more exposed to school—and through that

additional schooling acquire more learning. Second, by requiring that the students maintain passing

grades, the program may give students an incentive to study more.

Table 4 presents impacts of the interventions on the three measures of academic and cognitive

achievement: Mathematics test, Digitspan test, and Raven test. (Table 4 has the same structure as Table 3;

it present impact coefficients after controlling for baseline characteristics and district fixed effects;

Appendix A presents results without controls). Given that we use three, likely correlated, measures of test

scores, we estimate the model using Seemingly Unrelated Regressions (SUR) to gain efficiency. All three

measures are standardized using the mean and deviation of the respective control group—impacts can

therefore be interpreted as changes in a standard deviation of the achievement measure.

In contrast to the results on enrollment and attendance, Table 4 reveals different impacts on test

scores between the poverty and the merit treatment group. While the impact estimates show no effects

from treatment for students treated based on their poverty status, there is a clear trend of positive effects

from the intervention for students treated based on merit. All the point estimates for the merit treatment

are positive and statistically significant. For the merit sample when using additional controls, the effect of

the intervention on the math test is 0.170 standard deviations, the effect on the Digitspan test is 0.149

standard deviations, and the effect for the Raven test is 0.178 standard deviations. The results on math

are of particular interest, since math is potentially the measure that is the most sensitive to exposure to

additional schooling. Moreover, households have potentially less ability to substitute for teaching math

than vocabulary or other types of learning. These effects on test scores are similar in magnitude to the

merit-based scholarship program evaluated in Kenya (Kremer, Miguel and Thornton 2009).

13

In sum, the results suggest that the program incentivized both types of students – those from

poorer households and those with higher academic merit—to enroll in additional years of schooling and

have higher attendance. However, only students who received treatment based on merit show any gains in

test scores from the intervention.

Heterogeneity

All in all, it seems that the program incentivized both types of students –those with higher

academic merit and those from poorer households —to enroll and attend additional years of school.

However, only students who received treatment based on merit show any gains in academic achievement

from the intervention. That is, additional schooling (e.g. enrollment and attendance) results in better

learning outcomes (as captured by our four tests) only for those students who had better skills at the

baseline. At the light of this result, it is important to explore the heterogeneity of effects by baseline skill

and poverty levels. These effects are presented in Table 5. For each poverty school (treated and control)

we identify students that were above and below each school’s median baseline test, and then we run

separate regressions to estimate the effect for the two types of students (high baseline achievers and low

baseline achievers). For each follow-up test (math, Digitspan and Raven), Column (1) presents the results

on test scores of the effects of the poverty-targeting mechanism for low achievers at baseline and Column

(2) for high baseline achievers. In an analogous way, for merit-based schools (treatment and controls) we

identify students above and below each school’s median poverty index, and then, we run separate

regressions for the two populations. For each follow-up test, Column (3) presents the impact of merit-

based treatment for those above the school’s median poverty index (non-poor population) and Column (4)

presents the impact of merit-based treatment for students below the school median poverty index (poor

population).19

As Table 5 shows, the merit-based treatment has either similar or larger positive impacts on the

poor population than in the non-poor population. In contrast, the poverty-based treatment does not elicit

positive results from either the high baseline achievers or the low baseline achievers. In other words, the

asymmetry in test results for poverty-based and merit-based targeting mechanisms persists. More

importantly, the poverty-based treatment does not induce better test results among high baseline

performers, whereas the merit-based treatment does induce better test scores among baseline poor

individuals, despite the fact that the conditions of both scholarships—merit and poverty—are the same.

19

An alternative specification is to run the pooled regression with a dummy variable for treatment, a dummy

variable indicating the status at base line, and the interaction term. The results are very similar to the ones presented

in Table 5.

14

Given the results for poor individuals in the merit-based treatment, similar results were expected for the

high achievers in the poverty-based treatment.

Different mechanisms can explain the asymmetry in tests’ results. As argued before, both types of

scholarships provide incentives for students to increase effort. For instance, it is plausible that, due to the

scholarship, students increase hours of studying outside school. Also, families may be motivated to

invest more in education expenditures, textbooks and such and as a result, help the student in conserving

the scholarship. Likewise, the program can impact directly the behavior of the school and teachers. For

instance, under an altruistic model, teachers can increase attention to students with scholarships with the

hope that they can retain the money. Also, it is possible that presents from scholarship winners’ parents to

teacher can induce higher effort. As such, the school can change behavior. Banerjee and Duflo (2006),

while presenting the results of Kremer et al (2009), discuss changes in teacher motivation and higher

control of families. As mentioned before, the follow-up data comes from household interview. As such,

the drawback of household interviews is the lack of school level information. However, household

interviews allow us to gather evidence regarding potential channels that may explain the results, such as,

student effort outside of school and a household’s investment in education.

Students’ effort is captured by the amount of time spent studying, doing homework and taking

private lessons outside of school. The household’s effort is measured by the total amount of education

expenditure by the household and the proportion of this expenditure spent on textbooks. The effects of

treatment on each of these three variables are presented in Table 6. On average, control students spent 3.5

hours per week doing school tasks outside the school. Merit-treated students spent more time doing

academic work outside of school (an increment of 0.579 hours). The household response to scholarships

in terms of education spending and the nature of that spending differs across the scholarship types. On

average, household’s education expenditures are approximately U$17. Households with a merit

scholarship recipient spent U$ 5 more on education, and a higher proportion (1% more) of the

expenditure was on textbooks, than control students’ households.20

In contrast, there were no impacts on

these outcomes in households with poverty targeted recipients. The results suggest that only household

with merit-based students increase effort.

All in all, it seems that families in the merit-based treatment invest more in the education of

students; in contrast to what happen in families with the poverty-based treatment. Also, students in the

merit-based treatment put more effort outside school than the poverty-based students. These findings are

20

For the text book expenditures, the p value of the coefficient is 0.105

15

compatible with a motivation hypothesis: merit-based students are motivated to work more, and their

families are motivated to invest more. Ideally we would like to disentangle the motivation of students and

from the motivation of teachers, however, the data available do not allow us to carry such analyses.

Peer effects on enrollment, attendance and achievement

One concern with a program such as this is that there might be negative spillover effects. If

increased enrollment and attendance leads to classroom overcrowding, then this may hurt the learning

opportunities afforded to other students. At the same time, there might also be positive spillover effects if

scholarships create a positive energy in favor of schooling and learning which could affect all children in

a classroom.21

Moreover, peer effects might differ by the design of the targeting approach: for instance

applicants denied a merit-based scholarship might become discouraged, and eventually perform worse.

Since there will always be limited scope for containing spillover effects, they should be estimated

directly, and factored into the “equity/efficiency” tradeoff in program design.

As discussed above, the design of the intervention allows us to test directly for peer effects. Table

9 presents the coefficient estimates for the indicator variable that is one for those students not treated in

treated schools, and zero for students in control schools with analogous baseline test scores or poverty

index scores. The dependent variables are proportion of students reaching 6th grade, the highest grade

completed, and the hours of school attended. The estimates can be read similarly to those reported in

Table 3, but estimated off of the sample of non-recipients and their counterfactual. There seems to be a

positive peer effect in poverty-treated schools: the three coefficients are positive; and one of them is

significant at the 5% level. Moreover, as expected, the point estimates are lower than the direct effect of

the program (Table 3). Regarding the merit-based treatment, we do not find any evidence of peer effects,

negative or positive. 22

Table 10 is the analogous table for the three measures of test scores. None of the

coefficient estimates are statistically significant. All the coefficients are very close to zero.

In sum, it seems that the poverty-based treatment may induce more attendance from non-treated

peers. No other peer effects are detected, neither positive nor negative, from the program.

5. Conclusions

21

An additional positive spillover might be on younger cohorts who stay in school longer in order to potentially

benefit from scholarships. We don’t have the data available to address this issue. 22

The results therefore dispel the notion that there was discouragement among merit scholarship applicants who did

not receive a scholarship.

16

The fact that some students were able to take better academic advantage from more exposure to

school than others highlights an issue rarely addressed in previous evaluations of conditional cash transfer

programs. Recent evidence on monetary incentives for schooling shows that students are able to change

their behavior on the margins that are under their control—for example enrollment and attendance.

However, these positive effects do not necessarily translate into test score gains. For example, despite the

fact that Mexico’s Oportunidades program—a rigorously evaluated conditional cash transfer program—

induced students to enroll and attend more to school, the program did not induce higher test scores. A

recent set of papers has argued that education systems in developing countries are typically tailored

towards better-off and better-skilled students. Specifically, Glewwe, Kremer and Moulin (2009) show that

only the strongest students at baseline were able to take advantage of textbooks that were provided to

schools in Kenya; Duflo, Dupas and Kremer (2011), while studying the effects of tracking students into

classrooms according to initial achievement (also in Kenya), show that teachers who were assigned to

students at the bottom of the achievement distribution were less likely to teach.

Our findings based on the Cambodian Primary Scholarships Pilot add to this discussion. On the

one hand, additional exposure seems to pay off in terms of test scores to those students who are more

academically ready to take advantage of the opportunity. On the other hand, poorer students who are not

academically prepared are not able to measurably gain in terms of test scores from the additional

schooling. This evidence is compatible with the idea that teachers are not prepared, or do not have the

pedagogical skills, to take on the challenge of reaching the more academically challenged students.

Clearly more work is needed to establish how best to prepare and incentivize teachers to reach these

students—and this would be an important area for Cambodia (and other countries) to generate knowledge.

At the same time, however, this evaluation suggests that for students who are better academically

prepared—including poor students—incentivizing school attendance can pay off in measurable learning

outcomes. This suggests that remedial lessons for students in the early grades, or increasing school

readiness among poorer students, for example through early child development programs, might be

complementary approaches to increasing the impact of schooling, and programs that incentivize

schooling. Indeed, data from Cambodia suggest that children suffer from substantial delays in cognitive

development, which hampers school readiness (Naudeau and others 2011).

The Cambodian program uses two targeting approaches setting up a potential tradeoff between

efficiency—defined as achieving more learning per dollar transferred—versus equity—defined as

reaching the poorest population. Analysis of the socio-economic profile of program applicants and

recipients under the two targeting schemes—and comparing those to the national distribution of socio-

economic characteristics suggests that both targeting approaches are heavily weighted to the poor. The

17

first panel of Figure 5 shows that 50% of those who applied to the program are within the poorest

nationally-benchmarked quintile; fewer than 3% of applicants were from the richest quintile. Clearly the

program was targeted to poor areas and poor schools. Unsurprisingly, targeting the scholarships further

to the poorest from within each school yields an even greater pro-poor distribution of benefits: 85% of

applicants who were in the poorest half in their school (i.e. those targeted by poverty scholarships) were

from the poorest two quintiles of the population—63% were in the poorest quintile (Panel 2). Merit-

based targeting is not as pro-poor—but is still largely able to reach the poorest groups in the population:

76% of applicants who were in the top merit half of their school (i.e. those targeted by merit scholarships)

were from the poorest two quintiles of the population—54% were in the poorest quintile (Panel 7). A

complementary analysis of the within-school correspondence between high/low poverty applicants and

high/low test scoring applicants yields a similar conclusion: wealthier applicants were not necessarily

higher-scoring.23

Given the relatively effective geographic targeting it is unclear whether this result is

generalizable. In other settings (e.g. where there is more heterogeneity in student poverty levels) the

result may not hold. Nevertheless, the results suggest that for this program, the tradeoff between

efficiency and equity was not particularly stark. Scaling up an approach that targets students with high

academic potential—while ensuring that the poorest student are among that set—is likely to be the

approach that maximizes both the equity and effectiveness objectives of the program.

23

See Appendix B for further details.

18

References

Baird, Sarah, Craig McIntosh and Berk Ozler. 2009. “Designing Cost-Effective Cash Transfer Programs

to Boost Schooling Among Young Women in Sub-Saharan Africa.” World Bank Policy Research

Working Paper No. 5090. The World Bank.

Baird, Sarah, Craig McIntosh and Berk Özler, 2011. "Cash or Condition? Evidence from a Cash Transfer

Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 1709-

1753.

Banerjee, Abhijit, and Esther Duflo. 2006. "Addressing Absence." Journal of Economic Perspectives,

20(1): 117–132.

Behrman, Jere R., Susan W. Parker, and Petra E. Todd. 2005. “Long-TermImpacts of the Oportunidades

Conditional Cash Transfer Program onRural Youth in Mexico.” Discussion Paper 122, Ibero-America

Institute for Economic Research, Göttingen, Germany.

Behrman, Jere R., Piyali Sengupta, and Petra Todd. 2000. “The Impact of PROGRESA on Achievement

Test Scores in the First Year.” Unpublished manuscript, International Food Policy Research Institute,

Washington, DC.

Chaudhury, Nazmul and Dilip Parajuli. 2008. “Conditional Cash Transfers and Female Schooling: The

Impact of the Female School Stipend Programme on Public School Enrolments in Punjab, Pakistan.”

Applied Economics.

Duflo, Esther, Pascaline Dupas and Michael Kremer. 2011. “Peer Effects, Teacher Incentives, and the

Impact of Tracking: Evidence from a Randomized Evaluation in Kenya.” American Economic Review.

101(5): 1739-74.

Ferreira, Francisco H., Deon Filmer and Norbert Schady. 2009. “Own and Sibling Effects of Conditional

Cash Transfer Programs: Theory and Evidence from Cambodia” World Bank Policy Research Working

Paper No. 5001. The World Bank, Washington, DC.

Filmer, Deon and Lant Pritchett. 2001. “Estimating Wealth Effects without Expenditure Data – or Tears:

With an Application to Educational Enrollments in States of India.” Demography. 2001. 38(1):115-132.

Filmer, Deon and Kinnon Scott. 2011. “Assessing Asset Indices.” Demography. 2011. 49(1), 359-392.

Filmer, Deon, and Norbert Schady. 2008. “Getting Girls into School: Evidence from a Scholarship

Program in Cambodia.” Economic Development and Cultural Change 56(2): 581–617

Filmer, Deon and Norbert Schady. 2009. “School Enrollment, Selection and Test Scores.” World Bank

Policy Research Working Paper No. 4998. The World Bank, Washington, DC.

Filmer, Deon and Norbert Schady. 2011. “Does more cash in conditional cash transfer programs always

lead to larger impacts on school attendance?” Journal of Development Economics. 96(1): 150–157.

Fiszbein, Ariel and Norbert Schady. 2009. Conditional Cash Transfers: Reducing Present and Future

Poverty. The World Bank. Washington, DC

19

Glewwe, Paul, Michael Kremer and Sylvie Moulin. 2009. “Many Children Left Behind? Textbooks and

Test Scores in Kenya.” American Economic Journal: Applied Economics. 1(1): 112-135.

Kremer, Michael, Edward Miguel, and Rebecca Thornton. 2009. “Incentives to Learn.” Review of

Economics and Statistics. 91(3): 437-456.

Naudeau, Sophie, Sebastian Martinez, Patrick Premand, and Deon Filmer. 2011. “Cognitive Development

among Young Children in Low-Income Countries” in Alderman, Harold ed. No Small Matter: The Impact

of Poverty, Shocks, and Human Capital Investments in Early Childhood Development. The World Bank.

Washington, DC.

Orfield, Gary, “Foreword,” in Donald E. Heller and Patricia Marin (Eds.), Who Should We Help? The

Negative Social Consequences of Merit Aid Scholarships (2002) (Papers presented at the conference

“State Merit Aid Programs: College Access and Equity” at Harvard University). Document found in

http://civilrightsproject.ucla.edu/research/college-access/financing/who-should-we-help-the-negative-

social-consequences-of-merit-scholarships/

Royal Government of Cambodia. 2006. Student Achievement and Education Policy: Results from the

Grade Three Assessment—Final Report. Cambodia Education Sector Support Project—National

Assessment Component. Phnom Penh, Cambodia.

20

Figure 1: Proportion of 15 to 19 year olds who have completed each grade, by quintile.

Source: DHS 2010

0

0.2

0.4

0.6

0.8

1

1 2 3 4 5 6 7 8 9

Poorest quintile

Quintile 2

Quintile 3

Quintile 4

Richest quintile

21

Figure 2. Design of the intervention

209

Schools

Poverty-based

schools

Merit-based

schools

Lottery

Phase 1

292

0

Poverty index

29

Test score

0

Test score

Phase 2

Phase 1

Phase 2

Grade 4

Grade 3

Grade 3

Grade 4

Grade 3

Grade 3

Poverty index

Poverty index

292

0

Grade 4

Grade 4 Test score

Test score

29

0

Grade 5 Grade 6

Grade 4

Grade 4

Grade 4

Grade 5

Grade 4 Grade 5

Grade 5

Grade 5

Grade 6

Grade 6

Grade 5

Grade 5

Grade 6

Grade 5

TIME=0: BASELINE

INFORMATION; LOTTERY

TIME=1:

PHASE-IN

TIME=2:

FOLLOW-UP

= Treated

= No treated

Poverty index

22

Figure 3: Poverty score at baseline, Treatment versus Control

Source: Students at follow-up, using baseline information

0

.00

5.0

1

kd

en

sity p

ov_

scor

0 100 200 300Poverty score at baseline by school-level treatment status

Treatment (both merit and poverty) Control

23

Figure 4: Test scores at baseline, Treatment versus Control

Source: Students at follow-up, using baseline information

0

.02

.04

.06

.08

.1

kd

en

sity k

hm

_m

ath

0 5 10 15 20 25Test score at baseline by school-level treatment status

Treatment (both merit and poverty) Control

24

Figure 5: Distribution of selected populations across nationally benchmarked quintiles

Source: Analysis of Cambodia DHS 2010 and Primary Scholarship Application forms. Quintiles are defined on the basis of an index of household wealth-related variables that are collected in both the DHS 2010 as well as on the scholarship program application forms.

0

10

20

30

40

50

60

70

(1) Program applicants (2) High Poverty (3) High merit

Poorest quintile Quintile 2 Quintile 3 Quintile 4 Richest quintile

25

Table 1. Baseline Balance and mean and standard deviation of baseline characteristics

School Level

Student Level

Control Difference

Control Difference Control Difference

with treatment

Poverty with treatment Merit with treatment

(1) (2) (3) (4) (5) (6)

Gender 0.49 0.037

0.52 0.100*** 0.49 -0.03

'(0.50) '(0.02)

'(0.50) '(0.03) '(0.50) '(0.04)

No of minors 1.69 -0.026

1.79 0.075 1.73 -0.097

(1.11) (0.09)

'(1.12) '(0.12) '(1.12) '(0.12)

Own motorcycle 0.42 0.008

0.28 -0.035 0.42 0.003

(0.49) (0.04)

'(0.45) '(0.05) '(0.49) '(0.05)

Own car/truck 0.16 0.017

0.04 0.01 0.13 0.028

(0.37) (0.03)

'(0.19) '(0.03) '(0.34) '(0.04)

Own oxen/buffalo 0.55 0.032

0.39 0.109* 0.53 0.033

(0.50) (0.05)

'(0.49) '(0.06) '(0.50) '(0.06)

Own pig 0.56 0.028

0.43 0.117** 0.55 0.029

(0.50) (0.04)

'(0.50) '(0.06) '(0.50) '(0.05)

Own ox or buffalo cart 0.31 0.02

0.19 0.058 0.29 0.009

(0.46) (0.04)

'(0.40) '(0.05) '(0.45) '(0.05)

Hard roof 0.49 0.064

0.32 0.047 0.48 0.102**

(0.50) (0.04)

'(0.47) '(0.05) '(0.50) '(0.05)

Hard wall 0.54 0.032

0.38 0.045 0.55 0.018

(0.50) (0.04)

'(0.49) '(0.06) '(0.50) '(0.05)

Hard floor 0.85 0.039

0.79 0.037 0.84 0.068*

(0.36) (0.03)

'(0.41) '(0.05) '(0.37) '(0.04)

Have automatic toilet 0.07 -0.02

0.02 -0.01 0.05 0.005

(0.25) (0.02)

'(0.13) '(0.01) '(0.22) '(0.02)

Have pit toilet 0.12 0.018

0.11 0.02 0.13 0.001

(0.32) (0.03)

'(0.32) '(0.03) '(0.34) '(0.04)

Electricity 0.25 0.011

0.16 -0.01 0.23 -0.002

(0.43) (0.04)

'(0.37) '(0.04) '(0.42) '(0.05)

Pipe water 0.06 -0.001

0.03 -0.012 0.06 -0.013

(0.24) (0.02)

'(0.17) '(0.01) '(0.23) '(0.02)

Poverty Index (o to 292) 210.16 -1.609

245.13 -2.924 218.2 -11.771

(60.18) (5.43)

'(32.73) '(5.14) '(51.66) '(8.76)

Test score (0 to 25) 17.47 0.534

17.74 0.888 19.77 0.028

(4.81) (0.52)

'(4.71) '(0.68) '(3.22) '(0.48)

Number of students 940 2448

431 883 474 940

Number of schools 101 204 67 119 67 118

Columns (1), (3), (5): means and standard deviation, control group. Columns (2), (4), (6): difference with treatment, estimated by regressing each variable against corresponding treatment variable; standard error in parenthesis

26

Table 2. Outcome variables at follow-up

Student Level

Poverty-targeting

Merit-targeting

Control Treatment

Control Treatment

(1) (2) (3) (4)

Reach grade 6 0.61 0.8

0.64 0.77

'(0.49) '(0.40)

'(0.48) '(0.42)

Completed grades 5.38 5.73

5.45 5.68

'(1.22) '(0.93)

'(1.23) '(0.94)

Number of hours 8.83 12.29

9.27 10.64

'(12.97) '(14.87)

'(13.22) '(14.38)

Math test 0.02 -0.02

0.16 0.32

'(1.01) '(0.94)

'(1.04) '(1.04)

Digitspan 0.02 0

0.08 0.23

'(0.98) '(1.00)

'(0.99) '(0.97)

Raven Test -0.02 -0.07

0.11 0.21

'(0.98) '(0.92)

'(0.99) '(1.15)

Number of Students 431 452 474 466

Mean and () standard deviation

27

Table 3. Impact on Enrollment and Attendance

Reach Grade Six

Highest Grade Completed

Number of hours in school, last 7 days

(conditional on enrollment)

(1) (2) (1) (2) (1) (2)

Poverty-targeting treatment 0.170***

0.332***

2.865

'(0.04)

'(0.11)

'(1.87)

Merit-targeting treatment

0.120***

0.182*

0.635

'(0.04)

'(0.10)

'(1.55)

Constant 1.764 0.514

8.161** 5.285**

-100.065** 0.575

'(1.42) '(0.83)

'(3.35) '(2.33)

'(45.16) '(28.25)

Control Variables Yes Yes Yes Yes Yes Yes

No. Obs 883 940

831 897

665 713

F() 6.435 4.872

2.271 1.759

1.246 2.026

R2 Adj 0.18 0.155 0.145 0.122 0.131 0.199

Regression coefficient of dependent variable against treatment indicator controlling by Table 1 baseline characteristics and district fixed effects. All estimators using clusters at school level.

28

Table 4. Impact on test scores

Mathematics

Digispan Test

Raven Test

(1) (2) (1) (2) (1) (2)

Poverty-targeting treatment -0.041

-0.059

-0.021

'(0.06)

'(0.07)

'(0.06)

Merit-targeting treatment

0.170***

0.149**

0.178***

'(0.07)

'(0.06)

'(0.07)

Constant -3.204 1.831

-1.977 0.000***

0.000*** 2.055

'(3.26) '(2.19)

'(3.42) '(2.19)

'(0.00) '(2.19)

Control Yes Yes Yes Yes Yes Yes

No. Obs 883 940

883 940

883 940

Chi_ 2 177.525 178.627

112.442 122.093

130.253 169.017

R2 Adj 0.167 0.16 0.113 0.093 0.126 0.152

Regression coefficient of dependent variable against treatment indicator, controlling by Table 1 baseline characteristics and district fixed effects. All estimators using Seemingly Unrelated Regression (SUR) estimation

29

Table 5. Impact on test scores, by baseline test score and poverty index

Mathematics

Digispan Test

Raven Test

Baseline test score

Baseline poverty index

Baseline test score

Baseline poverty index

Baseline test score

Baseline poverty index

Low High

Non-poor Poor

Low High

Non-poor Poor

Low High

Non-poor Poor

(1) (2) (3) (4) (1) (2) (3) (4) (1) (2) (3) (4)

Poverty-targeting treatment -0.019 -0.069

-0.077 -0.077

0.163 -0.103

'(0.09) '(0.09)

'(0.11) '(0.09)

'(0.10) '(0.08)

Merit-targeting treatment

0.053 0.233**

0.194* 0.142*

0.107 0.222**

'(0.09) '(0.09)

'(0.10) '(0.08)

'(0.10) '(0.09)

Constant 0.000*** -7.26

0.000*** 0.000***

1.716 0.000***

0.000*** -2.981

7.520* 0.000***

0.000*** 0.000***

'(0.00) '(5.64) '(0.00) '(0.00) '(4.80) '(0.00) '(0.00) '(0.00) '(4.49) '(0.00) '(0.00) '(0.00)

Control Yes Yes

Yes Yes

Yes Yes

Yes Yes

Yes Yes

Yes Yes

No. Obs 332 551

427 513

332 551

427 513

332 551

427 513

Chi_ 2 108.149 119.13

128.474 182.44

78.243 95.644

84.833 89.501

81.429 99.055

96.842 149.375

R2 Adj 0.209 0.178

0.203 0.212 0.191 0.144 0.148 0.149 0.197 0.152 0.169 0.207

Regression coefficient of dependent variable against treatment indicator, controlling by Table 1 baseline characteristics and district fixed effects. All estimators using Seemingly Unrelated Regression (SUR) estimation

30

Table 6. Impact on test scores on Non-treated Peers

Mathematics

Digitspan Test

Raven Test

(1) (2) (1) (2) (1) (2)

Poverty-targeting treatment -0.116

-0.08

-0.173*

(0.09)

(0.09)

(0.09)

Merit-targeting treatment

0.033

0.091

0.076

(0.09)

(0.10)

(0.09)

Constant 0.000*** 0.000***

0.000*** -1.202

1.385 2.511

(0.00) (0.00)

(0.00) (0.00)

(2.44) (0.00)

Control Variables Yes Yes Yes Yes Yes Yes

No. Obs 591 503

591 503

591 503

F() 131.4 122.862

94.067 101.866

121.76 92.4

R2 Adj 0.181 0.127 0.137 0.168 0.171 0.155

Regression coefficient controlling by Table 1 baseline characteristics and district fixed effects. All estimators using SUR

31

Table 7. Impact on test scores, by baseline test score and poverty index

Mathematics

Digitspan

Raven Test

Baseline test score

Baseline poverty index

Baseline test score

Baseline poverty index

Baseline test score

Baseline poverty index

Low High

Rich Poor

Low High

Rich Poor

Low High

Rich Poor

(1) (2) (3) (4) (1) (2) (3) (4) (1) (2) (3) (4)

Poverty-targeting treatment -0.011 -0.128

-0.102 0.007

0.053 -0.076

(0.11) (0.12)

(0.11) (0.13)

(0.11) (0.12)

Merit-targeting treatment

-0.022 0.259**

0.201 0.046

0.063 0.185

(0.12) (0.11)

(0.13) (0.11)

(0.12) (0.12)

Constant 0.000*** 5.126

0.000*** 4.871

0.000*** 0.000***

0.000*** -5.445

0.000*** 0.000***

0.000*** 2.074

(0.00) (11.38)

(0.00) (5.28)

(0.00) (0.00)

(0.00) (5.28)

(0.00) (0.00)

(0.00) (5.28)

Control Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

No. Obs 319 352

312 394

319 352

312 394

319 352

312 394

Chi_ 2 83.269 103.635

127.389 135.978

60.946 81.716

92.082 79.902

84.686 80.668

129.416 122.469

R2 Adj 0.204 0.227 0.265 0.257 0.158 0.187 0.21 0.169 0.204 0.186 0.279 0.237

Regression coefficient controlling by Table 1 baseline characteristics and district fixed effects. All estimators using SUR

32

Table 8. Potential mechanisms for impact

School Responses

Student and Household (hh) effort

School Open Yesterday

Teacher Absence/Sick

Yesterday

Time doing homework,

studying or private lessons

Total expenditure in education, hh

Percentage of expenditure in textbooks, hh

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10)

Poverty-targeting treatment 0.039

0.044

0.314

-4013.14

-0.001

(0.07)

(0.03)

(0.45)

(14468.91)

(0.01)

Merit-targeting treatment

0.072

0.023

0.321

20936.620*

0.014*

(0.06)

(0.03)

(0.40)

(11708.08)

(0.01)

Constant -3.323 -0.019

-0.679 1.232

-19.035 3.64

1.35E+05 1.23E+06

-0.237* -0.095

(2.07) (1.22)

(0.55) (0.88)

(17.32) (11.76)

(212286.02) (790015.70)

(0.13) (0.13)

Control Variables Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

No. Obs 518 540

671 706

498 534

671 706

550 595

F() 3.529 3.072

1.883 1.236

8.016 3.39

4.487 3.207

3.368 1.878

R2 Adj 0.166 0.249 0.101 0.159 0.176 0.195 0.131 0.134 0.097 0.095

Regression coefficient controlling by Table 1 baseline characteristics and district fixed effects. All estimators using clusters at school level.

33

Table 9. Impact on Enrollment and Attendance on Non-treated peers

Reach Grade Six

Highest Grade Completed

Number of hours in school, last 7 days

(conditional on enrollment)

(1) (2) (1) (2) (1) (2)

Poverty-targeting treatment 0.082**

0.058

0.529

'(0.04)

'(0.11)

'(1.82)

Merit-targeting treatment

-0.009

-0.099

0.291

'(0.05)

'(0.12)

'(1.67)

Constant 0.168 2.201**

4.030** 8.599***

12.924 14.213

'(0.83) '(0.94)

'(1.92) '(1.81)

'(29.16) '(28.46)

Control Variables Yes Yes Yes Yes Yes Yes

No. Obs 785 678

732 633

576 486

F() 7.528 6.94

1.603 1.765

2.166 3.195

R2 Adj 0.172 0.183 0.125 0.118 0.191 0.21

Regression coefficient of dependent variable against treatment indicator controlling by Table 1 baseline characteristics and district fixed effects. All estimators using clusters at school level.

34

Table 10. Impact on test scores on Non-treated Peers

Mathematics

Digispan Test

Raven Test

(1) (2) (1) (2) (1) (2)

Poverty-targeting treatment -0.105

-0.009

-0.09

'(0.07)

'(0.07)

'(0.07)

Merit-targeting treatment

-0.061

-0.048

-0.028

'(0.07)

'(0.08)

'(0.08)

Constant 0.000*** 0.000***

-0.922 -0.302

0.359 2.405

'(0.00) '(0.00)

'(1.93) '(0.00)

'(1.91) '(0.00)

Control Variables Yes Yes Yes Yes Yes Yes

No. Obs 785 678

785 678

785 678

F() 143.244 148.59

101.985 101.983

132.628 97.794

R2 Adj 0.153 0.129 0.115 0.131 0.145 0.126

Regression coefficient of dependent variable against treatment indicator, controlling by Table 1 baseline characteristics and district fixed effects. All estimators using Seemingly Unrelated Regression (SUR) estimation

35

Appendix A. Enrollment and test result, without controls

Impact on Enrollment and Attendance

Reach Grade Six

Highest Grade Completed

Number of hours in school, last 7 days

(conditional on enrollment)

(1) (4) (1) (4) (1) (4)

Poverty-targeting treatment 0.186***

0.349***

3.466*

'(0.04)

'(0.11)

'(1.80)

Merit-targeting treatment

0.131***

0.234**

1.374

'(0.05)

'(0.11)

'(2.01)

Constant 0.613*** 0.635***

5.377*** 5.448***

8.829*** 9.270***

'(0.03) '(0.03)

'(0.09) '(0.08)

'(1.15) '(1.14)

Control Variables No No No No No No

No. Obs 883 940

831 897

665 713

F() 18.154 7.753

9.334 4.572

3.691 0.465

R2 Adj 0.042 0.02 0.025 0.011 0.015 0.002

Columns (1) and (3), regression coefficient of dependent variable against treatment indicator without controls; Columns (2) and (4), controlling by Table 1 baseline characteristics and district fixed effects. All estimators using clusters at school level.

Impact on test scores

Mathematics

Digispan Test

Raven Test

(1) (2) (1) (2) (1) (2)

Poverty-targeting treatment -0.035

-0.019

-0.047

'(0.07)

'(0.07)

'(0.06)

Merit-targeting treatment

0.150**

0.147**

0.109

'(0.07)

'(0.06)

'(0.07)

Constant 0.018 0.165***

0.015 0.081*

-0.023 0.106**

'(0.05) '(0.05)

'(0.05) '(0.05)

'(0.05) '(0.05)

Control No No No No No No

No. Obs 883 940

883 940

883 940

Chi_ 2 0.291 4.925

0.085 5.273

0.54 2.426

R2 Adj 0 0.005 0 0.006 0.001 0.003

Regression coefficient of dependent variable against treatment indicator without controls; All estimators using Seemingly Unrelated Regression (SUR) characteristics

36

Appendix B: Equity and efficiency trade-off

Benefit incidence

The primary scholarships pilot was in part motivated by the fact that secondary school

scholarships failed to reach the poorest of the poor as they had dropped out of school before becoming

eligible for those programs. In order to assess the benefit incidence of the primary scholarships pilot, the

data from the applicants and recipients need to be compared to a national survey. The Demographic and

Health Survey (DHS) data collected in 2010 provide a useful comparison point. These data are national

in scope, and include the same set of variables than were collected from program applicants, and the

timing of the survey overlaps with the Primary Scholarships Pilot.

The following procedure was followed to place the DHS and application form data on a common

metric. First, the DHS variables (e.g. ownership of a motorbike, a car, a pig; material of dwelling floor,

walls, roof) were aggregated using principal components (following the approach described in Filmer and

Pritchett 2001, and Filmer and Scott 2011).24

The first principal component from this procedure is

interpreted as a “wealth index” from which quintiles can be derived.

The first three panels of Figure A1 show the percentage distribution across quintiles of the

Cambodian population of 8-15 year olds, of those in rural areas, and of those in the program Provinces.25

Clearly the population in areas served by the program is poorer than the other areas of the country: while

the distribution of 8-15 year olds mirrors the population as a whole (i.e. 20% in each quintile), and the

rural population is slightly more skewed towards the poorer quintiles (with only 12% of the rural

population being in the richest quintile), the population of the program Provinces is heavily concentrated

among the poorest quintiles. Indeed, 30% of 8-15 year olds within program provinces are in the poorest

quintile, and only 9% are in the richest quintile (Panel 3 of Figure A1).

Considering children who have completed grade 3, however, reduces the share in the poorest

quintile—the poorest of the poor do not even make it to grade 3. While 30% of children in the program

24

The procedure was run on the sample of 8 to 15 year olds to reflect the age of those who applied to the program.

The full set of variables used is: number of household members 0-14; ownership of motorbike, car, oxen/buffalo,

pig; dwelling roof made of hard materials; dwelling walls made of hard materials (e.g. concrete); dwelling walls

made of wood; dwelling floor made of hard materials (e.g. concrete); dwelling floor made of wood; flush toilet, pit

latrine; availability of electricity; drinking water from pipe (either in house or yard). 25

The DHS data do not distinguish between Preah Vihear and Stung Treng, and these numbers therefore include

both Provinces, in addition to Ratanakiri and Mondolkiri.

37

provinces are in the poorest quintile, only 20% of those who have attained grade three are in that quintile

(Panels 3 and 4 of Figure A1).

The variables captured in the scholarship application forms can be aggregated using the same

weights derived from the national sample in the principal components procedure described above.

Applicants can therefore be assigned to nationally-benchmarked quintiles, and the distribution of

applicants and recipients can be compared to the national distribution. The program was able to reach the

poorest schools within the program provinces: 50% of all those who applied to the program are within the

poorest nationally-benchmarked quintile; fewer than 3% of applicants were from the richest quintile

(Panel 5 of Figure A1). Unsurprisingly, targeting the scholarships further to the poorest from within each

school yields an even greater pro-poor distribution of benefits: 85% of applicants who were in the poorest

half in their school (i.e. those targeted by poverty scholarships) were from the poorest two quintiles of the

population—63% were in the poorest quintile (Panel 6). Merit-based targeting is not as pro-poor—but is

still largely able to reach the poorest groups in the population: 76% of applicants who were in the top

merit half of their school (i.e. those targeted by merit scholarships) were from the poorest two quintiles of

the population—54% were in the poorest quintile (Panel 7).26

There are two main conclusions that can be drawn from this analysis. First, compared to

targeting schemes in other countries, the benefit incidence of the scholarships pilot is very pro-poor.27

Of

course, the program was implemented in some of the poorest and remote Provinces and Districts;

therefore the universe from which the merit-based recipients were selected was relatively poor. Scaling

up the program would not necessarily achieve a similarly pro-poor benefit incidence as expansion would

mean operating in less poor areas, and therefore the baseline poverty of the population served would be

less severe. Second, the fact that the benefit incidence of the merit-based approach to targeting is largely

pro-poor (and not particularly less pro-poor than the poverty based approach) suggests that the tradeoff

between equity (i.e. pro-poorness of the program) and efficiency (i.e. the impact on learning outcomes)

might not that stark.

Poverty- versus merit-based targeting at the school level

26

A recent review of programs globally reported the share of Conditional Cash Transfer program benefits that were

received by the poorest 20% of the population (World Bank Social Protection Atlas, http://data.worldbank.org/data-

catalog/atlas_social_protection). Globally the average was that 47% of benefits reached the poorest 20%; with a

range from 24% in Bangladesh, to 58% in Panama. These findings suggest that the Cambodian program performs

relatively very well. 27

For a review of the incidence across a variety of programs see World Bank Social Protection Atlas (forthcoming).

38

The finding that merit-based targeting did not result in an overall regressive scheme is a

reassuring result. Figure A2 shows that within schools, the association between poverty and test scores is

not as close as might have been feared. The horizontal axis of Figure A2 is the relative poverty ranking

of an applicant, where 0 is the 50th percentile (which was the cutoff for scholarship eligibility in poverty-

targeting schools), +1 is the applicant ranked one position higher on the poverty scale, and -1 is the

applicant ranked one position lower. The vertical axis is the relative ranking of an applicant on the merit

scale, again with 0 being the 50th percentile (which was the cutoff for scholarship eligibility in the merit-

targeting schools).

If only wealthier children were to score high on the merit test, and poorer children low, then all

the observations would be in quadrants (A) and (D) of Figure A2 (respectively: Low Poverty/High Merit

and High Poverty/Low Merit). Clearly this is not the case: the observations are roughly equally

distributed across the four quadrants.28

This means that a merit-based targeting approach (which targets

children in quadrants A and B) includes children from both wealthier backgrounds (quadrant A) as well as

children from poorer backgrounds (quadrant B). Analogously, a poverty-based targeting approach

includes both higher scoring (quadrant B) and lower scoring (quadrant D) applicants.

These school-specific rankings are consistent with the benefit incidence analysis. The

equity/efficiency tradeoff between poverty- and merit-based targeting is not particularly stark.

Nevertheless, if the purely merit-based approach is adopted, it must be borne in mind that roughly half of

the recipients will come from better off families.

28

In fact the regression line for this figure has a mildly positive slope: the regression of relative merit ranking versus

relative poverty ranking yields a coefficient of 1.2 with a standard error of 0.17 (significant at the 1% level).

39

Figure A1: Distribution of selected populations across nationally benchmarked quintiles

Source: Analysis of Cambodia DHS 2010 and Primary Scholarship Application forms. Quintiles are defined on the basis of an index of household wealth-related variables that are collected in both the DHS 2010 as well as on the scholarship program application forms.

Figure A2: The association between applicants’ relative poverty and relative merit rankings

0

10

20

30

40

50

60

70

(1) All

(2) Cambodia

rural

(3) Program Provinces

(4) Program

provinces, attained grade 3

(5) Program

applicants

(6) High

Poverty applicants

(7) High merit applicants

Poorest quintile Quintile 2 Quintile 3 Quintile 4 Richest quintile

-15

-10

-50

51

01

5

Rela

tive

me

rit ra

nkin

g

-15 -10 -5 0 5 10 15Relative poverty ranking

B) 27%A) 21%

D) 25%C) 27%