Quality Preschool for Ghana: Advancing Research … Conference 2016/Session… · Quality Preschool for Ghana: Advancing Research Methods to Support Policy Change . ... • Sustainable

Global TIES for Children: Transforming Intervention Effectiveness and Scale

http: www.nyu.edu/global-ties

Quality Preschool for Ghana: Advancing Research Methods to Support Policy Change

J. Lawrence Aber, Sharon Wolf, & Jere Behrman

1

Acknowledgements

2

Partners

Innovations for Poverty Action Funders UBS Optimus Foundation World Bank Strategic Impact Evaluation Fund

Outline

Part I: Background • Early childhood education • Urbanization • Quality Preschool for Ghana Part II: Innovations in measurement and methods to assess: • Children’s school readiness • Classroom quality Part III: Public and Private School Differences in ECE in Ghana Part IV: Implications for Policy and Research

3

+ Part I: Background Early childhood education Urbanization Quality Preschool for Ghana

4

Early childhood education globally

• Sustainable Development Goal 4, Target 4.2: “ensure that all girls and boys have access to quality early childhood development, care and pre-primary education so that they are ready for primary education”.

• Improving the capacity of teachers to provide high quality learning environments for children is key.

• One approach is to work with governments to improve the training and qualifications of public sector teachers.

• Engaging with the private sector is also critical, particularly given its major role in education in many countries.

5

African urbanization

• “Urbanization is the single-most important transformation taking place on the African continent.” (World Bank, 2013)

• At 455 million in 2014, the urban population in Africa is expected to triple by 2050 (UN, 2015).

• As urban populations expand, and governments struggle to provide services to these fast growing communities, the phenomena of urban slum proliferation is a great and pressing challenge, and is an increasingly salient context for children in developing countries.

6

The Ghanaian context • Lower-middle income country • Pre-primary enrollment rates are some of the highest in the region (World

Bank, 2015); in some peri-urban communities in Accra, it is 94% (Bidwell et al., 2014).

• The private sector has played an important role in expanding access to preprimary education, particularly in urban areas. One study found that parents know of almost 4 preschools on average in walking distance from their home (Bidwell et al., 2014).

• While access is high, quality is lacking. A key priority for the government is training the untrained KG teacher workforce, and aligning parents’ perceptions of quality ECE with the government curriculum.

7

8

Accra, Ghana – 89%

Source: McCoy, 2015. MICS data.

The Policy Context • In 2007, Ghana’s government became the first in SSA country to

expand to 2 years of pre-primary education–called Kindergarten (KG)–in free and compulsory basic education provided by the state.

• The 2012 Government Kindergarten (KG) Situational Report concluded that the 2004 curriculum established is sound, but that teacher behavior has not yet adapted to reflect new pedagogy.

• Top priority is to train untrained 27,000 KG teachers (out of ~45,000 total) in KG specific pedagogy.

• A second priority is engaging parents in schools and raising their awareness of high quality early childhood education.

9

Quality Preschool for Ghana (QP4G)

• In partnership with Innovations for Poverty Action, The World Bank, and the National Nursery Teacher Training Center (NNTTC)

• Develop and test a nationally scalable model for: 1. In-service teacher training with ongoing monitoring and coaching by

district coordinators 2. Parental education intervention delivered through school PTAs to align

demands with the accepted age-appropriate standards for quality in ECE introduced on the supply-side, and to encourage parents to be involved in child learning at school and at home.

10

Quality Preschool for Ghana: Research Design

240 KG schools (120 public and 120 private)

80 (40 public 40 private)

Control group

Randomization


T1

Teacher training and coaching program


T2

Teacher training and coaching program

Parental education about KG learning Research questions:

• What is the impact of teacher training on teaching practices? Child outcomes? • What is the added impact of parental education about KG learning? • Do impacts vary in public and private schools?

6 disadvantaged districts in the Greater Accra Region

Stratification

11

QP4G Theory of Change

Teacher training

+ Monitoring/

support

Intervention Child outcomes

12

School readiness

Teacher training

+ Monitoring/

support

Intervention Classroom-level mediators Child outcomes

Classroom Quality

Teacher professional well-being

School readiness

QP4G Theory of Change 13

Teacher training

+ Monitoring/

support


Classroom Quality


QP4G Theory of Change

Parental intervention

14

School readiness

Teacher training

+ Monitoring/

support


E.g., Self-efficacy, Motivation

Classroom Quality

TIPPS


QP4G Theory of Change – Innovative Measures

Parental intervention

15

School readiness

IDELA

+ Part II: Innovations in measurement and methods to assess classroom quality and children’s school readiness

16

International Development and Early Learning

Assessment

17

Presenter

Presentation Notes

Review concluded that despite the existence of ECCD tools in the global space, none of the instruments available at the time offered a balance between 1) international applicability, especially within low and middle income country contexts, 2) feasibility and ease of administration and adaptation and 3) psychometric rigor. With these criteria in mind, and lessons learned from years of early childhood programming, Save the Children began the process of developing and validating the International Development and Early Learning Assessment (IDELA). Goal for IDELA was to develop a holistic, rigorous, open source instrument that is feasible and easily adapted to different national and cultural contexts.

Overview of the IDELA

Play-based assessment tool designed for children in the 3-6 age group

Takes about 40 minutes per child Includes 28 core items that cover 4

developmental domains + learning approaches and aspects of executive function

Plus the enumerator’s overall assessment of the child’s approaches to learning

Approach to measurement

• Assess if the empirical structure of the data fits the conceptual structure to identify if the measure can be used as intended.

• Conceptually measures 5 distinct domains of development 1. Motor development 2. Early literacy 3. Early numeracy 4. Social-emotional development 5. Executive function

• Question: Does the tool in fact assess 5 distinct domains as intended? And, do all items fit better as a single measure of holistic child development?

19

Analytic Process (I)

1. 1-factor exploratory factor analysis for the social-emotional development constructs using every item assumed to load onto the construct

2. Bi-factor analysis to assess the presence of a general factor and any residual factors • Do all items load onto a general factor representing the social-emotional

development? • Are there residual factors, and do they resemble the distinct domains?

All items are administered through direct assessment All models adjust standard errors for clustering at the school level.

20

• Split sample to conduct (1) exploratory analysis (inductive approach; N=1717), and (2) confirmatory analysis (deductive approach; N=1718).

• Determined by an SEM power analysis, a sample size of N = 365 is needed to confirm the full bi-factor model with each domain; thus power was adequate.

• Only results from the final confirmatory models are shown.

Analytic Process (II) 21

Presenter

Presentation Notes

Exploratory Analysis Descriptive Statistics - Inductive Approach Look for flexible ways to examine data without preconceptions Attempt to evaluate validity of assumptions Confirmatory Analysis Hypotheses determined at outset Hypothesis tests and formal confidence interval estimation

Bi-Factor Analysis: General vs. Residual Factors

• A scale is unidimensional when a single latent trait accounts for all of the common variance among item responses.

• In a bi-factor analysis, all items are free to load on a single general factor.

o Each item may also load on one secondary or “residual factor.” Residual factors are specified to be uncorrelated with each other and with the general factor.

o If not accounted for, locally dependent items (i.e., items that form a residual factor) can distort the latent construct which create problems for construct validity.

22

Based on a set of exploratory analyses, the following models were proposed and then confirmed Motor development Social-emotional development Early Literacy Early Numeracy Executive Function

23

z

Motor Development

MOTOR

copy1

copy2

fold

human2

human3

human4

human5

.70

.55

.45

.86

.91

.66

.34

.26

.78

.85

24

χ2 = 28.96, df = 11 RMSEA = .031 CFI = .995 TLI = .990

.45 SOCIAL

EMOTIONAL

.53

.67

.78

.19

.58

.55

.36

.61

.70

.60

.50

.64

.54

.47

personal1

personal2

personal3

personal4

personal5

personal6

emp1

emp2

emp3

conflict1

conflict2

emot1

emot2

emot3

emot4

friends

.30

.44

.64 Emotion

identification(f1)

.47

.72

.50

.24

.50

25 Social-Emotional Development

χ2 = 237.9, df = 96 RMSEA = .029 CFI = .974 TLI = .968

26 Early Literacy

χ2 = 1876.6, df = 515 RMSEA = .039 CFI = .995 TLI = .993

EARLYLITERACY

pa1pa2pa3ltr1ltr2ltr3ltr4ltr5ltr6ltr7ltr8ltr9ltr10ltr11ltr12ltr13ltr14ltr15ltr16ltr17ltr18ltr19ltr20

wrdpr1wrdpr2wrdpr3expvoc1expvoc2

oral1oral2oral3oral4oral5

writlev

Oral comprehension

(f1)

.46

.42

.36

.91

.92

.88

.92

.89

.92

.92

.91

.93

.96

.92

.98

.96

.94

.97

.98

.96

.99

.95

.97

.41

.26

.39

.28

.51

.40

.35

.36

.20

.19

.67

.68

.50

.80

.75

.78

.54

.29

.19

.38

.18

.24

.27

27 Early Numeracy

χ2 = 2188.5, df = 690 RMSEA = .036 CFI = .993 TLI = .993

.65

.63

EARLY NUMERACY

Size comparison

(f2)

oneto1

oneto2

oneto3

num1

num2

num3

num4

num5

num6

num7

num8

num9

num10

num11

num12

num13

num14

num15

num16

num17

num18

num19

num20

shape1

shape2

shape3

shape4

shape5

sort1

sort2

size1

size2

size3

size4

add1

add2

sub1

pattern

puzzle

.76

.80

.79

.88

.90

.92

.92

.88

.85

.82

.90

.95

.89

.96

.97

.98

.96

.97

.97

.98

.94

.94

.94

.54

.36

.34

.38

.40

.34

.19

.49

.33

.36

.25

.58

.42

.54

.08

.37

.58

.51

.81

.62

.47

Shape identification

(f1)

.19

.51

.81

.62

.85

.86

EXECUTIVE FUNCTION

.88

.99

.86

.93

.33

.22

.37

headtoes2

headtoes3

headtoes4

headtoes5

headtoes6

memory1

memory2

memory3

memory4

memory5

.40 .40

Working memory (f1)

.78

.71

.97

.77

.66 .84

28 Executive Function

χ2 = 216.4, df = 30 RMSEA = .060 CFI = .995 TLI = .992

Correlations among developmental domains

1 2 3 4

1. Motor 1.000

2. Social-Emotional .647 1.000

3. Early Literacy .668 .521 1.000

4. Early Numeracy .701 .545 .888 1.000

5. Executive Function .472 .544 .462 .521

29

Does the tool in fact assess 5 distinct domains as intended? And, do all items fit better as a single measure of holistic child development?

Model χ2 (df) RMSEA (90% CI) CFI TLI

5 factor model 111310 (5305) .026 (.025, .026) .983 .982

Hierarchical model a 11659 (5310) .026 (.025, .026) .982 .981

Unidimensional (one factor) b 19397 (5315) .039 (.039, .040) .959 .958

30

a χ2 difference is statistically significant (χ2 diff = 178.3 (5), p<.001), indicating that adding a higher-order factor is not a better fit for the data.

b χ2 difference is statistically significant (χ2 diff = 1951.9 (10), p<.001), indicating that a unidimensional factor is not a better fit for the data.

Beyond Access - TIPPS: A tool to connect classroom practices and processes to quality academic and socio-emotional learning developed by Seidman and colleagues (Seidman et al., 2013)

• Observation tool that aims to understand the nature and quality of the classroom environment.

• 19 items assessing the nature of interactions in the classroom.

• Understanding the importance of teacher professional training on the classroom setting as well as on student outcomes can produce a wealth of critical information on how to better define goals and to appropriately and more effectively allocate resources in the classroom.

31

Approach to measurement

• Use exploratory and confirmatory factor analyses to assess what the empirical structure of the data is for KG classrooms in the Greater Accra Region of Ghana.

• Assess if the measurement structure is the same for public and private schools.

• Compare results with systematic findings from U.S. contexts that consistently show three dimensions of classroom quality in K-3 (Teachstone, 2014):

Emotional support • Positive climate • Negative climate • Teacher sensitivity • Regard for student

perspectives

Classroom organization • Behavior

management • Productivity • Instructional learning

formats

Instructional support • Conceptual

development • Quality of feedback • Language Modeling

32

TIPPS – Factor Analytic Results from Ghana

Emotional support &

behavior management

Positive climate

.83

Results shown are from confirmatory model: RMSEA = .086 CFI = .928 TLI = .912

Negative climate

-.46 Sensitivity & responsiveness

.88 Tone of voice

.78

Behavior management .74

Consistent routine .63

Student engagement .78

33


Instructional support

Connects lesson to teaching objectives

Provides specific & high quality feedback .53

Uses scaffolding for children’s learning and

mastery of subject matter

.74

Emotional support &

Behavior management

.62

34


Supports student expression

Student ideas and interests taken into

consideration

.85

Encourages reasoning/problem

solving .82

Connections to students’ daily lives

.60

Emotional support &

Behavior management


Language modeling .60

35

A comparative perspective for conceptualizing and measuring classroom quality


perspectives

Classroom organization • Behavior management • Productivity • Instructional learning

formats



United States (KG – 3)

Ghana (KG)

Emotional support & behavior management • Positive climate • Negative climate • Teacher sensitivity/tone • Behavior management • Consistent Routine

Supporting student expression

• Student ideas considered • Reasoning/problem solve • Connections to life • Language modeling

Instructional support • Scaffolding (concept

development) • Quality of feedback • Objectives explicit

36

• Measurement invariance is an important step to establish whether differences among subgroups’ observed scores can be explained only in term of mean differences on the constructs / factors of interest.

• Configural invariance establishes that the same number of factors and general pattern of item loadings is the same across groups

• Metric invariance establishes the value of each items’ loading on the factor is the same across groups

• Scalar invariance established that the value of the intercept/threshold for each item is equivalent across groups.

Is the factor structure the same in public and private schools?

37

The measurement structure of classroom quality is invariant across public and private sector classrooms

Configural invariance

Metric invariance

Scalar invariance

Chi-square (df) Contributions: Public Private

276.5 (176)

134.7 141.8

281.6 (184)

136.8 144.8

304.0 (194)

147.5 156.6

CFI .961 .962 .957 TLI .959 .962 .960 RMSEA .060 .058 .060 χ2 change (df) 10.1 (8) 10.0 (11) χ2 change p-value .260 .532

38

The measurement structure of the TIPPS is the same in public and private schools; thus we are confident that we are using the same “ruler” to measure quality in both settings.

+ Part III: Public and Private School Differences in ECE in Ghana School characteristics Teacher characteristics Caregiver and household characteristics Classroom quality Children’s school readiness

39

Private schools are newer than public schools and have smaller class sizes

40

32 32

12

22

0

5

10

15

20

25

30

35

Years established Teacher : pupil

Public Private

Private sector schools are much more likely to teach exclusively in English, and not in mother tongue

• No public schools in our sample teach KG in English only, while over 20% of private schools do.

• Public schools are much more likely to teach in mother tongue, or to teach in a mixture of mother tongue and English.

[VALUE]% English only

[VALUE]% Mother tongue only

[VALUE]% English & mother

tongue

[VALUE]% English & mother

tongue

Public

Private

Percent of schools

41

Teachers in private schools are younger, less educated and less likely to have training in ECD

95

72

30

63

0

20

40

60

80

100

Has any post-secondary training

Has training in ECD

Public Private

• Private schools teachers are 31 years old, on average, compared to public school teachers who are 41 years old on average

42

43

Private school teachers report higher levels of motivation than public school teachers. There are little differences in other aspects of professional well-being.

-0.21

0.07

-0.04

0.04

0.19

-0.06

0.04

-0.04

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

Motivation Burnout Job satisfaction Personal accomplishment

Aver

age

stan

dard

ized

scor

e

Public Private

***

KG classrooms in private schools have lower levels of quality as measured by emotional support/behavior management than public school classrooms.

-0.35

0.046 0.034

-0.4 -0.35

-0.3 -0.25

-0.2 -0.15

-0.1 -0.05

0 0.05

0.1

Emotional support/behavior

management



SD difference (private v. public)

*

44

Note: SD difference estimates derived from scalar measurement invariance model

Caregiver and household characteristics of children enrolled in public and private schools differ in several ways

• Children enrolled in private schools come from more educated and more affluent households compared to those enrolled in public schools. Specifically, primary caregivers* of children in private schools:

Are more likely to be male (56% versus 47%)

Have higher levels of education, including higher levels of post-secondary education.

Have higher levels of wealth and lower levels of food insecurity

Engage in more activities with their children and encourage them to recognize letter and numbers more than caregivers of children in public schools

Are much more likely to pay school fees (93% compared to 24%) and pay higher school fees on average (171 compared to 74 cedis per term)

45

*Primary caregiver defined as “"... the person who takes primary responsibility for the child’s education and who could best talk about the child and his/her experiences in school and at home”

46

KG Children enrolled in private schools more likely to be “school ready”, and are younger than KG children enrolled in public schools (4.9 compared to 5.7 years of age)

0.180 0.113

-0.079

0.420

0.205

0.065

-0.5

-0.3

-0.1

0.1

0.3

0.5

School readiness (total)

Motor Social-emotional Early literacy Early numeracy Executive function

SD difference (private v. public)

***

***

+

***

*

**

+p<.10; *p<.05, **p<.01, ***p<.001

Summary • KG teachers in public schools are better-educated and more likely to have

specialized training in ECD, but less motivated, than private school teachers.

• Public schools are much more likely to implement NALAP – policy to teach mainly in mother tongue in KG – than private schools.

• Yet, KG children in the private schools were more likely to be “ready” for primary school than those in the public schools. o This is probably due to the differences in the types of families that send

their children to public versus private schools.

o Caregivers who send their children to private schools are more educated, less poor, and more involved with their children’s learning and development.

47

+ Part IV: Implications for Policy and Research

48

Implications for Policy • Involvement in discussions with Ghana Education Service (GES) and USAID

on a USAID-funded initiative to train 51,000 KG-P3 teachers in Ghana over the next two years, with elements modeled on QP4G.

• Role of parents has not been considered to date, and there is interest by GES in the findings from the parental education intervention.

• The results of our project are likely to influence the development of programs both for KG teaching training and for educating parents about preschool learning.

• The results of our project will be informative about the value-added of private versus public schools and how policies might best deal with private versus public schools.

49

Implications for Research • Advances in measurement, both in assessment tools and in methods to

analyze their measurement properties, will allow for the most precise estimates of program impacts and advances in education research in SSA more broadly.

• Connections to the wealth of research in high-income countries on classroom quality and child development can serve as a framework for understanding the issues in LMICS, and knowledge gained from LMICs can build on and expand our understanding of education and child development.

• Establishes the foundations for studying children in Ghana longitudinally to learn how aspects of preschool affect their development when of school age.

• Assessment of the desirability of the interventions studied through benefit-cost and cost-effectiveness analyses is critical for scale up implications.

50

Conclusions • There is increasing global evidence indicating the importance of the early years,

including preschool experiences, in developments over the life cycle and across generations.

• But approaches that appear successful in one context do not necessarily carry over to other contexts.

• Our project will provide a careful assessment of the benefits and costs of training teachers in new pedagogy and educating parents about preschool learning in the context of peri-urban Ghana, a context understudied but of increasing relevance with the rapid urbanization in Ghana and in other SSA countries.

• In addition, our findings advance cross-cultural understanding of classroom quality and child learning and development.

51

References

Bidwell, Watine, & Perry (2014). Exploring Early Education Programs in Peri-urban Settings in Africa: Accra, Ghana. Innovations for Poverty Action.

McCoy, D.C. (2015). Author calculations with the Multiple Indicators Cluster Survey.

Seidman, Raza & Kim. Teacher Instructional Practices and Processes System. New York University, NY.

United Nations (2015). World Urbanization Prospects. Department of Economic and Social Affairs, Population Division.

Ghana Education Service (2012). Programme to Scale-Up Quality Kindergarten Education in Ghana. Ministry of Education, Ghana.

Teachstone Training LLC (2014). Teacher-Child Interactions in Early Childhood: Research Summary. Charlottesville, VA: Teachstone.

Milsap, R.E. (2011). Statistical Approaches to Measurement Invariance. New York NY: Routledge.

52

+ Appendix tables

53

54 Means Differences Test in School Characteristics, by Public and Private Schools

Private Public t-statistic p-value

No. of years school has been established 13.9 31.7 7.38 0.000 ***No. of years with KG classes 13.2 15.2 1.60 0.111School has written rules/regulations for staff 34.4% 49.1% 2.30 0.022 *School has formal mentoring system for teachers 40.6% 30.6% 1.61 0.109School has PTA 97.7% 100.0% -1.60 0.110School has curriculum for KG 96.9% 100.0% -1.86 0.064 +School admits all children who wish to enroll 61.7% 65.7% -0.64 0.525Total number of KG children in school 41 80 6.32 0.000 ***Total number of KG teachers 1.9 2.5 4.37 0.000 ***

Main language of instruction in KG1English only 21.2% 0.0% -5.26 0.000 ***Mother tongue only 0.0% 4.8% 2.28 0.024 *Mixture of English and Mother tongue 78.9% 95.2% 3.60 0.000 ***

Main language of instruction in KG2English only 22.2% 0.0% -5.42 0.000 ***Mother tongue only 0.0% 3.9% 2.03 0.044 *Mixture of English and Mother tongue 77.7% 96.1% 4.06 0.000 ***

Family/community outreach (1=not at all true; 2=a little true; 3=mostly true; 4=very true) 2.9 3.3 7.77 0.000 ***

Head teacher characteristicsHead teacher has training in ECD 50.8% 38.0% -1.98 0.049 *Years of experience of head teacher 6.4 4.5 -3.33 0.001 ***Satisfied with job at school 71.1% 68.5% -0.43 0.669Satisfied with decision to be head teacher 88.3% 90.7% 0.61 0.543Wants to transfer to another school 85.9% 77.8% -1.64 0.103Wants to leave the education profession 77.3% 84.3% 1.34 0.183

Sample size 132 108

Mean or %

55

Public Private t-stat p-value

Demographic characteristicsFemale 98.60% 96.60% -1.36 0.174

Age 40.5 30.8 -10.13 0.000***

Training & experienceYears as a teacher 6.79 6.14 -1.02 0.306

Years as a teacher in current school 2.86 3.79 2.56 0.011*

Highest level of education (at least SHS) 97.20% 91.00% -2.73 0.007**

Has any post-secondary training 94.80% 29.20% -18.96 0.000***

Has training in ECD 72.00% 63.10% 2.01 0.045*

Professional well-beingReading knowledge score (% correct) 57.00% 52.50% -2.73 0.007**

Depression and anxiety 2.03 2.01 -0.51 0.614

Motivation 4.61 4.76 3.23 0.001***

Job dissatisfaction (% true)

Satisfaction with job at this school 65.90% 69.10% 0.72 0.47

Satisfaction with decision to be a teach 86.30% 82.80% -0.99 0.321

Desire to switch to another school 20.40% 15.90% -1.23 0.219

Desire to leave the teaching profession 5.70% 11.20% 2.06 0.040*

Burnout 2.07 2.06 -0.13 0.9

Sample size 211 233

Mean or %

Means Differences Test in Teacher Characteristics, by Public and Private Schools

56 Means Differences Test in Caregiver Characteristics, by Public and Private Schools Private Public t-

statisticp-value

Demographic characteristicsFemale (%) 46.82 56.14 -4.30 0.000 *Age 37.58 38.90 -3.39 0.001 *Caregiver's education level (at least SSS/SHS degree) (%) 40.22 20.23 10.12 0.000 * Primary School Only (%) 6.10 8.79 -2.38 0.018 * JHS Only (%) 41.97 37.82 1.95 0.052 SHS Only (%) 16.89 8.47 5.76 0.000 * O/A level only (%) 3.85 2.97 1.11 0.269 Vocational only (%) 6.19 4.24 2.00 0.046 Diploma only (%) 5.43 2.44 3.48 0.001 * BA only (%) 6.10 1.59 5.25 0.000 * Masters/PhD (%) 1.25 0.42 2.03 0.042 *

Is primary caregiver (%) 100.00 100.00 -0.89 0.374Years as primary caregiver 4.64 4.86 -3.14 0.002 *One child listed in school for caregiver (%) 96.74 92.16 -3.80 0.000 *

Economic Well-beingGhana Poverty Scorecard (higher scores=less poverty) 66.67 53.44 25.30 0.000 *Food Security No food due to lack of resources in last 30 days (% yes) 14.77 27.72 0.60 0.551…frequencya 0.24 0.46 -7.01 0.000 *Going to bed hungry in last 30 days (% yes) 8.69 14.70 -0.61 0.545…frequencya 0.15 0.25 -4.32 0.000 *Whole day and night w/o eating in last 30 days (% yes) 3.97 8.95 -0.65 0.513

School Fees Do you or any other person pay school fees? (% yes) 92.64 24.25 -45.24 0.000 * On the average, how much do you currently pay as 170.68 74.36 11.36 0.000 * school fees per term? (Ghana cedis)

Sample size 1,196 944

Mean or %

57 Means Differences Test in Caregiver Involvement and Beliefs, by Public and Private Schools Private Public t-statistic p-value

Parental Involvement (% yes)In the past 30 days have you or any member >15 years… …...Read books to or looked at picture books 77.79 72.31 -1.12 0.265 …...Told stories to child 53.13 50.05 -0.99 0.324 …...Sang songs with child, including lullabies 67.74 60.92 -1.19 0.233 …...Taken child outside the home (market, events etc) 69.93 54.85 -1.58 0.114 …...Played with child 94.09 93.28 -0.63 0.529 …...Named, counted, or drew things to or with child 79.22 69.94 -0.13 0.894

Total Number of Activities Engaged In (max 6) 4.37 3.99 5.40 0.000 *Number of children’s books or picture books for child? 3.56 2.56 5.81 0.000 *

Number of times caregiver or other adult in HH has… …….attended a PTA meeting 1.78 1.75 0.34 0.734 .……attended scheduled meeting with child’s teacher 1.55 1.05 4.29 0.000 * …….attended school or class event 0.74 0.34 9.15 0.000 * …….volunteered or served on school committee 0.15 0.14 0.26 0.792 …….participated in fund raising for child’s school 0.47 0.44 0.70 0.482Total Number of Activities Engaged In 2.75 1.67 1.67 0.000 *Satisfaction with child’s school? (1=Highly Satisfied, 1.89 1.95 -1.92 0.055 + 5=Highly Dissatisfied)

Developmentally Appropriate Practice (1=not very important, 5=very important)How important is it that KG teachers…. …….know about children's needs as they grow and develop? 4.71 4.67 1.74 0.083 …….encourage children to recognize letters or words? 4.75 4.70 2.30 0.022 * …….encourage children to recognize numbers or shapes? 4.73 4.68 1.96 0.050 *

Supporting Child's Social and Emotional DevelopmentHow important is it that KG teachers…. …….help children to build relationships with peers and adults 4.57 4.57 -0.05 0.962 …….help children learn to control their behavior 4.62 4.58 1.50 1.496 …….encourage children to express thoughts and feelings 4.62 4.61 0.59 0.554 …….help children resolve conflicts with other children 4.58 4.55 1.42 0.156 …….discipline and/or behavior guidance styles match the parents 4.52 4.53 -0.17 0.864

Sample size 1,196 944

Mean or %

58 KG Children enrolled in private schools more likely to be “school ready”, and are younger than KG children enrolled in public schools (4.9 compared to 5.7 years of age)

69.4

42.8 40.0

42.7

54.7

72.3

41.2

48.9 46.8

56.4

0

10

20

30

40

50

60

70

80

Motor Social-emotional Early literacy Early numeracy Executive function

Perc

ent

corr

ect

scor

e

Public Private

*** +

*** *

**

+p<.10; *p<.05, **p<.01, ***p<.001

© Institute for Fiscal Studies

Improving the Quality of Centre-based Child-care in Colombia Alison Andrew (EDePo)

Orazio Attanasio (EDePo&UCL)

Raquel Bernal (University of Los Andes)

Sonya Krutikova (EDePo)

Marta Rubio-Codina (EDePo)


Main aim

Study the impacts of two different interventions to improve quality of centre-based child-care provision in urban Colombia to add to the state on knowledge on

effective scalable early childhood interventions.

Our contribution

Inform on the effectiveness of a large-scale government ECD programme in a context of high investment and political good will to ECD


Our contribution


Add to what we know about the relative effectiveness of different types of pre-school interventions


Our contribution


Add to what we know about the relative effectiveness of different types of pre-school interventions

Longer-term: Add to what we know about the channels through which pre-school interventions can impact child development


Outline

• Literature overview & where this study fits in

• ECD policy context in Colombia

• The interventions

• Evaluation design

• Data collection & measurement

• Main impacts

• Implementation fidelity

• Suggestive evidence on mechanisms

• Where next


Existing Evidence

• Widely cited, high, long-term returns to early education programmes such as High/Scope Perry Preschool Programme (Heckman et al, 2010); Abecedarian project (Barnett & Masse, 2007)


Existing Evidence

• Widely cited, high, long-term returns to early education programmes such as High/Scope Perry Preschool Programme (Heckman et al, 2010); Abecedarian project (Barnett & Masse, 2007)

• However evidence on universal pre-school programmes is very mixed:

– Attending pre-school can lead to better cognitive & health outcomes over the shorter and longer term (Berlinski et al, 2007,2009; Bernal and Fernandez, 2013)

– But there are also studies showing null and negative effects in developed countries and LMIC’s (Baker et al, 2008; Rosero & Oosterbeek, 2011)


Existing Evidence

• Quality is a key ingredient in effective center based programmes (Engle et al, 2011):


Existing Evidence


• Structural vs process quality (Yoshikawa et al, 2015)

– School literature: structural quality doesn’t explain a lot of variation in attainment (Hanushek & Rivkin, 2012; Glewwe et al, 2011)

– Pre-school evidence seems to be consistent(Bernal et al, 2016)

– New evidence that process quality matters (Araujo et al, 2015; Nores et al, 2014)


Existing Evidence


• Structural vs process quality (Yoshikawa et al, 2015)

– School literature: structural quality doesn’t explain a lot of variation in attainment (Hanushek & Rivkin, 2012; Glewwe et al, 2011)

– Pre-school evidence seems to be consistent(Bernal et al, 2016)

– New evidence that process quality matters (Araujo et al, 2015; Nores et al, 2014)

• Still work to be done to understand what dimensions of process quality matter for what areas of child development (Mendive et al, 2015)


ECD Policy context in LMIC’s

• Ongoing rapid expansion of pre-primary schooling in developing countries. E.g.:

– Ethiopia: roll-out of pre-school from GER of 4.2% in 2008/09 to 80% by 2020.

– ECD high on the agenda of other African countries: Ghana, Cote d’Ivoire, Rwanda






• But designing appropriate curriculum & ensuring adequate training is hard (e.g. Yoshikawa et al, 2015).






• But designing appropriate curriculum & ensuring adequate training is hard (e.g. Yoshikawa et al, 2015).

• Have the lesson of fast primary school expansion not accompanied by sufficient investment in quality, don’t want to repeat at pre-primary


ECD Policy Context in Colombia

• 2011 national early education strategy “From Zero to Forever” DCAS launched in response to evidence on importance of pre-primary education for growth

• Aim = to deliver high quality integrated ECD services to disadvantaged children

• Many programmes being rolled out within that with emphasis on making existing services more integrated and raising quality of service provision

• First initiative = improvements to Hogares Infantiles (childcare centres providing partly subsidised day care to 2-5 year-olds from poorer hh.)


The Interventions


Govt Improvements:

HIM

Support team + equipment

1)expert in socio-

emotional development (1

per 200 children)

2) pedagogical assistant (1

per 50 children)

3) one time payment of

$52/child for toys, books,

other

The Interventions


Govt Improvements:

HIM

Support team + equipment

1)expert in socio-

emotional development (1

per 200 children)

2) pedagogical assistant (1

per 50 children)

3) one time payment of

$52/child for toys, books,

other

Govt Improvements + private

foundation:

HIM+FE

Teacher training: (1) 17 monthly 3-hour

sessions; (2) 3 hours/week of video

tutoring sessions (3) On-site coaching

=one classtoom observation

Reading programme: (1) books for

participating centres (2) Assessment +

plan + reading promoter (3)Workshops

for teachers, children and parents

Teacher training & reading programme

Why is this comparison interesting?

• HIM focuses on “structural” dimensions of quality (teacher-pupil ratios, provision of materials)

• FE improvements are targeting process quality (teaching practices / daily experience of kids in the class)





• Many school and pre-school investments in LMIC’s (incl Colombia) are focused on structural improvements - nicer buildings , higher teacher-pupil ratios, better equipment but literature emphasises importance of process dimension





• Many school and pre-school investments in LMIC’s (incl Colombia) are focused on structural improvements - nicer buildings , higher teacher-pupil ratios, better equipment but literature emphasises importance of process dimension

This study is an opportunity to directly compare structural improvements to process ones in the context of “real-life” implementation


Mechanisms

Improved child cognitive, language

and socio-emotional development

• Better teaching practices in HI centres • Better stimulation practices at home

• Teachers have more time for pedagogical work • Increased availability of books in HI centres and homes • Children and families show greater interest in reading

HI centres hire psychologists

HI centres hire pedogogical

assistants

Additional pedagogical equipment

is utilised

(HIM)

Teachers complete pedagogical training

Children, parents and teachers

participate in reading workshops

Provision of books in HI centres

(HIM+FE)

Evaluation design


Total of 670 HI’s in Study cities (Bogotá, Cali, Medellín, Barranquilla, Bello, Palmira, Itagüí and Soledad)

198 randomly selected & organised into geographically close groups of 3

40 groups (120 HI’s) selected based on having at least 15 children 18-36 months @ baseline

Treatment 1 - HIM

Govt Improvements

40 HI’s

Total kids: 663

Control

40 HI

Total kids: 661

Treatment 2 - HIMFE

Govt + FE Improvements

40 HI’s

Total kids: 663

Random Assignment

Total sample = 1987 kids 18-36 months

Timeline


Roll-out of govt HIM improvements

• Feb 2013

Baseline

• Mar-May 2013

• Kids age 18 to 36 months

Introduction of FE

• Jun 2013

Follow-up

• Oct-Nov 2014

• Kids age 36 to 54 months

Empirical Strategy


1,1,0,0,0,221101, ''' icslicslicslicslicslslslicsl ZCXYTTY

Our main specification is:

1,icslY

slT1

slT2

0,icslY

0,'icslX

0,'icslC

1,'icslZ

1,icsl

= Outcome for child i in class c in child care centre s in city l at follow-up

= Dummy = 1 if child receives T1 = HIM

= Dummy = 1 if child receives T2 = HIMFE

= Baseline measure of outcome*

= Baseline child characteristics (age and sex)

= Complete set of city dummies (8 cities in total)

= tester/interviewer dummies

= random error term clustered at HI centre level (unit of randomisation)

Baseline Descriptives


HI Mean (SD)

HIM Mean

(SD)

HIM+FE

(SD) P-val :HIM-HI

P-val: HIMFE -

HI

P-val: HIMFE -

HIM N

Male 0.483 0.535* 0.534

0.041 0.070 0.958 1987 (0.500) (0.499) (0.499)

Baseline age (months) 29.525 29.929 28.866

0.287 0.098 0.018 1987 (4.647) (4.437) (4.907)

Parental income (COP000) 1.079 1.030 1.105

0.503 0.734 0.359 1973 (0.711) (0.725) (0.775)

Mother’s education 12.707 12.326 12.682

0.130 0.928 0.161 1968 (2.732) (2.643) (2.660)

Father’s education 12.124 12.010 12.188

0.687 0.811 0.479 1860 (3.080) (3.003) (2.950)

HH size 4.396 4.439 4.259

0.716 0.252 0.116 1987 (1.654) (1.627) (1.589)

Total ASQ score -0.025 0.076 -0.052

0.278 0.782 0.154 1987 (0.987) (1.027) (0.983)

MacArthur Bates Score 0.017 0.021 -0.038

0.961 0.563 0.516 1987 (1.027) (1.014) (0.957)

ASQ-SE Score 0.019 -0.022 0.003

0.661 0.865 0.788 1986 (0.996) (0.990) (1.014)

Weight for age z-score 0.002 -0.215** -0.098

0.003 0.165 0.095 1857 (0.982) (0.982) (0.929)

Height for age z-score -0.574 -0.659 -0.559

0.326 0.858 0.182 1855 (1.040) (1.009) (1.001)

Attrition


Control HIM HIMFE Total

Sample at baseline 661 663 663 1987

Sample at follow-up 616 617 629 1862

Attrition rate 6.81% 6.94% 5.13% 6.29%

Attrition not correlated with treatment status or baseline individual/household characteristics incl baseline

tests

Data Collection

• Household:

– Socio-demographic characteristics, details about mother, father, caregiver, head, child-friendly environment (books, toys, safety)

– Child in hh: basic characteristics, child-care history & current, health & nutrition stts, time-allocation, ASQ

• HI:

– Physical description, compliance with health & safety, classroom quality, headmaster characteristics

– Teachers in HI: qualifications & experience, contract, time allocation & practices, depression, burn-out, job-satisfaction

– Additional experts questionnaires (se expert, nutritionist, assistant)

– Child in HI


Outcome measures


ASQ:SE (Ages and Stages Questionnaire – Social-emotional health)

Daberon School readiness

Woodcock-Muñoz

Working memory, concept formation, phonological awareness,

pre-reading skills,

rhymes

PTT (Pencil Tapping Task) Inhibitory control

TVIP

Receptive Vocabulary

Socio-emotional

Development

Cognitive Development

Constructing outcome measures

• W-M & TVIP: Use same approach as official scoring method (IRT) but using a more flexible model

• Daberon: Same as W-M

• PTT: Number of correct responses






• Scores are then standardized non-parametrically to remove age effects so that measures can be interpreted as relative to the control group i.e. effects are relative to the control group standard deviation






• Scores are then standardized non-parametrically to remove age effects so that measures can be interpreted as relative to the control group i.e. effects are relative to the control group standard deviation

• Child development is composed of many different interrelated dimensions (conceptually and we see strong correlations btw the different measures in our data)

exploratory factor analysis to estimate common underlying constructs




•Fluid reasoning (WM 12), memory for words (WM17), receptive language (TVIP), expressive language (WM14), school readiness (DABERON), inhibitory control (PTT)

• Availble for children 48 months + at follow-up

Cognitive & language development & school

readiness (CLS)

• Receptive language (TVIP), expressive language (WM14), memory for words (WM17). •Availble for children 36 to 54 months at follow-up

Pre-literacy skills

•ASQ: SE raw scores standardised to have mean 0 and sd 1 in control group for full set of sub-scales: self-regulation, compliance, communication, adaptive functioning, autonomy, affect, interaction, total.

Social-emotional development

Main Impacts: Socio-emotional


Main Results: Cognitive impacts


Dif HIM-

Control

Dif

HIMFE-

Control

Dif HIMFE-

HIM

N

CLS (all tests) -0.030 0.151** 0.182*** 1071

(0.072) (0.076) (0.073)

CLS (excl PTT) -0.041 0.148** 0.189*** 1071

(0.073) (0.076) (0.074)

CLS (all kids) -0.066 0.066 0.132*** 1819

(0.069) (0.067) (0.066)

Pre-literacy -0.100 0.045 0.145** 1819

(0.069) (0.066) (0.064)

One-tailed hypothesis tests. All estimates control for gender, city effects, tester effects & baseline scores for

MacArthur Bades CDI and each sub-scale of the ASQ-III. All factors have a mean 0 and sd 1 in the control group.

Age effects removed from the standardized scores prior to factor construction. Each factor constructed using the

following standardized scores: (i) CLS= Cognitive & Language development and school readiness: WM12,

WM14, WM17, TVIP, DAB, PTT (ii) pre-literacy: WM14, WM17, TVIP

Main Results: Individual Assessments


Heterogeneity


Heterogeneity: older 50%

Mechanisms

Improved child cognitive, language

and socio-emotional development

• Better teaching practices in HI centres • Better stimulation practices at home

• Teachers have more time for pedagogical work • Increased availability of books in HI centres and homes • Children and families show greater interest in reading

HI centres hire psychologists

HI centres hire pedogogical

assistants

Additional pedagogical equipment

is utilised

(HIM)

Teachers complete pedagogical training

Children, parents and teachers

participate in reading workshops

Provision of books in HI centres

(HIM+FE)

Measures

• Home Environment:

– Instruments: Family Care Indicators

– Administration: baseline and follow-up in all households

– Measures: factors for each sub-scale

• Classroom Environment:

– Instruments: (1) ECERS (Early Childhood Environment Ratings Scale) /ITERS (2) teacher reported practices and time-allocation

– Administration: baseline and follow-up in classes of a sub-set of randomly selected HI’s

– Measures: first factor for each sub-scale, standardised relative to baseline

Impacts on Home Environment: FCI

Dif HIM-

Control

Dif

HIMFE-

Control

Dif HIMFE-

HIM

N

Play activities with

mother -0.037 0.026 0.064** 1840

(0.034) 90.030) (0.032)

Play activities with

father -0.112* -0.046 0.066 1460

(0.034) (0.056) (0.062)

Variety of play

materials -0.006 0.100 0.107* 1862

(0.057) (0.061) (0.0600

Analysis at child-level (total = 1987). Standard errors clustered at centre level. All estimates control for age, sex,

city effects, interviewer effects, baseline values of the indicator in question.

Impacts on Home Environment: Reading practices (FCI)

Dif HIM-

Control

Dif HIMFE-

Control

Dif HIMFE-HIM N

Number of story books at

home -0.505 -0.007 0.498 1862

(0.542) (0.508) (0.553)

Min/per week looking @

books w/hh member -7.092 6.868 13.960 1987

(10.073) (9.953) (9.047)

Min/per week looking @

books by self -9.982 -7.572 2.410 1862

(6.279) (6.282) (6.050)

Bought books for child to

read in last 6 months -0.054 -0.038 0.016 1862

(0.028) (0.026) (0.026)

Borrowed books for child

to read since June 2013 -0.025 0.136*** 0.161*** 1862

(0.025) (0.031) (0.030)

Analysis at child-level (total = 1987). Standard errors clustered at centre level. All estimates control for age, sex, city

effects, interviewer effects, baseline values of the indicator in question.

Impacts on Class Environment: ECERS/ITERS

Dif HIM-

Control

Dif

HIMFE-

Control

Dif HIMFE-HIM N

Space & Furnishings -0.09 -0.143 -0.08 144

(0.242) (0.335) (0.220)

Personal Care -0.09 -0.851** -0.763* 138

(0.295) (0.336) (0.381)

Language, reasoning,

interaction 0.443* 0.394

-0.079 145

(0.263) (0.241) (0.230)

Activities 0.102 0.438* 0.259 134

(0.274) (0.238) (0.212)

Estimates are at class level. All estimates include baseline controls for number of children registers in centre &

staff/child ratios at baseline & centre-level means of ECERS/ITERS scores. Each outcome is measured using first

factor from exploratory factor analysis of of scores on all items making up the sub-scale, standardised to have a

mean 0 and standard deviation of 1 relative to the baseline. Variation in N due to missing values in sub-scales.

Summary of evaluation results

• No effects on socio-emotional development (or nutrition)

• Children in HIM+FE centres performed significantly better on assessments of cognitive and language development and school readiness than those in HI and HIM centres

→Effect size in the range of 0.15 of a standard deviation

• Effects come through most strongly for older children, boys and children being raised in more favourable environments

• Mechanisms: Children borrow more books and a re-focus of teacher effort away from less productive activities?

Is 0.15 of a standard deviation a lot?


Directions for Future Work: Non-Compliance

• So far we have focused on the impact that the programmes had (ITT)

• For policy & external validity important to know what impact the programme could have had (ToT)

→ Need to study implementation fidelity and determinants of non-compliance

Implementation Fidelity

• 98% of HIM centres had at least one of each of the three supports hired

• 6.3% meets all staffing guidelines (30% socio-emotional expert; 50% pedagogical assistant; 14% nutritionist)

Hiring professionals : HIM & HIM+FE

• 2,592 children reached, 6,000 books delivered, more than 4000 workshops held (mainly for children)

• HIM+FE centres reporting having 73 more books than HI at follow-up

Reading Programme:

HIM+FE

• 327 teachers in 40 HIM+FE HI’s participated in at least one training session →89 received final certificate

• 153 attended all sessions

• 60% turned in all homework

• 58% turned in final project

Pedagogical Training:

HIM+FE

Is 0.15 of a standard deviation a lot?

• Effects in related studies range from none (e.g. Yoshikawa et al ) to 0.2-0.3 SD (Bernal & Fernandez, 2013; Behrman et al, 2004) to 0.3-0.5SD (Nores et al, 2014).

→ Not perfect comparators but in light of these & very partial implementation may suggest that intervention has potential


Directions for Future Work: Mechanisms

1. Getting robust ToT estimates

2. Study the link between the household and teacher behaviours that the intervention (may have) changed and child development


Naïve estimates of the PF: Home Inputs & background characteristics

Pooled Control group

only HIM (T1) only

HIM+FE (T2) only

Home inputs

Varieties of play activities with mother

-0.000 -0.113 0.138* -0.010

(0.044) (0.109) (0.071) (0.062)

Varieties of play materials

0.117** -0.016 0.165** 0.190**

(0.046) (0.074) (0.061) (0.073)

Mother’s education

0.356*** 0.369*** 0.182 0.273*

(0.079) (0.102) (0.160) (0.143)

N 574 190 179 205

Naïve Estimated of PF: Pre-School Inputs (Structural)

Pooled Control group only HIM (T1) only HIM+FE (T2) only

Teacher experience (years)

-0.003 -0.007 0.012* -0.021**

(0.005) (0.009) (0.007) (0.009)

Teacher education: no college degree

-0.186* 0.083 -0.565*** -0.291**

(0.095) (0.185) (0.166) (0.131)

Teacher has ECD qualification

0.006 0.088 -0.254* -0.094

(0.092) (0.141) (0.146) (0.162)

Number of children in the HI center

0.000 -0.004 -0.002 -0.001

(0.001) (0.004) (0.005) (0.002)

Teacher student ratio in HI center

10.890* -19.805 7.600 11.640

(6.294) (11.821) (9.618) (9.444)

HI wealth index 1.013* 0.399 1.777** 0.945

(0.549) (0.953) (0.680) (1.212)

Log per capita expenditure per pupil at HI

0.100 1.215** 0.431 0.694**

(0.222) (0.565) (0.705) (0.311)

Naïve Estimated of PF: Pre-School Inputs (Process)

Pooled Control group

only HIM (T1) only HIM+FE (T2) only

ECERS/ITERS sub-scales

Personal care -0.159*** -0.098 -0.064 -0.189**

(0.027) (0.059) (0.070) (0.076)

Language, reasoning, interaction

0.029 0.072 -0.006 -0.105

(0.046) (0.062) (0.073) (0.161)

Activities -0.000 -0.171** 0.177* -0.067

(0.047) (0.083) (0.092) (0.110)

Adjusted R2 0.309 0.345 0.337 0.391

Going beyond the naïve estimates

• Endogeneity: Parental and teacher behaviour could be correlated with unobserved shocks to the kids and/or missing inputs

• Measurement: measurement error in input variables – exploring ways of applying latent factor models

• Functional form: relaxing the perfect substitutability of inputs assumption in linear models – e.g. Attanasio et al estimate CES PF.

Summary of evaluation results

• No effects on socio-emotional development (or nutrition)

• Children in HIM+FE centres performed significantly better on assessments of cognitive and language development and school readiness than those in HI and HIM centres

→Effect size in the range of 0.15 of a standard deviation

• Effects come through most strongly for older children, boys and children being raised in more favourable environments

• Mechanisms: Children borrow more books and a re-focus of teacher effort away from less productive activities?

Additional slides


Tests

Daberon

General knowledge, body parts, colour concepts, number

concepts, prepositions,

following instructions, plurals,

visual perception, categories.

Vocabulary (WM 14)

Expressive language.

Inhibitory control (PTT)

Understanding instructions and

doing the opposite

Children 48 months +

Fluid reasoning (WM 12)

Long-term memory measure and verbal fluency = ability to learn, store and retrieve stored knowledge

e.g. tell me as many body parts as you can in 20

seconds .

Tests

Concept formation (WM 5)

Measuring inductive reasoning &

flexibility of thought. Child learns rules

based on observation while responding – feedback is given

continuously.

Word memory (WM 17)

Measures aspects of phonological

awareness skills and pre-reading. Child is

asked to repeat words with the

number of words gradually increasing

Sound descrimination (WM 21)

Measures phonetic coding ability. Child

is given a list of three words, two of which

end on the same phoneme – child

must identify which two rhyme.

Children over 48

months

Vocabulary (TVIP)

Measures receptive language

Child remains in

BL HI?

Yes=1344 (72,18%)

No=518 (27,82%)

Child attends

other HI?

Yes=171 (33.01%)

Is that HI in our

sample?

Yes=20 (11.7%)

No=151 (88.3%)

No=347 (66.99%)

In the last 30 days the child

atended another care service?

No=103 (29,68%)

Yes=244 (70,32%)

Which?

Jardín Privado=55 (22,54%)

Jardín de una Fundación sin

ánimo de lucro=23 (9,43%)

Hogar Comunitario=47

(19,26)

Hogar particular de un pariente= 3

(1,23%)

Hogar particular de un no pariente=

2 (0,82%)

Colegio o escuela pública=16

(6,56%)

Colegio privado= 75 (30,74%)

Al cuidado de una niñera=2 (0,82%)

Jardín del ICBF= 21 (8,61%)

Attrition from HI’s

Sample Sizes and Completed Interviews Instrumento

Total esperado en

LB

Total alcanzado en

PS

Tasa de re-

entrevista

Trat1

PS

Trat2

PS

Control

PS

Centros 120 120 100% 40 40 40

Total de niños 1987 1862 94% 617 629 616

Menos de 47 meses 678 674 99% 187 260 227

Más de 47 meses 1309 1188 91% 430 369 389

Peso y talla 1987 1835 92% 603 623 609

Tests cognitivos 1987 1835 92% 611 616 608

Cuestionarios del hogar 1987 1862 94% 617 629 616

Información de maestras a

366 847 231% 272 315 260

Calidad del cuidado: 216 211 98% 68 72 71

ECERS (2+ años de edad) 173 172 99% 55 60 57

ITERS (0-2 años de edad) 43 39 91% 13 12 14

CLASS - 72 - 25 23 24

Profesionales Socio-

emocionales b 98 97 99% 42 41 14

Profesionales de salud o

nutricionistas b 90 87 97% 38 41 8

Auxiliares pedagógicos b 274 264 96% 125 131 8

Consumo aparente 50 49 98% 16 17 16

Frecuencia de alimentos 777 777 100% 242 272 263 a En PS se entrevistaron tantas maestras de LB como pudieron encontrarse más las que fueron contratadas por los HIs entre LB y PS. b Las cifras de la segunda columna corresponden al número de entrevistas esperado, dados los reportes del director sobre el personal que trabaja en el HI en PS.

Main impacts: Nutrition

Dif HIM-

Control

Dif

HIMFE-

Control

Dif HIMFE-

HIM

N

Weight for age z-

score 0.034 0.109*

0.074 1835

(0.045) (0.048) (0.042)

Height for age z-

score 0.030 0.016

-0.014 1834

(0.029) (0.036) (0.034)

BMI for age z-score -0.016 0.109^ 0.125* 1834

(0.054) (0.058) (0.056)

Weight for length z-

score -0.008 0.125^

0.133* 1834

(0.055) (0.059) (0.056)

All estimates control for age, gender, city effects, interviewer effects & baseline z-scores for measure in question.

Z-scores calculated using WHO (2005) growth standards. P-values adjusted using the Sidak procedure. ^

significant with unadjusted p-values.

Main impacts: nutrition

Dif HIM-

Control

Dif HIMFE-

Control

Dif HIMFE-HIM N

Chronic malnutrition

(height for age<-2 sd) -0.007 -0.016

-0.009 1834

0.014 0.012 0.013

Adequate height (height for

age btw -1 and 1 sd) -0.005 -0.002

0.003 1834

0.020 0.022 0.021

Overweight

(zBMI is btw 1 and 2sd) -0.068*** -0.024

0.045** 1834

0.017 0.020 0.018

All estimates control for age, gender, city effects, interviewer effects & baseline z-scores for measure in question. Z-scores

calculated using WHO (2005) growth standards. P-values adjusted using the Sidak procedure.

Bad Apples in Early Education

Pedro Carneiro

Yyannú Cruz-Aguayo

Norbert Schady

Motivation

• Large literature on peer effects in education

• Difficult identification issues

– Reflection problem (Manski 1993)

– Non-random sorting into peer groups

Motivation

• Many papers consider whether the average quality of peers matters: – One set of papers focuses on “naturally occurring variation” in

peer quality, as proxied by gender, race, or SES (Hoxby 2000; Lavy and Schlosser 2011)

– Others focus on natural experiments whereby students were reassigned to different schools (Angrist and Lang 2004; Hoxby and Weingarth 2006; Imberman et al. 2012)

– Yet others focus on elite high schools, using RD strategies (Angrist et al. 2014; Dobbie and Fryer 2013); Pop-Eleches and Urquiola 2013; Jackson 2010)

• The findings from these papers are inconclusive – Angrist et al. (2014): “Findings in the voluminous education peer

effects literature are mixed and not easily summarized”

Motivation

• It may be that it is not the average quality of peers that matters, but the presence or absence of children who disrupt learning – Lazear (2001) focuses on classroom disruption:

• By poorly behaved kids

• By kids who ask a question the answer to which is known by all other students

• Holding constant each child’s propensity to disrupt, there will be more disruption in classes with more children

The Ecuador experiment

Ecuador experiment

• Multi-year experiment that randomly assigned an incoming cohort of ~13,500 kindergarten children to classrooms in 202 schools in Ecuador

• Children randomly reassigned to 1st and 2nd grade classrooms • Very rich data:

– On children: End-of-grade tests in math, language, executive function (4 tests each, at the end of each grade)

– On teachers: Experience, education, gender, tenure status, IQ, Big Five, CLASS

– On household investments in children (at end of kindergarten) – On child behavioral problems: At the end of each school year, we

asked teachers who were the three most poorly-behaved children in their classroom (“bad apples”), as well as specific questions about what it is that these children do: • 11 behaviors, with responses on a Likert scale (1=every day; 2=most days;

3=sometimes; 4=never)

Balance, compliance, and attrition

• Very high compliance with random assignment: 98 percent or higher in all three grades

• Proportion attrited reasonably low • Between k and 1st grade: 11 percent

• Between 1st and 2nd grade: 4 percent

• Every year, some children who transfer in from other schools

• At beginning of 1st grade: 24 percent

• At beginning of 2nd grade: 13 percent

Mean25

th

percentile

75th

percentile

Number of

Observatio

Age (months) 60.8 58.0 63.0 10313

Proportion female 0.50 0.0 1.0 10313

TVIP at baseline 83.3 71.0 94.0 10131

Mother’s age (years) 30.4 25.0 35.0 9819

Father’s age (years) 34.6 29.0 39.0 7858

Mother’s years of schooling 8.8 6.0 12.0 9810

Father’s years of schooling 8.5 6.0 12.0 7840

Proportion who attended preschool 0.62 0.0 1.0 10259

Age (years) 41.9 35 49.5 448

Proportion female 0.99 1.0 1.0 450

Proportion with 3 years of experience 0.04 0.0 1.0 450

Proportion tenured 0.65 0.0 1.0 450

Number of students in classroom 34.1 29 39 452

Notes : This table reports means and values at the 25th

and 75th

percentiles of children in the balanced panel we use for our

estimations and of their teachers, measured at the beginning of the 2012 school year. The TVIP is the Test de Vocabulario en

Imagenes Peabody, the Spanish Version of the Peabody Picture Vocabulart Test (PPVP). The test is standardized using the tables

provided by the test developers which set the mean at 100 and the standard deviation at 15 at each age.

Teachers and Classrooms

Children

Summary statistics, children and classrooms

What we do in this paper

Defining bad and really bad apples

• Really Bad Apples: Identified as the worst-behaved child in the class by his kindergarten, 1st grade, and 2nd grade teachers (28 children) – 6.5 percent of classrooms have a Really Bad Apple

– 1,332 children exposed at least once (13.1 percent of total)

• Bad Apples: Identified as one of the three-worst children in the class by his kindergarten, 1st grade, and 2nd grade teachers, excluding really bad apples (107 children) – 23.5 percent of classrooms have a Bad Apple

– 4,267 children exposed at least once (41.9 percent of total)

Results

Who are the (really) bad apples?

.

1 2 3

Really Bad

appleBad apple Never on list

(n=29) (n=117) (n=10,178)

Age (in months) 62.3 61.4 60.8 0.321 0.071

Proportion female 0.03 0.08 0.50 0.421 0.000

TVIP at baseline 0.13 -0.03 0.04 0.457 0.698

Mother’s age (years) 28.5 29.1 30.4 0.625 0.043

Father’s age (years) 33.8 34.9 34.6 0.632 0.872

Mother’s years of schooling 9.7 8.9 8.8 0.284 0.517

Father’s years of schooling 7.6 8.2 8.5 0.520 0.502

Proportion who attended 0.69 0.71 0.62 0.857 0.102

Average executive function 0.02 -0.24 0.07 0.088 0.000Notes : Bad apples 1 are children who teachers said were the worst-behaved in the class in all three grades; bad apples 2 are

children who were among the 3 worst-behaved children in all grades, and were the worst-behaved in 2 of the 3 grades; bad

apples 3 are children who were among the 3 worst-behaved children in all grades, and were the worst-behaved in 1 of the 3

grades; bad apple 4 are children who were among the 3 worst-behaved in at least one grade, but never the worst-behaved in the

class; never on list are children who were not reported by teachers as being one of the 3 worst behaved children in any of the 3

grades.

Observable characteristics of bad apples and other children in sample

Definition of bad apple

F-test:

(1)=(2)

F-test:

(1)=(2)=(3)

(1) (2) (3) (4) (5) (6)

I II I II I II

Really Bad Apples -0.070*** -0.051** -0.046* -0.034 -0.066*** -0.050**

(0.026) (0.025) (0.024) (0.023) (0.024) (0.021)

Bad Apples

One Bad Apple 0.023 0.011 0.014 -0.000 0.021 0.007

(0.019) (0.014) (0.017) (0.015) (0.018) (0.013)

Two Bad Apple 0.050 0.032 -0.002 -0.041 0.027 -0.006

(0.051) (0.048) (0.037) (0.030) (0.040) (0.036)

School-by-grade Fixed Effects Yes Yes Yes Yes Yes Yes

Student Fixed Effects No Yes No Yes No Yes

Observations 30,534 30,534 30,534 30,534 30,534 30,534

Classrooms 1347 1347 1347 1347 1347 1347

Schools 196 196 196 196 196 196

The Effect of Badly-Behaved Children on Learning Outcomes

Notes: Standard errors (in parentheses) clustered at the school level. * significant at 10%, ** at 5%, *** at 1%.

All regressions include lagged total tests scores (with baseline TVIP score in first year), average age in classroom, fraction of boys

in classroom; Model I also includes a student's age and gender.

Maths Language Total

.

Really bad apple effects, by class size

Schools with small classes

Schools with large classes

(1) (2) (3) (4)

I II I II

Panel A: Math Score Effect of Bad Apple on Peers (Def 1) -0.044 0.009 -0.089** -0.092*** (0.030) (0.034) (0.037) (0.031)

Panel B: Language Score Effect of Bad Apple on Peers (Def 1) -0.026 0.026 -0.054* -0.070*** (0.039) (0.033) (0.030) (0.026)

Panel C: Math and Language Score Effect of Bad Apple on Peers (Def 1) -0.043 0.017 -0.081** -0.093*** (0.028) (0.027) (0.035) (0.025) School-by-grade Fixed Effects Yes Yes Yes Yes

Student Fixed Effects No Yes No Yes

Observations 13,350 13,350 17,505 17,505

Classrooms 679 679 669 669

Schools 111 111 85 85

Notes: Standard errors (in parentheses) clustered at the school level. * significant at 10%, ** at 5%, *** at 1%.

All regressions include lagged total tests scores (with baseline TVIP score in first year), average age in classroom, fraction of boys in classroom; Model I also includes a student's age and gender.

What do they do that disrupts learning?

Child Behaviors, Bad Apples and Really Bad Apples

Really Bad Apple (n=28)

Bad Apple

(n=107)

Does he get angry or frustrated easily? 25% 11% 0.065

When you speak to him or give him instructions, does he answer with bad manners? 4% 2% 0.592

Is he disobedient or does not respect the rules of the class? 11% 3% 0.073

Does he misbehave intentionally (e.g. bother other children)? 18% 7% 0.064

Does he blame others when he makes a mistake or misbehaves? 11% 1% 0.007

Is he resentful with other children? 4% 1% 0.308

Does he fight with other children or threaten them? 11% 6% 0.342

Does he beat, kick or bite other children? 18% 5% 0.019

Does he intentionally break toys, books or other objects? 4% 2% 0.592

Does he frequently interrupt the class? 11% 1% 0.007

Is he extremely restless and has difficulty sitting in his seat? 21% 14% 0.346

Notes: A child is defined as a Really Bad Apple if his kindergarten, first grade, and second grade teachers all identified him as the worst-behaved child in the class, and as a Bad Apple if his kindergarten, first grade, and second grade teachers all identified him as one of the three-worst behaved child in the class (excluding Really Bad Apples). The numbers in the table refer to the proportion of Bad Apples and Really Bad Apples whose kindergarten, first grade and second teachers all reported that the child carried out the behavior in question every day

Conclusions

• First large-scale experiment (in developed or developing countries) in which children randomly assigned to classrooms in year t, re-assigned in year t+1, reassigned again in year t+2

• Clean identification, very high levels of compliance • Main result: Substantial effects of really bad apples on

learning outcomes – Having a really bad apple in the classroom lowers learning

outcomes by ~0.05 standard deviations – Bad apples have a bigger effect on learning outcomes in larger

classrooms (0.09 standard deviations) – By way of comparison, using these same data, Araujo et al.

(2016) estimate teacher effects of 0.10 standard deviations

Discussion of Early Schooling Session 1

EDePo/IDB Conference

Sarah Cattan

Institute for Fiscal Studies

June 9 2016

Overview of the session

• Three very interesting papers on the development of children during preschool

and public policies to improve preschool quality

• Papers are particularly relevant given rapid expansion of preschool education and

concerns over quality in many developing countries

• Themes and questions addressed in the session:

1. What is preschool quality?

2. How is quality produced?

3. What is the impact of preschool quality on child development?

4. What interventions can improve preschool quality in a scalable way?

• In this discussion, I will reflect on how the 3 papers advance our knowledge of

these questions and attempt to draw lessons for the design of public policies

Discussion





What is preschool quality?Structural vs. process quality

• Two distinct measures of quality of preschool setting

1. Structural quality: child-staff ratio, classroom size, qualifications of the staff, etc.

2. Process quality: nature of the care provider-child interactions and activities to

which the child is exposed

• Process quality is harder to measure than structural quality

• Assessments are developed by psychologists

• Assessed through observation of classrooms

• Rare in datasets

• Emphasis of these instruments is to measure aspects of classroom and teaching

that matter for child development, broadly or holistically defined

• Requires a knowledge of the production function for child development

What is preschool quality?Advances in measurement

• A contribution of this session is to make advances in the measurement of quality

• TIPPS in Ghana (paper 1)

• ECERS/ITERS in Colombia (paper 2)

• Classroom Assessment Scoring System (CLASS) in Ecuador (Araujo’s presentation

tomorrow)

• Process quality is a multi-dimensional construct

• Use of latent factor models can help identify key underlying dimensions

• Emotional support

• Behaviour management

• Instructional support

• Student expression support

• Classroom organization

What is preschool quality?Advances in measurement

• How do different scales compare with each other within the same context?

• Comparative work could help design better/cheaper instruments

• How does the same scale compare across contexts?

• TIPPS lacks configural invariance between US and Ghana

• Is it possible and even desirable to have comparable measures across contexts?


perspectives

Classroom organization • Behavior management • Productivity • Instructional learning

formats



United States (KG – 3)

Ghana (KG)

Emotional support & behavior management • Positive climate • Negative climate • Teacher sensitivity/tone • Behavior management • Consistent Routine


• Student ideas considered • Reasoning/problem solve • Connections to life • Language modeling

Instructional support • Scaffolding (concept

development) • Quality of feedback • Objectives explicit

Discussion





How is quality produced?

• Knowledge of the production function of childcare quality is important:

• Can tell us about the inputs that are most productive and thus the interventions

that can increase quality

• A relatively old and small literature in economics attempts to estimate

production functions for childcare quality (Blau, 1997; Blau and Mocan, 2002)

• In this literature, structural measures of quality (among others) are thought to

be inputs to the production of ”process quality”

The production function for childcare qualityA simple framework

• The production function for childcare process quality of room i in center j is:

Qij = Qi(Nij1, ..., NijT ,Mij , Rj , εij) (1)

where:

• Qi: Technology of childcare is likely to differ across age groups

• Nijk: Number of staff-hours of skill k in room i of center j

• Mij : vector of characteristics of children in the room (group size, race, parents’

education)

• Rj : vector of center characteristics (location, for profit/non-profit, characteristics

of head teacher, cleanliness)

• εij : unobserved inputs

The production function for childcare qualityExisting literature

• Blau (1997) and Blau and Mocan (2002) estimate the model using ECERS as

measures of process quality in a dataset of US childcare settings

• Findings suggest that:

• Center characteristics (staff-pupil ratio, average qualifications of staff, group size,

staff turnover) not hugely important for process quality

• Inputs in production function explain at most 50% of the variation in center or

classroom quality

• Implications:

• We are missing important inputs

• Regulating childcare quality on ”standard” measures of structural quality may not

be very effective

The production function for childcare qualityPapers presented today offer important avenues of research

• What teacher characteristics/skills matter?

• Other skills, besides those captured by qualifications, may matter - Ghana and

Colombia projects collect rich measures of skills that could matter (e.g.

motivation)

• HIM and HIM+FE in Colombia boosted skills of teachers (knowledge of child

development, pedagogical strategies) in ways that matter for process quality

• What pupil characteristics matter?

• Existing literature measures pupil characteristics as group size and demographics

• Ecuador study suggests that behaviour of students may affect teaching quality

• An interesting direction for research would be to incorporate standard quality

production function with information on the distribution of pupil achievement and

behaviour

• Heterogeneity results in Ecuador suggest the presence of interactions between

group size and distribution of behaviour

Discussion





What is the impact of preschool quality on child development?

• Many evaluations of early intervention programs delivered through the provision

of childcare (Almond and Currie, 2006, 2010)

• Most famous are Perry PreSchool Program, Abecedarian Program, and Head Start

• Important methodological advances in the identification and estimation of

non-linear production functions for child development

• Cunha and Heckman (2007), Cunha, Heckman and Schennach (2010)

• Recent work extends the framework to understand the mechanisms through

which early interventions work and improve child development

• Attanasio et al. (2016) presented by Costas Meghir tomorrow

• A natural direction for research is to use production function framework to

understand the mechanisms underlying center-based interventions

• Colombia study is heading this way

• Another interesting direction would be to think about the role of peer effects in

this framework

• Ecuador study provides a unique opportunity to look at interactions between

teaching quality and peer effects (and other inputs)

Discussion





What interventions can improve preschool quality in a scalable way?

• Why is there even a need to intervene to improve quality?

• One might argue that if quality improves children’s outcomes and parents care

about their children’s development, the presence of competition in the childcare

markets should be enough to drive quality up

• There are a few reasons why this may not be the case

• Parents may not know the importance of preschool quality for their children’s

development

• Parents may care about other attributes of childcare outside (or along with) quality

• It is hard for parents to distinguish high-quality and low-quality providers

• We need policies aimed at improving quality (while maintaining wide access)

• Papers in this session encompass both supply-side and demand-side interventions

Supply-side interventions

• An example of supply-side intervention is to require that preschool staff go

through a particular training

• Ghana and Colombia studies evaluate particular forms of training

• Colombia model (HIM+FE) seems to be effective, relatively cheap and potentially

easy to scale up because it builds on the existing infrastructure

• Regulations about minimum qualifications or training have been a common

policy in developed countries

• The government of Rwanda is currently working on a regulatory framework

• Regulations can ensure minimum standards of quality, but are not without

unintended consequences (Hotz and Xiao, 2011)

• They can increase price of preschool and price some families out of the market

• They can force some providers to exit and thus reduce supply of preschool services

• Instead of regulations, we may want to think about financial incentives for

providers to provide high quality care

• Subsidies linked to quality (New Zealand)

Are supply-side interventions enough?

• Probably not, because they don’t address the market failures driven by lack of

information among consumers (parents)

• Parents may not know what constitutes quality and its importance for child

development

• Emerging evidence on the extent to which parents have distorted beliefs about

the role of home investments for child outcomes

• Elo, Culhane and Cunha (2016), Attanasio and Cattan (2016), Attanasio, Cunha

and Jervis (2016)

• It is reasonable to think that such distortions also apply to the role of childcare

quality for child development

⇒ Parental education about the importance of preschool quality for child

development and about what constitutes childcare quality will help correct this

market failure

• Parental education component of the Ghana intervention

Making information about preschool quality available is important

• Even if parents care about quality, differences in process quality across

alternative providers is hard to observe (interactions, quality of nap time...)

• If parents cannot distinguish between high-quality and low-quality centres, they

will not be willing to pay higher fees for quality and high quality centres could be

driven out of the market (adverse selection)

• Some empirical evidence of asymmetric information in US (Mocan, 2007)

• Parents make some errors in their predictions of process quality

• Parents try extract signals of quality from center attributes, but these attempts

are for the most part unsuccessful

⇒ Making information about the quality of preschool settings widely available is

likely to be an important policy component

• Interventions could test different ways to make this information available

Documents

Quality Preschool for Ghana: Advancing Research … Conference 2016/Session… · Quality Preschool for Ghana: Advancing Research Methods to Support Policy Change . ... • Sustainable