RTI International

SYMPOSIUM:

FOUNDATIONAL FIRST FIVE NEED, MEASUREMENT, RESPONSE IN EARLY CHILDHOOD

CHAIR: RACHEL HINTON, DFID

PAPERS:

LUIS CROUCH, RTI INTERNATIONAL

ABBIE RAIKES, TECHNICAL LEAD, MELQO, UNICEF

MARTIN WOODHEAD, OXFORD UNIVERSITY

UKFIET CONFERENCE, OXFORD, UK, 15-17 SEPTEMBER 2015

RTI International

RTI International is a trade name of Research Triangle Institute. www.rti.org

Stumbling at the First Step: A Triple Crisis at the Foundations of Education Systems

UKFIET Conference

Oxford, UK

Luis Crouch

20 August 2015

Note: author realizes it will be difficult to get through this in 15 minutes. I will skip some slides. Also, some of the slides go quickly as they build on each other. Finally, it is hoped, due to early publication, that some will read ahead of time.

RTI International

Summary of claims

► “Learning crisis” starts with the “Foundational First Five” (FFF)

► Addressing it without resolving the FFF will not work

► Much of the cost is already being paid for

► We know (more or less) what to do – enough to get started

RTI International

Outline

1. Clues as to the problem– Bulge in the early grades– Cross-sectional evidence on correlates with provision of

early childhood care– Reading inability by grade 3, 4

2. Cost considerations

RTI International

What we don’t do

There is vast literature (Heckman, etc.) on the returns to investment in the early years

This is great but…

The paper takes its cue from “planning” sorts of information that MoE planners and policy-makers may be familiar with

“We sensed what you are talking about, but we had never looked at the statistics in this way…”

RTI International

Clue 1: The bulge in the early grades

0.25

0.5

0.75

1

1.25

1.5

1.75

2

2.25

2.5

2.75

Enro

llmen

t Gra

de fo

r Age

Age for Grade and the Early Grades Bulge

RTI International

Results for key four indicators

Medians of key ratios in countries with ratio of Grade 1 enrollment to population > 1.3Ratio or indicator Median in

“high bulge” countries

“Ideal”

Grade 1 enrollment to population aged 7 1.50 ≈ 1 to 1.1

Grade 2 enrollment to Grade 1 enrollment

0.82 ≈ 1 or so

Grade 2 enrollment to population aged 8 1.28 ≈ 1 to 1.1

Gross intake ratio into primary schooling 1.27 ≈ 1 to 1.1

Pre-primary gross enrollment ratio 0.24 ≈ 1.0

RTI International

P1

P2

P3

P4

P5

P6

P7

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000 1,400,000 1,600,000 1,800,000 2,000,000

Enrollment Pyramid

Ugandan case: What do the most basic numbers tell us?

RTI International

P1

P2

P3

P4

P5

P6

P7

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000 1,400,000 1,600,000 1,800,000 2,000,000

Enrollment Pyramid

What do the most basic numbers tell us?

What is the size of the job?

RTI International

P1

P2

P3

P4

P5

P6

P7

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000 1,400,000 1,600,000 1,800,000 2,000,000

Enrollment Pyramid

What do the most basic numbers tell us?

What is the size of the job?

This big?

RTI International

Trick question:

Common claim in Uganda: Survival rate to Grade 5 is only 60% Can this be?

RTI International

P1

P2

P3

P4

P5

P6

P7

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000 1,400,000 1,600,000 1,800,000 2,000,000

Enrollment Pyramid

What happens if we overlay the population?

RTI International

P1

P2

P3

P4

P5

P6

P7

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000 1,400,000 1,600,000 1,800,000 2,000,000

Enrollment with Population Overlay

Population of Appropriate Age Enrollment

What happens if we overlay the population?

What do we notice?

Compare population and enrollment carefully.

RTI International

P1

P2

P3

P4

P5

P6

P7

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000 1,400,000 1,600,000 1,800,000 2,000,000

Enrollment with Population Overlay

Population of Appropriate Age Enrollment

What is the size of the job now?

This small?

What happens if we overlay the population?

RTI International

Can the bulge, and the 120% gross intake ratios be due to over- and under-age intake?

15

Most likely no

Not if it has been going on for a decade or more

One a child is enrolled as new, the child is enrolled, full stop…

You cannot be “new” twice

So the 40% of over-enrolment would have to be new and over-aged or under-aged… But that is not possible…

RTI International

2001 2002 2003 2004 2005 2006 2007 2008 2009 20100

20

40

60

80

100

120

140

160

180

200

Gross Intake Ratio into Grade 1: One Decade

Uganda case… take a look at the intake ratio for a decade…

16

RTI International

2001 2002 2003 2004 2005 2006 2007 2008 2009 20100

20

40

60

80

100

120

140

160

180

200

Gross Intake Ratio into Grade 1: One Decade

Now, what is the “normal intake”? 100%

17

RTI International

Now, what is the “normal intake”? 100%

182001 2002 2003 2004 2005 2006 2007 2008 2009 2010

0

20

40

60

80

100

120

140

160

180

200

Gross Intake Ratio into Grade 1: One Decade

Now suppose you start out with half the kids out of school in 2000…

RTI International

Now, what is the “normal intake”? 100%

192001 2002 2003 2004 2005 2006 2007 2008 2009 20100

20

40

60

80

100

120

140

160

180

200

Gross Intake Ratio into Grade 1: One Decade

… Even with 50% OOS in 2000, in just a few years the over-age due to that would have been taken in, the top triangle compensating for the bottom one, because the system was over-intaking by 60% to 80%.

And then you have all that rectangle to spare: impossible.

RTI International

20Source: DHS Surveys

Yet we know that even in 2000 the total enrollment in all of primary was already > 100% of the population!

RTI International

Perhaps the numbers are exaggerated? Or bad data?

There may be some exaggeration in the numbers

In 2000 and 2011, the official statistics were: 130% and 110% Gross Enrollment Ratio in Primary

We see from DHS estimates of 120% and 100% respectively. Only a 10 point difference in each case.– This is an independent survey and the numerator and

denominator are the same source.– So, unlikely to be exaggerated or bad data.

So if there is some exaggeration, it is relatively minor.

21

RTI International

An interesting other take

Look at aging in school

23Note: am not sure when these data are for, but probably reasonably recent; problem may be smaller today but probably persists

Data from IDS Sussex’s (Keith Lewin)“CREATE” project

24

Look at the peak for grade 1:Very peaked and right at 6 (green dotted lines)

Then by the time you get to grade 3, the peak is for age 9, and curve is much flatter (non-peaked)

It is the system that is over-aging the kids, not late enrollment

Data from IDS Sussex’s (Keith Lewin)“CREATE” project

RTI International

Which means that it is the system that is over-aging the children

It is not late entry that is aging them actively, though there is some late entry

The children age 1.7 years in grade 1 in Moz, 1.25 in Uganda

And age again 1.15 years in grade 2, 1.25 in Uganda

By the time they get to grade 3-4 they are often 1 full year over-age even if they entered at normal age

So the children are over-aging while in school

25

RTI International

Clue 2: All this correlates strongly with lack of ECD The “over-enrollment,” apparent dropout, permanent

“fake” intake of 125%, are all correlated with lack of ECD, lack of preparedness

Bottom Quintile 2nd Quintile Middle Quintile 4th Quintile Top Quintile0

20

40

60

80

100

120

140

160

G1to7 G2toG1 GERPRE INTAKE

GDP per capita groupings

Pre-primary enrollment ratio

RTI International

Internal correlation between key "early years" variables

Grade 1 Enrollment to Population of Age 7

Grade 2 Enrollment to Grade 1 Enrollment

Pre-primary Gross Enrollment Ratio

Gross Intake Ratio into Grade 1

Grade 1 Enrollment to Population of Age 7 1

Grade 2 Enrollment Grade 1 Enrollment -0.64 1

Pre-primary Gross Enrollment Ratio -0.27 0.47 1

Gross Intake into Grade 1 0.86 -0.55 -0.16 1

Note: in the paper these correlations are “controlled” for the effect of overall development as proxied by GDP per capita. The results don’t change much.

RTI International

Clue 3: “Reading crisis”Testing in some 53 language/country combinations: about 50% of kids unableto read any words in the language of instruction

RTI International

Efficiency implications… Countries “with bulge” spend 46% of ed budget on

primary Countries that control the bulge spend 36% of ed budget

on primary Bulge in P1 and P2 in “worst 40” countries = 1.2 years of

education Spend it on ECD! Spending / pupil explains only 9% of variance in

completion rates The “foundational factors” explain 34% of variance in

completion rates Control the bulge, provide ECD, get better completion Fiscally neutral? Maybe. Socially efficient, yes!!!

RTI International

Optional technical note

A regression of the primary completion rate against public expenditure per pupil as a share of GDP per capita finds has R2 of only 0.09.

A regression of completion on the “foundational factor” has R2 of 0.34: the foundational factors are the “efficiency explanation”

Principal Component Coherence of Foundational Factors

Proportion of variance in Primary School Completion Ratio accounted for by the first component

0.67

 

FactorsCoefficients in Principal Component

“Excess” of Grade 1 Enrollment to Population of Age 7 0.55

Grade 2 Enrollment to Grade 1 Enrollment -0.53Pre-primary Gross Enrollment Ratio -0.36“Excess” of Gross Intake into Grade 1 0.54

RTI International

Suggested discussion questions

For those with field experience in the classroom (especially early childhood):– Does “bulge in grade 1” (due to too-young kids), drop-off to grade

2, etc., jibe with your experience of not-enough ECD and grade 1 being used as substitute?

For those with an interest in data: could the data be wrong?– Have checked against MICS, other ways to confirm?

For those with interest in policy: – Can arguments such as “over-enrollment in grades 1 and 2” =

1.2 “wasted” years of schoolingdo ECD be useful?

RTI International

More Information

Luis CrouchVP and Chief Technical Officer

International Development Group

RTI

+1 202 728 2058

lcrouch@rti.org

Global Measurement in

Early Childhood Abbie Raikes

Measuring Early Learning Quality and Outcomes (MELQO)UNICEF

UKFIET, Oxford, UK, September 2015

Important Era for Early Childhood Development

• Increasing awareness of the importance of investments in early childhood development

• Many new program models and growing evidence of the importance of ECD through randomized trials and other research studies

• Measurement as one important element to track trends, investments at the population level

Sustainable Development Agenda, Proposed Goal 4, Target 4.2

4.2 by 2030 ensure that all girls and boys have access to quality early childhood development, care and pre-primary education so that they are ready for primary education

Realizing Promise of ECD Requires Good Measurement

Measuring Education in the Sustainable Development Era

• Shift in direction of national monitoring • Emphasis on measurement of learning, not just access• Expansive definitions of learning that include many undefined

concepts• Resources invested in education overall, let alone measurement, are

limited

Data must serve multiple users – globally comparable is one factor among many, and must be able to use the data for improvement

Measurement Framework for Children Birth to 8 Years

Raikes, Britto & Dua, 2014. A Measurement Framework for Early Childhood: Birth to 8 Years of Age, Institutes of Medicine

What is MELQO?

• Consortium following from Learning Metrics Task Force, WHO Birth to 8 Measurement Framework

• Build on existing assessments: Great work to date

• Develop prototypes for measurement of quality and child development/learning between ages 4 and 6, with emphasis on low-income countries

• “Core Team” led by UNICEF, UNESCO, World Bank and Brookings Institution

Vision of what we will achieve …

• Goal of free, open-source items to allow easy use and adaptation • Clear application to decision-making in policy and practice

Flexible, feasible measures to provide on-going information about children’s development and learning and quality of learning environments

Field-testing in 2015 followed by full validation in upcoming years

What do the MELQO tools include?

• Child development and learning (teacher/parent report and direct assessment):

• Pre-academic skills: Math and literacy • Social/emotional skills• Executive Function

• Quality of Learning Environments: 7 Domains• Observation of classroom settings• Teacher characteristics

Framework for Child Development/Learning

Child Development & Learning

InteractionsMaterials

Inclusiveness

Professional DevelopmentTeacher Motivation

ResourcesSchool Leadership

Policy structure, including M&E Framework

Child counts objects

Child Names

NumbersCounts up

to 20

Teacher has training on

math instruction for young children and receives

regular feedback

Teacher works with small groups of

children to teach each child

at individual level

Child has access to counting

materials like blocks and other

small objectsCurriculum with age-appropriate expectations for math

Adequate funding for

preschools to buy materials,

ensure reasonable class sizes

Talk about numbers – open-ended questions and dialogue

If the child is to achieve these learning outcomes …

Classrooms should have these characteristics …

And programs must ensure this happens...

Through policies and funding at the national level

Existence of a strong, effective monitoring and evaluation system with reliable data and incentives to improve

Phases for Testing Measures

• Do items work well? Are they feasible to use and train?

• Can we find a common core, or do items need to be generated everywhere?

Feasible

• Can we link child development/quality in a meaningful way in the item design?

• Can we establish validity for both instruments?

Valid • Are measures used for improvement?

• How do measures link to longitudinal achievement in early grades?

Useful and Predictive

Data Collection to Date

• Partnerships with RTI, Save the Children, USAID and many experts

• Child Development and Learning • Tested in Kenya, Sudan, Bangladesh, Laos, Mongolia, Madagascar• Beginning Tanzania this fall for validation study, followed by nationally-

representative sample in 2016

• Quality• Observational measure and teacher/supervisor interviews developed• Refining and testing in Tanzania in 2016

Interpreting Data

• Cultural influence vs. ability – how best to pull these apart

• Hard to get a normal distribution – many of the tasks tend to be more binary; children either get them or not

• Estimating how sensitive to age – best to have information on context

• Construct validity – are the measures measuring the same construct in all countries?

Kenya: Forward Digit Span by Age: % Of Children Answering Items Correctly at Each Age

Age 4 Age 5 Age 6 70

5

10

15

20

25

30

35

40

45

50

0 1 2 3 4

Sudan: Forward Digit Span: % Of Children Answering Items Correctly at Each Age

Age 5 Age 6 Age 70

10

20

30

40

50

60

0 1 2 3 4

Where to go from here? MELQO Phase 2• On-going technical development

• Defining what is comparable and not• Testing quality instrument and drawing connections with M&E system

• Clarifying workable use for population-based tools• Program evaluations • Occasional use by governments for monitoring • On-going use by governments or other entities as part of continuous quality

improvement

• Getting to “scale”: Strong institutional backing (UNICEF) coupled with clear plans for implementation and alignment with purpose for tools

• Estimating cost

Conclusions

• Demand for ECD measurement is strong

• Important to create tools that are efficient and culturally relevant

• Ensuring use for improvement is a central question

Opportunity for great improvement in coordination and accuracy of ECD data in upcoming years

Placeholder for Martin Woodhead’s PPT