Positive Impacts and Return Positive Impacts and Return on Investment: What We on Investment: What We

Can Learn from Head Start Can Learn from Head Start ResearchResearch

Positive Impacts and Return Positive Impacts and Return on Investment: What We on Investment: What We

Can Learn from Head Start Can Learn from Head Start ResearchResearch

Hilary ShagerHilary Shager

WI Department of Children and WI Department of Children and FamiliesFamilies

April 25, 2012April 25, 2012

Overview

• Being a good consumer of Head Start research

• Early cohort research• The National Head Start Impact

Study• Early Head Start Research• What’s new in Head Start Research• Remaining research questions

“...finally there is indisputable evidence about the program's effectiveness, provided by the Department of Health and Human Services: Head Start simply does not work.”

-Joe Klein, Time Magazine,

July 2011

“Head Start improves the odds and the options for at-risk kids for a lifetime. Kids that have been through Head Start and Early Head Start are healthier, more academically accomplished, more likely to be employed, commit fewer crimes, and contribute more to society. Simply put: Head Start works. It’s been proven.”

-Letter to Congress, signed by over 270 top early childhood researchers

March, 2011

Being a good consumer of Head Start research• Lots of Head Start research• Lots of rhetoric

– Which research is good research?– What do the results tell us about

Head Start’s effectiveness?– What are the caveats?

• Bottom line…– There is a lot to be excited about!

Challenges in Head Start Research

• Lots of things affect educational, health, family outcomes– Hard to isolate the “true” impact of

Head Start programming

• Finding the appropriate control group– Head Start targeted at particularly

disadvantaged children and families

Questions to ask1) Is the evaluation design strong enough to

produce trustworthy evidence?2) What program services were actually

received by participating children and families in the comparison group?

3) How big an impact did the program have?

4) Do the program’s benefits exceed its costs?

5) How similar are the programs, children, and families in the study to those in my community?

Research 101: Research Design

• Gold Standard=Experiment– Randomized controlled trial

• Quasi-experimental approaches– Does the control group really look like

the Head Start population?– What services do children and families

in the control group get?• External validity

– If I reproduced the study in my community, would I get the same results?

Research 101: Effect Size

• What is an effect size?– Measure of “magnitude” of impact

• The bigger, the better!• More meaningful than “statistically

significant”

– Fraction of the difference between the treatment and control group over variation• Allows comparison across different

measures• Need this to do a cost/benefit analysis

Early cohort research: Some history

• The Westinghouse study (1968)– Quasi-experimental approach– Focus on cognitive outcomes only– Almost defunds Head Start– Consortium of researchers rallies

against the study

• We need better research on more outcomes!

• We waited a long time...

Janet Currie and colleagues: Sibling fixed effect studies

(1990s)

• Use nationally representative data sets• Include families in which one sibling

went to Head Start and one didn’t• Teases out all the “fixed” family-

related factors that influence outcomes• Still questions about why one sibling

attended and not the other

Currie et al. findings• Large gains in vocabulary test scores

– Equal to about 1/3 of achievement gap– Effects persist into adulthood for whites, but fade quickly for

African-Americans• Reduced probability of retention (47%)

– Again, only for whites• More likely to be immunized (8-11%)

– Both whites and African-Americans• Cost-benefit implications

– Increased test scores would increase expected future wage by 4%

– Reduction in retention would lead to 5% decline in probability of high school drop-out

– Preventative health savings– Definitely passes test for whites, question for African-Americans

More Currie et al. findings

• Educational attainment (whites only)– 22 percentage points more likely to complete high school– 19 percentage points more likely to attend some college

• Less likely to be arrested or charged for crime– 12 percentage points, African-Americans only

• Positive effects for Hispanic children– Closes 1/4 gap in test scores– Closes 2/3 gap in probability of grade repetition

• Maybe effects less, fade for African-American children because they attend lower quality schools later

Ludwig & Miller (2007)• 50 to 100% increase in Head Start funding in

1960s and 70s associated with– Increase of schooling attainment of about 1/2 year– 15% increase in likelihood of attending some

college– Found for both African-Americans and whites– Decrease in infant mortality

• Benefit/cost ratio as large as 7 to 1– Decline in mortality $120 per 4-year-old– Extra half year of schooling increases earnings 2%

or $15,000– Even higher if consider increased schooling

reduces crime

Deming, 2009• Uses same method as Currie et al., but more

recent cohort• Effects on math and reading that fade

– 0.15 at age 5-6– 0.13 at age 7-10– 0.06 at age 11-14– Larger for African-Americans at first, but fade more

quickly• Larger positive effects on long-term outcomes

– 18.6 percentage points more likely to complete high school (African-Americans)

– “Index” of long-term outcomes, 0.23• Mechanism = less retention, special education

Hard to generalize these findings to today

• More working parents• More early childhood

programming options• Decrease in achievement gaps• Increase in Head Start quality

The National Head Start Impact Study

(NHSIS)

• Randomized controlled trial required by Congress in 1990s

• Nationally representative sample– 84 Head Start programs– Approximately 4,600 children– Begin attending in 2002

• Follow kids through end of 1st grade

First year findings: 4-year-olds

• 7 positive cognitive effect sizes – Effect sizes = 0.16 to 0.31

• More likely to visit the dentist– (73.2% vs. 56.9%)

• Parents more likely to read to children– (3x/wk vs. 2.8x/wk)

• No significant findings for more advanced reading skills, math, or behavior problems

First year findings: 3-year-olds

• Same positive impacts as 4-year-olds, but bigger, plus...– Math– Fewer problem behaviors– Parents spanked children less– Overall health

• Still no effects on more advanced reading skills

Lots of Caveats

• “Intent to treat” analysis– 17% of 3-year-olds and 14% of 4-year-

olds never got Head Start services • Crossovers

– 18% of 3-year-olds and 14% of 4-year-olds in control group attended Head Start

• 60% of control kids in other programs• Evidence of circumventing

randomization– Center directors enroll those with highest

needs

Ludwig & Phillips re-analysis

• “Treatment on the treated” (TOT) – Adjust for crossovers and no-shows– Larger effect sizes (0.14 to 0.50)

• Pool together 3- and 4-year-olds– Larger sample, more “power”– Statistically significant impacts for

nearly all main cognitive outcomes

First grade follow-up

• Significant effects from first year mostly disappear– For 4-year-olds, only impact on

receptive vocabulary remains (0.09)– For 3-year-olds, only impact on oral

comprehension remains (0.08)

Why we shouldn’t compare these results to pre-k study

results

• Pre-k studies use different research design that is not as strong as NHSIS

• Fewer kids in pre-k control groups attend other programs

• Different populations• Pre-k only looks at academic

outcomes• No long-term follow-up of pre-k

(therefore, no info about “fade out”)

So, what does this study really tell us?

• Impacts represent short-term effects of offer of one year of Head Start

• Measures impact of Head Start versus everything else available

• Initial impacts found are about the same size as those found for children attending Head Start in the 1960s-1980s

• For those cohorts, Head Start seems to have produced long-term benefits large enough to outweigh program costs, despite fade-out of initial test score impacts (nothing like that measured here)

• But test score impacts do seem to fade more quickly in new cohort—not sure why or whether we should worry about this

Does Head Start pass the benefit/cost test today?

• Ludwig & Phillips (2008) – Cost = $9,000 per child– Head Start might pass a benefit-cost

test if short-term effects on achievement test scores were as small as 0.1 to 0.2

Early Head Start research

• Randomized controlled trial• Follow through 5th grade• Positive impacts at age 3

– Cognitive and language development– Socio-emotional development– Home environment and parenting behavior– Parents’ progress toward self-sufficiency– Positive impacts specifically for fathers

• “Sleeper” effects• Differences across different service models

Early Head Start Home Visiting Evidence

What’s new in Head Start research?

• Focus on “quality”• Curriculum enhancements• Professional development• Early Head Start and home visiting• Head Start Family and Child

Experiences Survey (FACES)– New! “Baby FACES”

Meta-analysis (Shager et al., 2012)

• Study of 33 Head Start evaluations• Overall, positive effect of Head Start on

cognitive and achievement outcomes (effect size = 0.27)

• What happens in the control group matters a lot!– Effect sizes for studies that have an “active”

control group are 0.33 smaller– Type of skill, measures used, and where

published also matter

Conclusions• Some high quality Head Start research exists,

but you need to be an alert consumer• Head Start and Early Head Start have had

positive impacts on children and families across a variety of domains

• Head Start appears to pass the benefit/cost test

• We still have a lot to learn about how and why early childhood programs improve long-term outcomes

Remaining questions• Instead of asking whether Head Start is

effective at all, we must ask how effective Head Start is compared to the range of other options available.

• What happens to kids after they attend Head Start?– Do effects really “fade out,” or do other kids

“catch up”?• Which behavioral and socio-emotional

skills are important, and how can Head Start foster those?

• Other benefits? – Health, parenting, family self-sufficiency

What are your questions?