+

Experimental Evidence on the Early Effectiveness of Intervention in ChildhoodOrla Doyle (UCD School of Economics & UCD Geary Institute)

Conference on Irish Economic Policy1st February 2013

+Importance of Early Childhood Investment

Targeted early intervention programmes effective way of reducing socio-economic inequalities in children’s skills

Well-designed interventions can generate a return to society ranging from $1.80 to $17.07 for each dollar spent

YET primarily US-based evidence only on the effectiveness & cost-effectiveness of such interventions

Argument for early intervention in more generous welfare systems is not rooted in strong empirical evidence

+Irish Context

Prevention & Early Intervention Programme (The Atlantic Philanthropies) & Government , DCYA) initiated in 2004

Aim: to support the development, implementation and evaluation of strategies for disadvantaged children

All programmes evaluated using experimental or quasi-experimental design

Little tradition of social experimental evaluation in Ireland

Comparing the effectiveness of multiple programmes will enhanced resource allocation and policy design

+ Preparing for Life Programme

Community-led initiative: operated by Northside Partnership in highly disadvantaged area of Dublin, Ireland

Bottom-up approach: community initiative involving 28 community groups, service providers, & local representatives

Evidence of need: Children scored below the norm on cognitive & non-cognitive skills on school entry (Doyle & McNamara, 2011)

Aim: Improve levels of school readiness by assisting parents in developing skills to prepare their children for school

Evaluation Design: RCT & embedded quasi-experimental design

+Design of Preparing for Life

+PFL Evaluation

Impact Evaluation Aim: To determine if the programme/service is effective

PFL data collection: Pre-intervention (baseline), 6mths, 12mths, 18mths, 24mths, 3yrs, 4yrs, school entry

Implementation Evaluation Aim: Delve into the blackbox of programme effectiveness & evaluate fidelity

to the model

PFL data collection:1. Implementation data2. Focus groups with participants3. Semi-structured interviews with programme staff

+ Recruitment

Eligibility Criteria: Cohort of pregnant women residing in PFL catchment area between Jan

2008-August 2010 (32 months) Includes nulliparous and multiparous women

Recruitment: Maternity hospital at first booking visit (b/w 12-26 weeks) Within the local community

Population-based recruitment rate, based on all live births during the recruitment phase, was 52%

+Randomisation Procedure

Individual-level randomisation Unconditional probability randomisation strategy

Design procedure that preserves the integrity of randomization ie. no opportunity for recruiter to intentionally influence assignment

Randomisation process After informed consent obtained, participant pressed key on tablet laptop to

allocate her assignment condition

Evaluation & Implementation team automatically received an email with the assigned ID number and treatment condition

Thus no opportunity to compromise the randomisation process

+ Baseline Analysis

Determine the effectiveness of the randomisation procedure

Baseline assessment conducted post randomisation, pre intervention

Randomisation worked!

PFL Baseline Assessment Proportion of Measures Not Significantly Different at Baseline

Topic PFL Low – PFL High PFL – Comparison group

Family Socio-demographics 33/33 (0%) 27/33 (82%)

Maternal Well-being 24/24 (0%) 18/24 (75%)

Maternal Health & Pregnancy 35/35 (0%) 26/35 (74%)

Parenting & Childcare 10/13 (74%) 6/13 (46%)

Social Support 17/18 (94%) 9/9 (0%)

Total NOT Statistically Different 119/123 (97%) 86/114 (75%)

+ Baseline Differences in Selected Socio-Demographics

Low Treatment – High Treatment

InstrumentN

(nLOW/ nHIGH)

MLOW

(SD)MHIGH

(SD)p Effect Size

(d)

Mother’s Age 205(101/104)

25.30(5.99)

25.46(5.85)

ns .03

First-time Mothers 205(101/104)

0.50(0.50)

0.54(0.50)

ns .09

Mother Married 205(101/104)

0.18(0.38)

0.14(0.35)

ns .09

Mothers with Junior Certificate Qualification or Lower

205(101/104)

0.40(0.49)

0.34(0.47)

ns .12

Mothers with Primary Degree 205(101/104)

0.03(0.17)

0.03(0.17)

ns .01

Mothers Unemployed 205(101/104)

0.41(0.49)

0.43(0.50)

ns .05

Fathers Unemployed 198(97/101)

0.31(0.46)

0.43(0.50)

ns .24

Residing in Social Housing 204(101/103)

0.55(0.50)

0.55(0.50)

ns .00

In Possession of a Medical Card 205(101/104)

0.66(0.47)

0.60(0.49)

ns .14

+Methodology

Estimate treatment effects at birth, 6 months, 12 months

Permutation based hypothesis testing (Heckman et al. 2010) Classical hypothesis tests unreliable when the sample size is small and the

data are not normally distributed Permutation test are distribution free, thus suitable in small samples Based on the assumption of exchangeability between treatment

conditions under the null hypothesis

Stepdown procedure (Romano & Wolf, 2005) Ignoring the multiplicity of tests may lead to the rejection of “too many”

null hypotheses Test multiple hypotheses simultaneously by controlling overall error rates

for vectors of hypotheses using the family-wise error rate (FWER) as a criterion

Less conservative & more powerful than other methods as takes account of statistical dependencies between tests

+Analysis of Maternity Records

Maternity records N(nHIGH/ nLOW)

MHIGH

(SD)MLOW

(SD)Individual Test

p

Infant Outcomes Apgar 5 180

(95/85)9.79(0.48)

9.69(0.63)

ns

Birthweight (grams) 181(95/86)

3281(664)

3294(614)

ns

Low BW (<2500gr) 181(95/86)

0.10(0.30)

0.05(0.23)

ns

High BW (>4000gr) 181(95/86)

0.09(0.29)

0.12(0.33)

ns

Labour Outcomes

Weeks at gestation 174(92/82)

39.60(2.51)

39.56(2.01)

ns

Prematurity (<37 wks) 174(92/82)

0.07(0.26)

0.06(0.24)

ns

Labour onset (Spontaneous=0, Induced=1)

178(93/85)

0.31(0.46)

0.43(0.49)

p=0.056

Caesarean section 178(93/85)

0.15(0.35)

0.25(0.44)

P=0.036

Data Collection Summer 2012 accessed maternity records for PFL participants who attended

Rotunda Hospital (85%) & NMH (15%) 283 parents gave consent to access records - available records for 272

+PFL Maternity Records – Summary

No programme impact on birth weight or gestational age, but programme altered the birth experiences of the participating families

Significantly fewer mothers in the high treatment group had a c-section compared to those in the low treatment group

Result remains when the incidence of a previous c-section and pariouness are controlled for

Programme particularly effective in preventing high treatment mothers from having a repeat c-section

Result possibly linked to Tip Sheets which specifically focus on preparing the mother for the birth and being aware of the birthing process and possible labour outcomes

+ Analysis of 6 Month Results

PFL 6 Month Results Proportion of Measures Significantly Different at 6 Months

PFL Low – PFL High Individual Tests

Multiple Hypothesis Tests

Child Development 0% (0/13) 0%

Child Health 10% (3/30) 0%Parenting 23% (5/22) 20%Home Environment & Safety 36% (8/22) 50%

Maternal Health 5% (1/20) 25%

Social Support 38% (5/13) 0%

Childcare & Service Use 7% (1/14) 0%

Household Factors & SES 0% (0/26) 0%

Total Statistically Different 14% (23/160) 12% (3/25)

Test for treatment effects across 160 measures & 25 Step-down categories

Sample: PFL High/Low treatment groups: 84/90

+

ASQ Scores & Difficult Temperament

N(nHIGH/ nLOW)

MHIGH

(SD)MLOW

(SD)Individual

Test p1

Step Down Testp2

Effect Sized

ASQ Gross Motor Score 173(83/90)

40.78(11.93)

38.50(12.99)

ns ns 0.18

ASQ Communication Score 173(83/90)

53.07(7.84)

51.78(8.49)

ns ns 0.16

*Difficult Temperament 173(83/90)

11.70(5.71)

12.21(5.50)

ns ns 0.09

ASQ Personal Social Score 172(82/90)

46.52(12.09)

45.94(13.57)

ns ns 0.05

*ASQ Social-Emotional 173(83/90)

14.76(10.68)

15.17(13.75)

ns ns 0.03

ASQ Fine Motor Score 173(83/90)

50.84(9.46)

51.39(10.17)

ns ns 0.06

ASQ Problem Solving 173(83/90)

51.87(9.39)

52.56(9.92)

ns ns 0.07

Notes: 1 one-tailed (right-sided) p value from an individual permutation test with 1000 replications. 2 one-tailed (right-sided) p value from a Step Down permutation test with 1000 replications. * indicates the variable was reverse coded for the testing procedure. ‘ns’ indicates the variable is not statistically significant.

EXAMPLE OF RESULTS: CHILD DEVELOPMENT @ 6 MONTHS

+Positive Treatment Effects @ 6 Months

Child Development: none

Child Health: Appropriateness & frequency of eating; Immunizations

Parenting: Parent-child dysfunctional interactions; Baby comparison score; Parental hostile-reactive behaviour; Interaction with baby

Home Environment: HOME sub-scales – variety, childcare, toys & books, physical environment, learning materials; Electrical socket covers, Framingham safety survey

Mother’s Health: Mother less hospitalisation for complications after birth

Social support: Frequently sees grandparent; Discuss programme with partner & others; Knows children same age as baby; Meet friends frequently

Childcare & Service use: Voted in last election

Household & SES: none

+Analysis of 12 Month Results Test for treatment effects across 140 measures & 23 Step-down

categories

Sample: 12 Months: PFL High/Low treatment groups: 83/82

+Positive Treatment Effects @ 12 Months

Child Development: ASQ Fine Motor skills; BITSEA Social Competence cut-off

Child Health: Chest Infections; Immunizations; Appropriateness eating

Parenting: none

Home Environment: none

Mother’s Health: Drank alcohol

Social support: Meet friends frequently; Voted in last elections

Childcare & Service use: ~

Household & SES: none

Results consistent with the home visiting literature

+Attrition & Dosage

Attrition/disengagement Attrition may bias evaluation results if non-random

29% of high treatment & 29% of low treatment dropped out/disengaged between randomisation & 12 months

Few individual characteristics predict attrition – IQ more important

Dosage/engagement Variation in dosage, may lead of variation in effect

Collected data on number/duration of home visits

On average, high treatment group received 21 home visits of ~1 hr in duration between pregnancy & 12 months 58% of prescribed visits delivered based on bi-weekly visits

Higher maternal IQ associated with higher engagement

+ Conclusions to date

Many of the results are in hypothesized direction

Few significant effects, yet consistent with literature Significant findings on birthing experiences, parenting, home environment, child

health, social support Corresponded directly to information on the PFL Tip Sheets No significant impact on many key domains such as child development, birth weight,

breastfeeding, maternal well-being etc.

Attrition & disengagement between 6 & 12 months very low

Engagement below prescribed dosage

Participate satisfaction is relatively high (focus groups)

+ Update on PFL

Oldest PFL child is 4 years and old & youngest is 18 months

24, 36 & 48 month surveys are currently in the field

18 month results available very soon

Over 1,600 interviews conducted as part of impact evaluation

Conducted five school readiness surveys with junior infant cohorts in 2008, 2009, 2010, 2011, 2012

Evaluation will continue until all children start school

Programme website: www.preparingforlife.com

Evaluation website: http://geary.ucd.ie/preparingforlife/