Hopkins PSMG Oct 20051 An Incomplete Model for Trial Designs for Moving to Scale: The Whole Day Program C Hendricks Brown Sheppard Kellam Jeanne Poduska

Hopkins PSMG Oct 2005 1

An Incomplete Model for Trial Designs for Moving to Scale:

The Whole Day ProgramC Hendricks BrownSheppard KellamJeanne PoduskaAmy Windham

John ReidCarla Ford

Natalie Keegan

Supported by R01 DA15409-02 PI: KellamRo1 MH40859-17 PI: Brown

Baltimore City Public School System


Premise

The Whole Day Program is where the school would want to be in 5 years, systemwide.

What specific steps should be examined along the way?


Primary Question

Effective?Who benefits, for how long?Who doesn’t benefit, who is harmed?

Efficacy Trial12 Schools24 Classrooms600 Students3 Time Periods


Differential impact on spring of first grade reading achievement by

baseline reading (fall of first grade): control against two interventions

Control

Classroom / Parent Interventions

Brown: Statistical Power


Presuming Effectiveness

SustainabililtyDo WD teachers continue to sustain the same

effects on children that they have in following years?

12 WD Teachers vs Themselves & 12 Wait-Listed Controls

Scalability Can program be successfully implemented

outside the original 12 schools throughout the school system?


Scale to What?

• From 2 To all 4 of the School Areas

• From 12 To 20 To 135 Schools

• From 24 To 40 To 400 First-Grade Classrooms


An Evidence-Based Decision Tree

Efficacious especially w/ High-Risk Youth?

Sustainable?

Feasible to Scale?

Y

Y

Go to Scale!

Y

N

N

N


The Real Decision Tree for BCPSS

Something Different


Person, Place, and Time

• Child• Classroom / Teacher (1st Grade)• School• Area (Region of Baltimore)

• Time – Scale(s): 10 Secs, Min, Quarter, Year• Child : Track from 1st through 3rd Grade• Teacher: Track over 3 years (cohorts)


Randomization

• 20 Schools in 2 Areas randomly assigned to One of 12 schools where trial begins

One of 8 schools that are “wait-listed”

• Children randomly assigned to classes within school

• Teachers/Classes randomly assigned to WD or wait-list


Baltimore Whole Day TrialAssignment of Classrooms/Teachers in First YearHalf the Teachers are Wait-Listed for WD to 3rd

Year

Replicate Same Design

w/ SC and and WD

BOTH in 12 Different Schools

Brown: Statistical Power


Effectiveness Evaluation

Compare 3 years of Outcomes for Children in the 12 WD classes to those in 12 Standard classes -- 1st Cohort

Examine Level of Program Implementation in the 12 WD and 12 Standard Classrooms in 1st Cohort


Classroom Observations by Independent Observers

3 times per year WD and SC classroomsAM-Reading InstructionPM-Other subjects

Teacher PracticesTimed ObservationsChecklist Global Ratings of Quality

Student BehaviorOn-Task/Off-taskRatings of Disruptive, Aggressive, and Shy Behavior


Classroom Observation Data Collection FormTeacher Practices and Student Behavior

Actual Form


Domains of Reading Instruction

• Word Work – phonemic awareness, alphabetic instruction, letter recognition, work work

• Reading Comprehension – reading comprehension, previewing a book, reading books

• Language – oral language, vocabulary, reading own writing, writing, grammar

• Non-reading Instruction – interruptions, teacher out of classroom

• Giving instructionally related reading directions• Feedback in response to behavior• Feedback in response to a reading relevant

comment or event


Classroom Observations BaselineData Overview

12 schools24 teachers (12 Whole Day, 12 Standard setting)

Each teacher observed 3 times per year, at each time point:

– 1 reading instruction (am observation)– 1 non-reading instruction (pm observation)

Length of observations at each time point:

Teacher: mean = 164 minutes Each student: mean = 9 min/student


Given Teacher Instructional Practices, Classroom Behavior(Standard Practice Classrooms during Reading Instruction at

Baseline)Well behaved v Fairly well behaved: X2=36.83 p<10-7

Word Work Reading Comprehension

Language

Well behaved 216

67.3%

184

61.3%

84

39.6%

Fairly well behaved

99

30.8%

109

36.3%

115

54.2%

Poorly behaved

1

.3%

7

2.3%

13

6.1%

Chaotic 5

1.6%

0 0


Student behavior: Reading v Other subjects(Standard Practice Classrooms at Baseline)

Never off-task v Ever off-task: X2=29.80 p<10-8

AM-Reading PM-Other subjects

Never off task 1138

95%

679

88.3%

Off task-1 interval 13

1.1%

13

1.7%

Off task-2 intervals 5

.4%

10

1.3%

Off task-3 intervals 42

3.5%

67

8.7%


Effectiveness Results for Cohort 1 (1 of 2)

• At Baseline Level of Off-Task Behavior is Well Balanced across Intervention Condition

• Reduction in Off-task BehaviorControl 21% WD 9%

• 50% Reduction in Aggressive Behavior for Males in Less Structured Settings


Graphical Comparison for Off-Task Behavior

Q-Q Plot of Males Off-Task Behaviorin Afternoons at End of First Grade

Global Rating for Control Classrooms

Glo

ba

l Ra

ting

fo

r W

D C

lass

roo

ms

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0


Comparison of Teachers

Q-Q Plot of Teacher Global Overall RatingEnd of First Grade

Global Rating for Control Classrooms

Glo

ba

l Ra

ting

fo

r W

D C

lass

roo

ms

1 2 3 4 5

3.0

3.5

4.0

4.5

5.0


Unsolved Multilevel Modeling Issues

Child Gender, Baseline Off-TaskClassroom Intervention, Classroom

Baseline AggrSchool

Time-Varying Classroom Context

Binary Outcomes


Sustainability Evaluation (Funded)

Multilevel Support

Compare Teacher Implementation (and Child Outcomes) for 12 WD Teachers in 1st through 3rdYear for First Grade Classes


Sustainability Hypotheses and N’s & Years

• Teacher practices over time will be related to the quality of WD integration at the level of the School Building Team.

12 20 135

2• Teacher practices over time will be related to

the quality of WD integration at the level of the School Area over time.

2 4


Scalability Design

Change in fidelity (Teacher practices) and impact (Child Outcomes) in succeeding years after initial training

12 24 135


Combined RCT and Implementation Trial

Trial After a Trial Design

Full Implementation Control Waiting

Effectiveness Trial

Yr1 Yr2

Yr3

Moving to Scale

Full Implementation Control

Varying Levels of Implementation Due to Sustainability and Scalability


How to Go From 24 to 400 Classes?

A Dynamic Wait-Listed type of design

Randomly Assign WHEN schools and classes obtain training, coaching, and full implementationContinue to assess intervention fidelityPaired-down measure of Teacher Practices

Random Assign Children to Classes?


ClassesSchools

Waiting

Active School

Trained

Wait Listed

Sustained

Trained Sustained


Plausible Time Scale # Classes (Schools)

Time Period/Cohort

New WD Sustainability Comparison Wait-Listed

Effectiveness 12 (12) 0 12 (Same 12) 376 (123)

Sustain 0 12 (12) 12 (Same 12) 376 (123)

Scalability 12 (12) 12 (Same 12) 20 (8) 356 (115)

Full Scalability

Yr 1

90 (30) 44 (20) 40 (30) 230 (85)

Yr 2 90 (25) 130 (50) 40 (25) 140 (60)

Yr 3 90 (30) 220 (70) 40 (30) 50 (35)

Yr 4 90 (35) 310 (100) 0 (0) 0 (0)

Documents

Hopkins PSMG Oct 20051 An Incomplete Model for Trial Designs for Moving to Scale: The Whole Day Program C Hendricks Brown Sheppard Kellam Jeanne Poduska