Whole Day First Grade Program: Effectiveness, Sustaining, &

Building a Base for Going to Scale

Jeanne Poduska, ScD

Center for Integrating Education and Prevention Research in Schools, American Institutes for Research

Presented at: New Directions for Mental Health and Drug Abuse Effectiveness

and Dissemination Trial Research and MethodologyOctober 28, 2005

Contributors and Support

Sheppard G. Kellam, M.D. 1

Jeanne Poduska, Sc.D. 1

C. Hendricks Brown, Ph.D. 2

Carla Ford, Ph.D. 1

Amy Windham, Ph.D. 1

Natalie L. Keegan 1

John Reid, Ph.D. 3

Bonnie Copeland, Ph.D. 4

Linda Chinnia, 4

National Institute on Drug Abuse,Division of Epidemiology, Services and Prevention Research,

Prevention Research Branch 1R01 DA015409 and 1R01 DA019984 (and support from NICHD)

1 American Institutes for Research, Center for Integrating Education and Prevention Research in Schools2 University of South Florida, Prevention Science Methodology Group

3 Oregon Social Learning Center4 Baltimore City Public School System

Efficacy

Effectiveness

Sustainability

Going to Scale

Sustaining Systemwide

Efficacy

Effectiveness

Sustainability

Going-to-Scale

Sustaining

Systemwide

Phases of Education and Prevention Trials

Developmental Epidemiology: A Strategy for Prevention Research

Community Epidemiology

Life Course Development

Preventive Interventions

Research roots of WD• First Generation of Baltimore trials tested separately

Mastery Learning (ML) aimed at reading, and Good Behavior Game (GBG) aimed at classroom management of student behavior in First Grade.

– Each had short-term impact on its target; GBG had long term impact on very high risk children.

• Second Generation tested combination of instructional improvement and GBG and showed impact on reading and behavior into middle school, still room for improvement.

– A family/classroom program also seemed promising.

• Oregon Prevention Research Center tested family/ classroom partnership (LIFT program) and found impact among higher risk children.

Add Outcome Slide

How do you Establish Community & Institutional Partnerships?

1. Analyze social/political structure

2. Learn the vision of problems and priorities

3. Identify mutual self-interest within and across leadership

4. Fit your interest under the vision of leadership

5. Request ad-hoc oversight committee of leaders

6. Work through trust

This process takes about a year

Functions of a Community & Institutional Board

• Establish vision and priorities

• Examine/critique/approve/support proposed programs

• Communicate constituents’ concerns, values, priorities, and acceptable language to program leaders

• Communicate program needs to constituent leaders

• Continually assess absent constituencies

• Work toward institutionalizing programs

Baltimore City Public School System Partners

Patricia Welch, Ph.D., Chair, BCPSS Board of School Commissioners

Bonnie Copeland, Ph.D., Chief Executive Officer

Linda Chinnia, Chief Academic Officer

Ben Feldman, Ph.D., Research, Evaluation & Accountability Officer

Sue Cutter, Area 1 Interim Academic Officer

James Smith, Area 2 Academic Officer

Gwen Cleage, Curriculum and Instruction Officer

La Verne Sykes, Director, Parent and Community Involvement

Deborah Wortham, Ph.D., Director of Professional Development

Delores Alexander, Coordinator of Reading

Marietta English, President, Baltimore Teachers Union

James Gittings, Public School Supervisors and Administrators Association

20 Schools, Principals, and Teachers Areas 1 & 2

Three Foci of Baltimore Whole Day First Grade Program (WD)

• Improve reading instruction in all academic subjects thru the day, based on prior Baltimore trials.

• Improve classroom behavior management, based on prior Baltimore trials.

• Improve parent/classroom partnership re reading and behavior, based on prior Baltimore and Eugene, Oregon trials.

Lessons Learned From First Two Generations of Trials in Baltimore

Without a system to mentor, model, and monitor teacher practices over time, practices are not sustained.

A multi-level institutional structure with mentoring and monitoring is required to implement interventions with fidelity.

Teachers need support from principals; principals from area leaders; area leaders from chief academic and executive officers.

Goals of WD Trial

• Can we find synergy from the combination?

• Can the practices be sustained over time?

• Can practices be scaled up by school system with same effect?

Baltimore Whole Day First Grade Program Analytic Model

ClassroomBehavior

Management Classrooms

DecreasedAggressive, Disruptive

Classrooms

Teachers’ EffectiveAcademic

Instruction

PoorReading Skills

ImprovedReading Skills

Decreased Later Conduct & Anti-

social PersonalityDisorders

Effective Family-Classroom

Partnerships

•••

Other mediating or moderating variables:

• Family structure and poverty• Deviant peersWhole DayWhole Day First Grade

Education and Prevention Program

Decreased Later DepressiveDisorders

Depressive Symptoms

DecreasedDepressive Symptoms

PoorAchievement

IncreasedAchievement

Decreased Later Substance Abuse

Individual Aggression

DecreasedIndividual Aggression

School Success & Decreased

Drop-Out

• School building• Community economic health, resources, drugs, and violence

Multi-Level Structure to Sustain Practices and Go to Scale BALTIMORE CITY PUBLIC SCHOOL SYSTEM

Board of School Commissioners

Chief Executive Officer

Chief Academic Officer

ProfessionalDevelopment Officer

Curriculum &Instruction Officer

Area AcademicOfficers

Lead Coaches

Principals Coaches

Teachers

Research and Evaluation Office

Core Team

BALTIMORE CITY PUBLIC SCHOOL SYSTEM

Board of School Commissioner

Chief Executive Officer

Chief Academic Officer

ProfessionalDevelopment Officer

Curriculum &Instruction Officer

Area AcademicOfficers

Lead Coaches

Principals Coaches

Teachers

Research and Evaluation Office

Schools Committee

BALTIMORE CITY PUBLIC SCHOOL SYSTEM

Board of School Commissioner

Chief Executive Officer

Chief Academic Officer

ProfessionalDevelopment Officer

Curriculum &Instruction Officer

Area AcademicOfficers

Lead Coaches

Principals Coaches

Teachers

Research and Evaluation Office

Within-school Random Assignment* of Teachers/Classrooms and Children in the

Whole Day First Grade Program

IN EACH SCHOOL:

WD

SC (a) SC (b)

Random

Random

Children randomly assigned to classrooms

Teachers randomly assigned to classrooms

*Brown & Liao. AJCP, 1999.

Effectiveness Design of the Whole Day First Grade Program:

Follow 2 Cohorts of 1st Graders to 3rd Grade2003-04 2004-05 2005-06

12TeachersWD

Cohort 1 Grade 1

Cohort 1 Grade 2

Cohort 2 Grade 1

Cohort 1 Grade 3

Cohort 2 Grade 2

12 TeachersSC

Cohort 1 Grade 1 Cohort 1 Grade 2

Cohort 2 Grade 1

Cohort 1 Grade 3

Cohort 2 Grade 2

Remaining Teachers

Sustainability Design of the Whole Day First Grade Program:

Follow Teachers over Cohorts of 1st Graders2003-04 2004-05 2005-06

12TeachersWD

Cohort 1 Grade 1

Cohort 2 Grade 1

Cohort 3 Grade 1

12 TeachersSC

Cohort 1 Grade 1

Cohort 2 Grade 1

Remaining Teachers

Scaling-Up Design of the Whole Day First Grade Program:

3 Cohorts of 1st Graders2003-04 2004-05 2005-06

12TeachersWD Cohort 1 Grade 1

Cohort 2 Grade 1Cohort 3 Grade 1

12 TeachersSCTrained in WD

Cohort 1 Grade 1Cohort 2 Grade 1

Cohort 3 Grade 1

Remaining TeachersTrained in WD

Cohort 3 Grade 1

Design of the Whole Day First Grade Program

2003-04 2004-05 2005-06

12TeachersWD

Effectiveness

Cohort 1 Grade 1

Effectiveness

Sustainability

Cohort 1 Grade 2

Cohort 2 Grade 1

Effectiveness Sustainability

Cohort 1 Grade 3

Cohort 2 Grade 2

Cohort 3 Grade 1

12 TeachersSC

Effectiveness

Cohort 1 Grade 1

Effectiveness

Sustainability

Cohort 1 Grade 2

Cohort 2 Grade 1

Effectiveness Scaling Up (to WD)

Cohort 1 Grade 3

Cohort 2 Grade 2

Cohort 3 Grade 1

Remaining Teachers

Scaling Up (to WD)

Cohort 3 Grade 1

Primary Measurement Objectives in the WD Trial

• Implementation

• Proximal effects of intervention on student behavior, achievement, and psychological well-being

• Longer term (distal) effects

• Teacher practices

• Hypothesized or known mediating and moderating variables

• Measures of the multi-level structure

Fidelity of Whole Day First Grade Program Implementation Checklist Excerpt

CLASSROOM BEHAVIOR MANAGEMENT INDICATORS QUALITY RATING

20. Instructs students on class rules by:

Announcing the game and reviewing rules.

Setting and adhering to time limits for the game.

Providing appropriate independent assignments during the game.

Y

Y

Y

Y

N

N

N

N

1 2 3 4 5 6

Classroom Behavior Management Global Rating:Global Rating is not necessarily the sum of individual quality ratings.

1 2 3 4 5 6

Measures of Effectiveness Social Adaptational Status

OUTCOME MODE MEASURE

ACADEMIC ACHIEVT Student interview Woodcock Johnson

    Houston Word Reading Scales

  Teacher Interview Teacher Observation of Classroom Adaptation (TOCA)

  Record review School records, Standardized scores

BEHAVIOR    

Time on-task/off-task Classroom Obs. Teacher Observation/Student Engagement (TOSE) - Foorman

Aggressive behavior Classroom Obs. Teacher Interview

TOSETOCA

Disruptive behavior Classroom Obs. Teacher Interview

TOSETOCA

Concentration prob’s Classroom Obs. Teacher Interview

TOSETOCA

Social Isolation Classroom Obs. Teacher Interview

TOSETOCA

Measures of Effectiveness Psychological Well-being

OUTCOME MODE MEASURE

Depression Student interview Direct Clinical ObservationBaltimore How I Feel

Anxiety Student interview Direct Clinical ObservationBaltimore How I Feel

Flat affect Student interview Direct Clinical Observation

Hyperkinesis Student interview Direct Clinical Observation

Bizarre behavior Student interview Direct Clinical Observation

Teacher Practices

Observations by Independent Observers

(3x per year)

• Teacher Practices

• Timed Observation of Instruction (Foorman)

• Global ratings

Measures of Multi-Level System to Support Sustaining

Practices and Scaling-Up

• Defined expectations for each level of the multi-level structure along four organizational variables: power/structure, tasks, resources, human factor

• Attendance logs and observers’ ratings of occurrence and quality for expectations

• 1 Core Team; 2 AAOs; 12 school teams

Chief Academic Officer

Core Team

Area Academic Officers &Lead Coaches

School Coaches / Principals

Monitoring and Mentoring Feedback Loop

MentoringMonitoring

Data from Monitoring

Effective Mentoring

Teacher Practices

CLASSROOM BEHAVIOR MANAGEMENT INDICATORS QUALITY RATING

22. Plays the Good Behavior Game by:

Dividing students into balanced teams.

Reviewing class rules and announcing the game.

Setting and adhering to time limits for the game.

Placing checks on the board when individual students break rules.

Providing rewards to winning teams.

Y

Y

Y

Y

Y

Y

N

N

N

N

N

N

1 2 3 4 5 6

Classroom Behavior Management Global Rating:Global Rating is not necessarily the sum of individual quality ratings.

1 2 3 4 5 6

Fidelity of Whole Day First Grade Program Implementation Checklist Excerpt

Patterns of Quality of GBG Implementation in

12 Intervention Schools: 2004-05

High4 - 6

▲▲▲▲▲▲

▲▲▲▲▲▲

▲▲▲▲▲▲

▲▲▲▲▲▲

Low1 - 3

11 18 25 36

Week of Implementation

Leve

l of

Impl

emen

tatio

n

Patterns of Quality of GBG Implementation in

12 Intervention Schools: 2004-05

High4 - 6 ● ● ● ● ● ● ● ● ●

Low1 - 3

● ● ● ● ● ● ●

11 18 25 36

Week of Implementation

Leve

l of

Impl

emen

tatio

n

Patterns of Quality of GBG Implementation in

12 Intervention Schools: 2004-05

High4 - 6

■ ■ ■

Low1 - 3 ■ ■ ■ ■ ■

11 18 25 36

Week of Implementation

Leve

l of

Impl

emen

tatio

n

Patterns of Quality of GBG Implementation in

12 Intervention Schools: 2004-05

High4 - 6

▲▲▲▲▲▲

▲▲▲▲▲▲

● ●■ ■

▲▲▲▲▲▲

● ● ●■

▲▲▲▲▲▲

● ● ● ●

Low1 - 3

● ● ● ●■ ■

● ● ●■

■ ■

11 18 25 36Week of Implementation

Leve

l of

Impl

emen

tatio

n

Professional Development Model as Moving to Scaling Up

• AIR (Carla Ford) transitioning from classroom/school based focus to multi-level professional development

• Carla monitoring coaches; coaches monitoring teachers

• 3 Area Lead Coaches being trained fall 2005; train teachers in spring 2006

• Developing monitoring system

Summary—Prevention and Service Researchers

Researcher/Community Institution Partnerships

Defining intervention and practices with precision

Defining and measuring multi-level structure for sustaining and scaling-up practices

Continual monitoring of practices deliberately linked to professional development

Dual design requires equal emphasis on measurement of effectiveness and going to scale

Summary—Statisticians and Methodologists

• Modeling mediation of teacher practices

• Relationship of observational data on teacher practices and checklist of teacher practices: thick to thin data

• Modeling multi-level structure data

• Help us prepare for missing data in follow-up!