Effects of the My Teaching Partner Intervention in Secondary School Classrooms Joseph P. Allen...

Effects of the My Teaching Partner Intervention in Secondary School Classrooms

Joseph P. AllenRobert C. Pianta

University of Virginia

Co-Collaborators:Amori MikamiAnne Gregory

Project Team:Chris HafenSharon DealJudith WassermanRachel BorenJanetta Lun

Context

Number of Secondary School Students in U.S.: 24 million

Number of Secondary School Classesbeing taught each week 6 million

% of 9th graders who won’t finish High school by the end of 12th grade 25%

Number of programs in ‘What Works’ Clearinghouse with demonstrated efficacy improving teaching quality enough to improve student achievement in these classrooms

0

Key Questions

• Can we identify teacher-student interaction qualities that predict student engagement and achievement?

• Can we change these qualities?

• Will changes lead to sustainable student achievement gains?

• What are the mechanisms of change?

Key Questions

• Can we identify teacher-student interaction qualities that predict student engagement and achievement?

• Can we change these qualities?

• Will changes lead to sustainable student achievement gains?

• What are the mechanisms of change?

Classroom Learning Assessment & Scoring System- Secondary (CLASS-S)

Emotional Support

Positive Climate

Teacher Sensitivity

Regard for Adolescent Perspectives

Negative Climate

Instructional Support

Instructional Learning Format

Content Understanding

Analysis & Problem Solving

Quality of Feedback

Classroom Organization

Behavior Management

Productivity

Student Outcomes

Student Engagement

Evaluation Design

• 43 teachers within 8 schools (640 students)(The control condition in an RCT).

• 1 focal classroom selected per teacher

• Predicting Future Achievement after Covarying Baseline Achievement Test Scores

• Teacher Demographics:– 64% female

– 83% White, 8% African-American; 6% Mixed-Ethnicity; 3% Other

– 54 middle school, 34 high school

– 35% BA degree; 65% at least a year of course work beyond BA

– Average 8 years of teaching experience

Classroom Characteristics

• School type: 39% High school;

61% Middle School

• Subject: 52% Language/Social Studies;

48% Math/Science

• Average class size: 23 students

• Gender: 47% girls

53% boys

• Ethnicity: 23% African American

2% Asian4% Hispanic70% European-American

Observational Assessment of Classroom Environment

• Videotaped observations of a classroom

– spread throughout course of year

• Two 20-minute segments per class session/tape

– Each tape rated by 2 raters

• Coded Using CLASS-S System

• High inter-rater reliabilities; ICC’s range from– .73 - .82 for overarching domains

– .50 -.78 for specific dimensions (all but one dimension > .64)

Student Academic Success

• Score on State “Standards of Learning” End of Year Subject Test

• The measure by which schools/students are judged for accreditation/graduation.

• Extensive seven-year validation/standardization process.

Analytic Approach

• Multi-level modelling • All models covary:

– Student factors:• Grade level• Gender• Family poverty status

– Classroom factors:• Classroom size

– Teacher Factors• Teacher experience• Teacher education• Teacher gender and race

• Moderating effects of covariates were also examined.

Predicting Student AchievementAchievement

Emotional Positive Climate .22**

Support: Negative Climate -.04

Teacher Sensitivity .16*

Regard for Adol. Perspectives .21**

Classroom Behavior Management .06

Organization: Productivity .15

Instructional Content Understanding .12

Support: Analysis & Prob. Solving .18*

Instructional Learning Formats .22**

Quality of Feedback .09

Composite of Significant Dimensions Above .32***

Key Questions

✔Can we identify teacher-student interaction qualities that predict student engagement and achievement?

• Can we change these qualities?

• Will changes lead to sustainable student achievement gains?

• Why?

MyTeachingPartner Overview

• Consultant and teacher work together using the CLASS-S in cultivating: – Observation– Reflection– Development of knowledge and expertise

Classroom Observation Teaching Practice

Knowledge Expertise Support

The Steps of the Consultancy

MTPS Website

www.mtpsecondary.net

Detailed Video Examples

www.mtpsecondary.net

Key Questions

✔Can we identify teacher-student interaction qualities that predict student engagement and achievement?

• Can we change these qualities?

• Will changes lead to sustainable student achievement gains?

• What are the mechanisms of change?

Evaluation Design

88 classrooms

45 Tx. 43 Control(Classrooms Randomized within school)

2237 Students

Assessed Across 2 Years

Evaluation Design

Treatment group:Year 1:– Introductory Workshop (late summer)– Ongoing consultancy– ~ 2 days total in-service timeYear 2:– Booster Workshop (late summer) only + Web site

access• Control group:

Usual in-service practice.

Intervention Effect on Change in Classroom Qualities

Intervention Target January MarchApril/May

Overall Teacher-Student Interaction Composite

ns

Student Engagement ns

Intervention Effect on Change in Classroom Qualities

Intervention Target January MarchApril/May

Overall Teacher-Student Interaction Composite

ns ns

Student Engagement ns ns

Intervention Effect on Change in Classroom Qualities

Intervention Target January MarchApril/May

Overall Teacher-Student Interaction Composite

ns ns .19*

Student Engagement ns ns .34*

Year 1 Change in OverallTeacher-Student Interactions

Pre-Test Post-Test3.553.603.653.703.753.803.853.903.954.00

Control GroupMyTeachingPartner

Standardized Effects: Baseline = .45***Intervention = .19*

MTPS participation predicts higher quality teacher-student

interactions

Key Questions

✔Can we identify teacher-student interaction qualities that predict student engagement and achievement?

✔Can we change these qualities?

• Will changes lead to sustainable student achievement gains?

• What are the mechanisms of change?

Year 1Intervention Effects on Achievement

• No relation of intervention to either baseline or exit achievement test scores in Year 1 (all p’s > .35).

• Why?– No evidence we changed the classroom until the

very end of the year when most teaching was past.

Year 2Change in Achievement

Pre-Test Post-Test460.00

465.00

470.00

475.00

480.00

485.00

490.00

495.00

Control GroupMyTeachingPartner

Standardized Effects: Pre-test = .54***Intervention = .22*

MTPS is predicting increases in End of Course Achievement

Tests

Year 2Intervention Effects on Achievement

• Real-world effect size = .22 SD increment in Achievement Test scores

• Average ‘Bump’ of students in MTP from 50th to 59th percentile in achievement

• If effect applies equally at all parts of achievement spectrum (as appears to be the case): a .22 SD boost would reduce failure rates from:

14% without the intervention to

10% with it

Reducing the number of failing students each year by 29%

*** This occurs in the year AFTER the intervention year (i.e., sustainability), across diverse subject matter/content areas.

Key Questions

✔Can we identify teacher-student interaction qualities that predict student engagement and achievement?

✔Can we change these qualities?

✔Will changes lead to sustainable student achievement gains?

• What are the mechanisms of change?

A Preliminary Mediational Analysis

“My Teaching Partner”

Intervention

Intervention

Observed Change in Student

Achievement

*

Environmental Outcome

A Preliminary Mediational Analysis

“My Teaching Partner”

Intervention

InterventionIntervention

Target

Observed Change in Student

Achievement

Environmental Outcome

??

Mediational Analyses

• Assessed via Multi-level Structural Equation Modelling, followed up via parametric bootstrapping analysis (Preacher et al., 2010)

• Focus on target of intervention (Teacher-student interactions assessed via CLASS-S)

• Using Centered/Standardized data for ease of interpretation.

“My Teaching Partner”

Intervention

InterventionIntervention

TargetStudent

Outcome

.37** .16**

Change in Student Achievement

MTP-S Effect as Mediated via Observed Interactions

Initial Model (Simple Direct Effects)

.12*

Observed Teacher-Student

Interactions

“My Teaching Partner”

Intervention

InterventionIntervention

Target

.06* *

StudentOutcome

.37** .16**

Change in Student Achievement

MTP-S Effect as Mediated via Observed Interactions

.06 ns

Initial Model (Simple Direct Effects)

Final Model (Including Mediated Effect)

.12*

Observed Teacher-Student

Interactions

Limitations

• Design only supports causal interpretations for outcomes, not for mediating processes with analyses thus far.

• Some Attrition Took Place (though it was unrelated to the intervention in every possible way we could test).

• Modest statistical significance with small sample

Conclusions

• We CAN identify elements of the classroom environment that predict student achievement.

• We CAN change these environmental factors.

• If we do, student achievement will change as well, eventually.

• Changes can be sustained over time and in new classrooms, post-intervention.

• We can identify potential mechanisms of change linked to the intervention.

• Which has implications for cost effectiveness…

*BOE = Back of Envelope

Potential Significance – Costs vs. Benefits(BOE* Calculation)

Resources per classroom Estimated Cost

20 Teacher hours No additional cost to system (in lieu of Regular In-service)

1 Teacher-consultant per 20 teachers Maximum of $3,500 per teacher including benefits (Potentially offset by ongoing teacher supervision personnel costs)

Video equipment $200 per teacher

TOTALS: Maximum of $3,700/23 children = $160/child (i.e., < 2% of annual per pupil expenditures) **

Benefits

Average ‘Bump’ in achievement of ALL students from 50th to 59th percentile

Reduction of 1 course failure per classroom

(**Results may apply to multiple classrooms taught by a teacher)

MyTeachingPartner Secondary

Replication is ongoing with the support of IES

Further information available at:www.myteachingpartner.net