Effects of the My Teaching Partner Intervention in Secondary School Classrooms
Joseph P. AllenRobert C. Pianta
University of Virginia
Co-Collaborators:Amori MikamiAnne Gregory
Project Team:Chris HafenSharon DealJudith WassermanRachel BorenJanetta Lun
Context
Number of Secondary School Students in U.S.: 24 million
Number of Secondary School Classesbeing taught each week 6 million
% of 9th graders who won’t finish High school by the end of 12th grade 25%
Number of programs in ‘What Works’ Clearinghouse with demonstrated efficacy improving teaching quality enough to improve student achievement in these classrooms
0
Key Questions
• Can we identify teacher-student interaction qualities that predict student engagement and achievement?
• Can we change these qualities?
• Will changes lead to sustainable student achievement gains?
• What are the mechanisms of change?
Key Questions
• Can we identify teacher-student interaction qualities that predict student engagement and achievement?
• Can we change these qualities?
• Will changes lead to sustainable student achievement gains?
• What are the mechanisms of change?
Classroom Learning Assessment & Scoring System- Secondary (CLASS-S)
Emotional Support
Positive Climate
Teacher Sensitivity
Regard for Adolescent Perspectives
Negative Climate
Instructional Support
Instructional Learning Format
Content Understanding
Analysis & Problem Solving
Quality of Feedback
Classroom Organization
Behavior Management
Productivity
Student Outcomes
Student Engagement
Evaluation Design
• 43 teachers within 8 schools (640 students)(The control condition in an RCT).
• 1 focal classroom selected per teacher
• Predicting Future Achievement after Covarying Baseline Achievement Test Scores
• Teacher Demographics:– 64% female
– 83% White, 8% African-American; 6% Mixed-Ethnicity; 3% Other
– 54 middle school, 34 high school
– 35% BA degree; 65% at least a year of course work beyond BA
– Average 8 years of teaching experience
Classroom Characteristics
• School type: 39% High school;
61% Middle School
• Subject: 52% Language/Social Studies;
48% Math/Science
• Average class size: 23 students
• Gender: 47% girls
53% boys
• Ethnicity: 23% African American
2% Asian4% Hispanic70% European-American
Observational Assessment of Classroom Environment
• Videotaped observations of a classroom
– spread throughout course of year
• Two 20-minute segments per class session/tape
– Each tape rated by 2 raters
• Coded Using CLASS-S System
• High inter-rater reliabilities; ICC’s range from– .73 - .82 for overarching domains
– .50 -.78 for specific dimensions (all but one dimension > .64)
Student Academic Success
• Score on State “Standards of Learning” End of Year Subject Test
• The measure by which schools/students are judged for accreditation/graduation.
• Extensive seven-year validation/standardization process.
Analytic Approach
• Multi-level modelling • All models covary:
– Student factors:• Grade level• Gender• Family poverty status
– Classroom factors:• Classroom size
– Teacher Factors• Teacher experience• Teacher education• Teacher gender and race
• Moderating effects of covariates were also examined.
Predicting Student AchievementAchievement
Emotional Positive Climate .22**
Support: Negative Climate -.04
Teacher Sensitivity .16*
Regard for Adol. Perspectives .21**
Classroom Behavior Management .06
Organization: Productivity .15
Instructional Content Understanding .12
Support: Analysis & Prob. Solving .18*
Instructional Learning Formats .22**
Quality of Feedback .09
Composite of Significant Dimensions Above .32***
Key Questions
✔Can we identify teacher-student interaction qualities that predict student engagement and achievement?
• Can we change these qualities?
• Will changes lead to sustainable student achievement gains?
• Why?
MyTeachingPartner Overview
• Consultant and teacher work together using the CLASS-S in cultivating: – Observation– Reflection– Development of knowledge and expertise
Classroom Observation Teaching Practice
Knowledge Expertise Support
The Steps of the Consultancy
MTPS Website
www.mtpsecondary.net
Detailed Video Examples
www.mtpsecondary.net
Key Questions
✔Can we identify teacher-student interaction qualities that predict student engagement and achievement?
• Can we change these qualities?
• Will changes lead to sustainable student achievement gains?
• What are the mechanisms of change?
Evaluation Design
88 classrooms
45 Tx. 43 Control(Classrooms Randomized within school)
2237 Students
Assessed Across 2 Years
Evaluation Design
Treatment group:Year 1:– Introductory Workshop (late summer)– Ongoing consultancy– ~ 2 days total in-service timeYear 2:– Booster Workshop (late summer) only + Web site
access• Control group:
Usual in-service practice.
Intervention Effect on Change in Classroom Qualities
Intervention Target January MarchApril/May
Overall Teacher-Student Interaction Composite
ns
Student Engagement ns
Intervention Effect on Change in Classroom Qualities
Intervention Target January MarchApril/May
Overall Teacher-Student Interaction Composite
ns ns
Student Engagement ns ns
Intervention Effect on Change in Classroom Qualities
Intervention Target January MarchApril/May
Overall Teacher-Student Interaction Composite
ns ns .19*
Student Engagement ns ns .34*
Year 1 Change in OverallTeacher-Student Interactions
Pre-Test Post-Test3.553.603.653.703.753.803.853.903.954.00
Control GroupMyTeachingPartner
Standardized Effects: Baseline = .45***Intervention = .19*
MTPS participation predicts higher quality teacher-student
interactions
Key Questions
✔Can we identify teacher-student interaction qualities that predict student engagement and achievement?
✔Can we change these qualities?
• Will changes lead to sustainable student achievement gains?
• What are the mechanisms of change?
Year 1Intervention Effects on Achievement
• No relation of intervention to either baseline or exit achievement test scores in Year 1 (all p’s > .35).
• Why?– No evidence we changed the classroom until the
very end of the year when most teaching was past.
Year 2Change in Achievement
Pre-Test Post-Test460.00
465.00
470.00
475.00
480.00
485.00
490.00
495.00
Control GroupMyTeachingPartner
Standardized Effects: Pre-test = .54***Intervention = .22*
MTPS is predicting increases in End of Course Achievement
Tests
Year 2Intervention Effects on Achievement
• Real-world effect size = .22 SD increment in Achievement Test scores
• Average ‘Bump’ of students in MTP from 50th to 59th percentile in achievement
• If effect applies equally at all parts of achievement spectrum (as appears to be the case): a .22 SD boost would reduce failure rates from:
14% without the intervention to
10% with it
Reducing the number of failing students each year by 29%
*** This occurs in the year AFTER the intervention year (i.e., sustainability), across diverse subject matter/content areas.
Key Questions
✔Can we identify teacher-student interaction qualities that predict student engagement and achievement?
✔Can we change these qualities?
✔Will changes lead to sustainable student achievement gains?
• What are the mechanisms of change?
A Preliminary Mediational Analysis
“My Teaching Partner”
Intervention
Intervention
Observed Change in Student
Achievement
*
Environmental Outcome
A Preliminary Mediational Analysis
“My Teaching Partner”
Intervention
InterventionIntervention
Target
Observed Change in Student
Achievement
Environmental Outcome
??
Mediational Analyses
• Assessed via Multi-level Structural Equation Modelling, followed up via parametric bootstrapping analysis (Preacher et al., 2010)
• Focus on target of intervention (Teacher-student interactions assessed via CLASS-S)
• Using Centered/Standardized data for ease of interpretation.
“My Teaching Partner”
Intervention
InterventionIntervention
TargetStudent
Outcome
.37** .16**
Change in Student Achievement
MTP-S Effect as Mediated via Observed Interactions
Initial Model (Simple Direct Effects)
.12*
Observed Teacher-Student
Interactions
“My Teaching Partner”
Intervention
InterventionIntervention
Target
.06* *
StudentOutcome
.37** .16**
Change in Student Achievement
MTP-S Effect as Mediated via Observed Interactions
.06 ns
Initial Model (Simple Direct Effects)
Final Model (Including Mediated Effect)
.12*
Observed Teacher-Student
Interactions
Limitations
• Design only supports causal interpretations for outcomes, not for mediating processes with analyses thus far.
• Some Attrition Took Place (though it was unrelated to the intervention in every possible way we could test).
• Modest statistical significance with small sample
Conclusions
• We CAN identify elements of the classroom environment that predict student achievement.
• We CAN change these environmental factors.
• If we do, student achievement will change as well, eventually.
• Changes can be sustained over time and in new classrooms, post-intervention.
• We can identify potential mechanisms of change linked to the intervention.
• Which has implications for cost effectiveness…
*BOE = Back of Envelope
Potential Significance – Costs vs. Benefits(BOE* Calculation)
Resources per classroom Estimated Cost
20 Teacher hours No additional cost to system (in lieu of Regular In-service)
1 Teacher-consultant per 20 teachers Maximum of $3,500 per teacher including benefits (Potentially offset by ongoing teacher supervision personnel costs)
Video equipment $200 per teacher
TOTALS: Maximum of $3,700/23 children = $160/child (i.e., < 2% of annual per pupil expenditures) **
Benefits
Average ‘Bump’ in achievement of ALL students from 50th to 59th percentile
Reduction of 1 course failure per classroom
(**Results may apply to multiple classrooms taught by a teacher)
MyTeachingPartner Secondary
Replication is ongoing with the support of IES
Further information available at:www.myteachingpartner.net