Ceppe march 2011 final (Susanna Loeb)

Teacher Trajectories and Teacher Effectiveness: the role of policy and leadership

Susanna LoebCEPPEMarch 11, 2011

A Collaborative Effort

• Don Boyd, Hamp Lankford & Jim Wyckoff – Teacher workforce

• Pam Grossman– Teaching

• Jason Grissom, Eileen Horng, Demetra Kalogrides– School Leadership

Problems and Contexts

Generally low student

outcomes

Systematic inequality in

student outcomes

Importance of teachers for

students’ opportunities

Great differences in

teacher effectiveness

Systematic differences in

teachers across communities

Little knowledge of how to improve

effectiveness

Knowledge and Abilities of Teachers in the Classroom

is a Function of…

Recruitment and Selection

Development and Supports

Assignment RetentionHow can

we improve each of these?

In the US, Differences and Similarities

Recruitment and Selection

Development and Supports

Assignment Retention

Similarly, little direct recruitment or selection until recently

Similarly, substantially differences in teachers across schools (also within)

Similarly, little differentiation of the teaching career

Preparation more similar to the 4+1 model for high schools – state policy

More hours for PD, with recent focus on learning communities

Much bigger problem. (e.g. in NYC about 18% of teachers leave their school each year).

Low agreement / knowledge on how to select

Low agreement / knowledge on support

hypothesis: improvement rests on…

WHO MAKES DECISIONS?

central government

local government

schools

teachers

families / communities

for example…

WHO CAN TEACH? many

options

Central government

selects

Central government sets requirements, local governments

select

Central government set requirements and schools

select

Schools select

Central government sets requirements,

teacher prep programs select, then schools

select

what is

best ?

In order to make good decisions, the decision

makers need…

TO MAKE GOOD DECISIONS

incentives

ability

information

So considering our decision makers…

WHO MAKE

S DECISIONS?

central government

local government

schoolsteachers

families / communities

Often little information to

make good decisions

Often better information (still imperfect) but little

ability and often unaligned incentives

Key is to improve incentives and abilities

locally so they make better decisions

But also to improve overall knowledge since the knowledge base in teaching is weak

May be missing different information and abilities

rest of today: how incentives, information and

abilities can improve..

Recruitment and Selection

Development and Supports

Assignment Retention

3 Examples Incentives

influenced by regulations

Information

Ability of local decision makers

1. Federal Regulations Affecting Recruitment and Selection Incentives

considering the past decade in NYC

THE STARTING PLACE: YEAR 2000, NY

Unequal Distributionteacher attributes by student test scores

(New York State, in 2000)

Little Enforcement of State Policy

Why did we see this distribution?

• Teacher Preferences• High achieving students• White students (at least for white teachers, majority)• Location - Close to home

• Little State Enforcement

• Little Direct Recruitment• Teacher education programs(little outreach, rarely selective)• Difficult-to-staff districts recruited certified teachers, little school-

level control, late hiring (July and August), often relied on waivers

• Leadership labor market subject to same forces, which in turn affect teachers

A BIT OF CHANGE

Innovation in Direct Recruitment (prior to certification)

• Teach for America - 1990

• The New Teacher Project – 1997

• UTeach

Change in Accountability

• State requirements no longer based solely on compliance (inputs)

• Standardized tests used for accountability instead of just to inform parents or for student opportunities (e.g NY regents)

• Started in the states but moved to Federal with NCLB – increased emphasis on subgroups (and teacher quality).

Highly Qualified Teacher Provision

• Intent - all students have a good teachers and equity in teacher quality

• State flexibility

• Weak research base for defining HQ – use professional consensus & State policy process

• defines HQT as: (1) fully State-certified; (2) holding a bachelor’s degree; (3) demonstrating competency in the core academic subject or subjects he or she teaches – multiple options

Highly Qualified Teacher• All teachers have the option of passing a

state exam.

• Middle and secondary teachers may complete an undergraduate or graduate degree in their field or advanced certification or credential.

• All veteran teachers also have the option of completing a High Objective Uniform State Standard of Evaluation (HOUSSE)

IMPLICATIONS FOR TEACHERSa dramatic change in the workforce

10%

15%

20%

25%

30%

35%

2000 2001 2002 2003 2004 2005

% o

f n

ew

te

ac

he

rs

Lowest quartile 2nd quartile 3rd quartile Highest quartile

Certification Exam Failure Rate of New Elementary

Teachers, 2000-’05 High-poverty schools

Low-poverty schools

10%

15%

20%

25%

30%

35%

2000 2001 2002 2003 2004 2005

% o

f n

ew

te

ac

he

rs

Lowest quartile 2nd quartile 3rd quartile Highest quartile

Certification Exam Failure Rate of New Elementary

Teachers, 2000-’05 High-poverty schools

Low-poverty schools

Certification Exam Failure Rate of New Elementary Teachers, 2000-’05

10%

15%

20%

25%

30%

35%

40%

2000 2001 2002 2003 2004 2005

Per

cen

t o

f te

ach

ers

Low est quartile 2nd quartile 3rd quartile Highest quartile

High-poverty schools

Low-poverty schools

Effect of changes in teacher characteristics on student math

scores

25% decrease in the predicted gap

Conclusion 1

• A small change in regulation can create a large change in incentives

• Incentives can inspire meaningful change in schools and for students

2. Improvements to a Weak Knowledge Base

linking teacher preparation to effectiveness to improve information

Current Knowledge Base

• Virtually no research linking preparation experiences to teachers’ classroom performance and, particularly to student outcomes

• Lack of information hinders simple decisions about certification policy at the state level as well as program design locally

Approach to Understanding Prep:

first explore

Program FeaturesTeachers’ Reported

ExperiencesStudent test Performance

a first try

program features spring/summer 2004

• 18 institutions: 26 univ. programs + 4 TF + 1 TFA

• Document collection, interviews with director and director of field experiences, surveys of math and ELA methods faculty

• Many many measures but here focus mainly on link to practice. Program data are not ideal for this.– whether or not the program required a capstone project – a composite measure of the extent that the program maintains

oversight over student teaching experiences• requires a minimum number of years of teaching experience for its

cooperating teachers (32%)• program picks the cooperating teacher (42%) • program supervisor observes their participants a minimum of five

times during student teaching (27%)– for comparison math and English content course requirements and

the percent of tenure-line.

reported experiences: survey spring 2005• All first year teachers in NYC

• Practice Variables– extent to which preparation included links to practice;

• opportunities to listen to an individual child read aloud for the purpose of assessing his/her reading achievement;

• …Plan a guided reading lesson, and • …Study or analyze student math work (each 5-point scale).

– opportunities to study curriculum used in New York City – whether or not the teacher had student teaching

experiences, not as the teacher-of-record in the classroom;

– the congruence between student teaching placement and their current job assignment in terms of subject matter or grade level;

• Additional measures of preparation as controls and comparisons– opportunities to learn about teaching math and ELA; learning;

handling student misbehavior; and teaching English language learners.

opportunities to learn math methods

• alpha = 0.97• In your teacher preparation program, prior to September

2004, how much opportunity did you have to do the following (5-point scale)? – learn typical difficulties students have with place value; – learn typical difficulties students have with fractions; – use representations (e.g., geometric representation, graphs, number

lines) to show explicitly why a procedure works; – prove that a solution is valid or that a method works for all similar

cases; – study, critique, or adapt math curriculum materials; – study or analyze student math work; – design math lessons; – learn how to facilitate math learning for students in small groups; – adapt math lessons for students with diverse needs and learning

styles; – And practice what you learned about teaching math in your teacher

preparation program in your field experience.

first year results: program features

* *

1st year results: reported experiences

Conclusion 2• Evidence the aspects of preparation can

matter• Well-supervised practice• Content course (effects in the second year)• Preparation for teaching English Learners

• Surprisingly little research on preparation (little systematic identification of best practices)

• Stems, at least in part, from little codification of practice

Similar work linking teaching practice to student learning

• Find systematic differences in the practices of more effective teachers

• Similarly still early on in developing the knowledge base.

• Useful because it can be used in– decision making in general– and in developing supports

Not today…

3. Understanding and Improving the Abilities of Local Decision Makers

variation in the abilities and behaviors of school principals

Motivation: Why Teachers Leave

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

Close

r Hom

e

Plac

e

SALA

RY

Pres

tige

Benefi

ts

Job

Secu

rity

Preg

nanc

y/Chi

ld

Health

Fam

ily/P

erso

nal

Scho

ol A

ction

Oppor

tuni

ty

Dissat

isfac

tion

College Recommended (N=156)Teaching Fellows (N=36)

Other (N=114)

when we look within dissatisfaction…

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Assin

gmen

t

Facilit

ies

Safe

ty

Adm

in

Behav

ior

Colle

ague

s

Teac

hing

Phi

loso

phy

Auton

omy

Test

ing

Distric

t Policy

Resp

ect

Own

Abilit

y

College Recommended (N=150)Teaching Fellows (N=35)

Other (N=112)

Motivation, more generally• Principals linked to teacher satisfaction and career

choices

• Principals central actors in most recent school reforms (accountability, school-based budgeting, charter schools)– Better local information– Improved incentives from accountability system– Concern about abilities

• Increased policy attention on attracting and preparing effective school leaders, but– Lack understanding of principal abilities to look for when hiring

or to target development – Lack of organized systems for recruiting and developing

leaders (in most places)

our approach: again, exploratory

What do Principals do?• How do these

behaviors vary across schools?

• Do principals in high performing behave differently

What abilities do principals need to do these tasks?• How do

these abilities vary across schools?

Given these findings,

explore in more detail

the behaviors and abilities

identified

How are schools affected by leadership labor market dynamics• Do principals, like

teachers, demonstrate preferences for working in some schools and not in others?

What do principals do? Developed list of 47+ tasks that principals might do

based on: Research literature Discussions with principals and Piloting and shadowing in local California schools

Collected observational time use data Observed each principal for one full day Recorded time use on 47 (later more) tasks every five minutes

Sample All high school principals in Miami-Dade County Public schools

(plus 6 elementary and 6 middle school principals) All schools serving 6th graders and above in Milwaukee Public

Schools All schools in San Francisco and Kansas City Today focus only on Miami-Dade County schools

How Do Principals and Teachers Perceive Principal

Effectiveness on These Tasks? Surveyed all principals, assistant principals and teachers Principals in M-DCPS (n = 314; 89% response) Assistant principals (n = 585; 85%) Teachers (n = 15,842; 83%) -- satisfaction

Asked principals how effective they felt at each of the tasks

Asked assistant principals how effective their principals were at each task

Identify groupings of self-assessed task effectiveness reflecting underlying skills

Link responses and observations to administrative data, other survey data, and interviews to assess the relationship between skills and school outcomes

5 Primary DimensionsInstruction

Management

Skills for promoting

and improving

the implementin

g of curricular

programs in classrooms

Using assessment results, providing

instructional feedback,

implementing PD

Internal Relations Skills for building strong

relationships within the

school

Handling staff conflicts,

counseling students and

teachers

Organization

Management

Skills employed to maintain a

highly functioning

organization

Maintaining facilities,

budgeting, hiring personnel

Admin-istration

Skills related to

compliance and

regulatory tasks

Maintaining records, fulfilling

special ed requirements,

managing attendance

External Relations Skills for

working with outside stake-

holders

Communicating with the

district office, fundraising, working with

the community

Split into program and day-to-day for observations

Across Districts: Most time on Administrative Tasks

Org

anizatio

n Mana...

Admin

istra

tion

Day-to

-Day

Inst

ruc.

..

Inst

ruct

ional P

rogra

m

Inte

rnal R

elatio

ns

Extern

al Rela

tions

-5%

0%

5%

10%

15%

20%

25%

30%

High School Principal Time-Use by District

SFUSD (n=17) MPS (n=30) M-DCPS (n=40)

Most Time Spent On:• Disciplining students• Supervising students• Observing classrooms• Internal relationships• Compliance requirements• Managing budgets

Least Time Spent On:• External relationships• Coaching teachers• Using data and

assessments• PD for teachers• PD for themselves• Teaching students

Findings: principal time-use

MPS, M-DCPS

The Importance of Organization Management

Time-Use

Organization Management

Student Performance

Gains

Instructional Program

Teacher Satisfaction at Current

School

Day-to-Day Instruction

Teacher Assessmen

ts of School

Parent Assessmen

ts of School

positive relationship negative relationship

Internal Relationships

Principals’ Organization

Management Efficacy

Student Performance Gains

ParentAssessments

TeacherSatisfaction

Other domains NOT related to outcomes.

The Importance of Organization Management

Efficacy

Corroboration Using Assistant Principal

Assessments• Principal self-assessments may not be the best

way to measure task effectiveness– Principals can’t be objective about themselves

• Administer the same inventory to their APs (multiple per school) and look for the same patterns

• Uncover 3 factors from AP responses: Instruction, Internal Relations, Operations

Assistant Principals’ Assessment of

Principals’ Organization

Management Efficacy

Gains in Student Achievement

ParentSatisfaction

TeacherSatisfaction

Note: Other domains of efficacy NOT related to outcomes.

Assistant Principal Assessment of Principal Task Efficacy and Outcomes

M-DCPS

Conclusion 3• Evidence of variation in the behaviors and abilities of

local decision makers

• Given current behaviors and abilities – improvements in Organization Management appear beneficial

• Current focus on instruction likely useful but poorly focused• Classroom visits good for gaining information• But, only useful if information is used for improvement – rare.

• Now looking at how to select leaders for organizational leadership skills as well as develop skills in principals

• Principals subject to many of the same market forces as teachers

In summary: US faces many of the same

ChallengesGenerally low

student outcomes

Systematic inequality in

student outcomes

Importance of teachers for

students’ opportunities

Great differences in teacher

effectiveness

Systematic differences in

teachers across communities

Little knowledge of how to improve

effectiveness

A range of factors affect and can improve the teaching workforce

Recruitment and Selection

Development and Supports

Assignment Retention

solutions are complex and changing

• Needs vary– For example, best professional development depends

on the skills of the teachers and context of the teaching

• Best decisions vary and change over time

• Useful to consider who makes decisions and their abilities, information and incentives

• Our aim is to understand these dimensions and how policies can influence them

Some findings to date• Simple regulations (such as higher test requirements)

can incentivize local recruitment and improve the teaching workforce.

• Systematic knowledge generation in teacher development and teaching practice is both useful and doable

• Local variation in needs and options lead to benefits of local decision making but local decision makers (e.g. principals) vary in their abilities – a similar policies and information are needed for the leadership

labor markets.

Teacher Trajectories and Teacher Effectiveness: the role of policy and leadership

Susanna LoebCEPPEMarch 11, 2011