1

Concentration of Low-Performing Students (4th grade Math, 2002)

0

20

40

60

80

100

0 10 20 30 40 50 60 70 80

percent of schools

per

cen

t o

f al

l L

evel

1 s

tud

ents

2

Teacher Sorting in NYS Elementary Schools (Math 2002)

Percent of Students at Level 1 Percent of teachers Highest Quartile Lowest Quartile

0-1 years prior 21.3 12.3teaching experience

Failed general knowledge 31.0 10.1or LAST exam

BA from a least 24.0 11.4competitive college

Not certified in 24.6 4.3any assignment

3

85 Percent of New York State Teachers Take First Job within 40 Miles of Home

0

20

40

60

80

0-15 15-40 40-100 100 or more

Miles from First Job to High School

Per

cen

t o

f T

each

ers

NYS

NYC

4

Policy Challenge

How do public schools:

Attract potentially excellent teachers to teaching, especially in traditionally difficult-to-staff schools?

Provide the skills and experiences that help potentially excellent teachers develop into excellent teachers in those schools?

Retain strong teachers, especially in traditionally difficult-to-staff schools?

5

Data Collection Program analysis

State documents, program documents, accreditation reports, interviews, surveys, course syllabi; 18 institutions that prepare most traditional route teachers for NYC schools

Alternate route programs that prepare about 25 percent of NYC teachers

Surveys Graduating program participants (2004), new NYC teachers

(2005), second year and former teachers (2006) Administrative data

All NYC teachers 1990-2006; rich measures of teacher qualifications, including certification exams and areas, teacher retention.

Student achievement 2000-2006; value-added scores in math and ELA, grades 4-8 linked to teachers.

Data on schools and students

6

Examining Teacher Preparation Paths

What is the relative effectiveness of alternatively certified teachers compared to those prepared in traditional college preparation programs?

Selection: alternative certification teachers appear to have stronger academic credentials

Preparation: traditional preparation teachers have an undergraduate teacher education program; alternative certification teachers a 7 week summer program

7

Data and NYC Context Achievement gain data from NYC for 1999-2000 through

2003-2004 – student gains in grades 4-8 linked to teacher.

DEFINING PATHWAY - The route teachers follow into their first teaching job. College Recommended

Individual Evaluation (transcript review)

NYC Teaching Fellows

Teach for America

Temporary License (uncertified)

Other (i.e. reciprocity, other Transitional B programs, provisional)

Caveat: All teachers in New York must ultimately complete the same set of requirements to receive certification. A lot of overlap and a lot of variation within pathways. For example, this past year about 1/4th of traditional did student teaching as teacher of record.

8

Entering NYC Teachers by Pathway: Teaching Fellows Replace Temp License

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000

1997 1998 1999 2000 2001 2002 2003 2004 2005

Nu

mb

er o

f T

each

ers

College Recommended Individual Evaluation Teaching Fellow

Teach for America Temporary License Other

9

Average Certification Exam Scores First Taking: (2004; Passing= 220,SD=~30)

246

247

267

276

242

200 210 220 230 240 250 260 270 280 290 300

CollegeRecommended

Individual Evaluation

Teaching Fellow

Teach For America

Temp license

10

Characteristics of Students by Pathway of First-Year Teachers

0

20

40

60

80

100

Free Lunch Non White English atHome

Per

cen

t

CR

TL

TF

TFA

11

MethodologySchool, Grade, and Year Fixed Effects

Aisgjt=0+1Aisgj(t-1)+2Kit+2Cj+Ej1+1Pj+πg+πt+πs+εisgjt Specification Checks

Interactions with Experience Gain as the outcome Gains with student fixed effect Only those with three or fewer years of experience School by year by grade fixed effects two stages Analysis of the importance of school and class by omitting school

fixed effects and classroom controls Alternative Definitions of Pathway Subgroups of Students Incorporating Turnover

12

Results of Student, Peer and Experience Variables, Base Math Model with grade, year & school fixed effects (n=960,970)Student Measures Class Average Measures Teacher Experience

Lag score 0.659*** Lag score 0.154*** 2 0.048*** 12 0.065***

  (0.00)   (0.01) (0.01) (0.01)Lag sq 0.001 Std Dev 0.006 3 0.067*** 13 0.062***

  (0.00)   (0.01) (0.01) (0.01)Female -0.009***     4 0.073*** 14 0.080***

  (0.00)     (0.01) (0.01)Other Eth -0.088*** Other Eth -0.177 5 0.077*** 15 0.056***

  (0.01)   (0.12) (0.01) (0.01)Asian 0.114*** Hispanic -0.182*** 6 0.080*** 16 0.054***

  (0.00)   (0.03) (0.01) (0.01)Hispanic -0.083*** Black -0.230*** 7 0.068*** 17 0.053***

  (0.00)   (0.03) (0.01) (0.01)Black -0.109*** Asian -0.061* 8 0.075*** 18 0.059***

  (0.00)   (0.03) (0.01) (0.01)English Lang. -0.047*** English 0.042** 9 0.070*** 19 0.040***

  (0.00)   (0.02) (0.01) (0.01)Free lunch -0.046*** Free lunch -0.057*** 10 0.074*** 20 0.049***

  (0.00)   (0.01) (0.01) (0.01)Lag Abs -0.004*** Lag Abs -0.010*** 11 0.071*** >20 0.055***

  (0.00)   (0.00) (0.01) (0.01)

Lag Susp. -0.052*** Lag Susp. -0.078**    (0.00)   (0.03)        

13

Total Pathway value-added findings for math with school-fixed effects

(relative to College Recommended)

*= significant at 10% **= significant at 5%***= significant at 1%

 Pathway Math ELA

Independent Evaluation -0.012** -0.005NYC Teaching Fellows -0.023*** -0.030***TFA 0.007 -0.031***Temporary License -0.021*** -.012***Other -0.021*** -0.021***

14

Specifications checks, Base Model for Math

  Gain <= 3

Years

Gains with Stud

FE

Schl/Year/

Grade FE

Two Stage

2 Stage Exp<=3

IE -0.012** -0.028** -0.009 -0.011** -0.015** -0.032**

  (0.005) (0.011) (0.006) (0.005) (0.006) (0.013)

Fellows -0.023*** -0.025*** -0.014 -0.030*** -0.025*** -0.031***

  (0.008) (0.009) (0.009) (0.007) (0.009) (0.010)

TFA 0.007 -0.001 0.015 -0.005 -0.004 -0.011

  (0.015) (0.016) (0.017) (0.015) (0.016) (0.018)

Temp -0.021*** -0.026*** -0.010** -0.017*** -0.023*** -0.034***

  (0.004) (0.006) (0.004) (0.004) (0.005) (0.007)

Other -0.021*** -0.025*** -0.018** -0.015** -0.026*** -0.039***

  (0.008) (0.010) (0.008) (0.007) (0.009) (0.012)

15

Effects on Student Achievement by Pathway by Experience, Math, Grades 4-5 (Coll. Rec. 1 yr =0;)

-0.08

-0.04

0

0.04

0.08

0.12

1 2 3

Teacher Experience

Ach

ieve

men

t st

d.

dev

.

Coll. Rec.

Fellows

TFA

Temp. Lic.

*Caution – small sample size for third-year TFA makes interpretation of results tricky

16

Effects on Student Achievement by Pathway by Experience, Math, Grades 6-8 (Coll. Rec. 1 yr =0;)

*Caution – small sample size for third-year TFA makes interpretation of results tricky

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

1 2 3

Teacher Experience

Ach

ievem

en

t S

td D

ev. Coll Rec

Fellows

TFA

Temp Lic

17

How Important Are These Effects? Statistical significance

Comparison to other effects 1st-year Teaching Fellows vs. CR teachers—0.04

difference in student gains in grade 4-5 math Similar in magnitude to 80% of the gain attributed to 1st

year of teaching experience (0.048) 3rd year TF vs. CR teachers—0.056 difference in

student gains in 6-8th grade math (0.091 minus 0.035)

greater in magnitude than 1st year of experience gain

18

How Important Are These Effects (con’t)?

Conversion to scale score For example, 7th grade math exam in 2002 had mean score=670, and std dev = 48.

The 0.056 advantage of 3rd year Teaching Fellow versus 3rd year CR from prior example equals about 2.7 scale points (48 x .056) a 2.7 scale point increase moves past 2.5% of students at

mean

19

Given the effectiveness of teachers improves with their first few years of experience, any such differences in attrition could affect differences in value-added across pathways.

Is there differential attrition by pathway?

20

Grades 4-8 Teacher Attrition Following the 1st Four Years of Teaching, 1999-2003 Cohorts

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

CollegeRec.

Indiv.Eval.

Fellows TFA Temp

1st yr

2nd yr

3rd yr

4th yr

Adjusted for grade School and Year

21

Simulated Student Achievement Gains by Pathway Accounting for Differential

Attrition, Math 4-5

Year Coeff Sim Coeff Sim Coeff Sim1st year 0.000 0.000 -0.040 -0.040 -0.034 -0.034

3rd year 0.080 0.065 0.053 0.037 0.106 0.052

5th year 0.091 0.073 0.064 0.041 0.117 0.048

CR TF TFA

22

Simulated Student Achievement Gains by Pathway Accounting for Differential

Attrition, Math 6-8

Year Coeff Sim Coeff Sim Coeff Sim1st year 0.000 0.000 -0.012 -0.012 0.046 0.046

3rd year 0.035 0.032 0.091 0.067 0.113 0.085

5th year 0.059 0.047 0.115 0.071 0.137 0.082

CR TF TFA

23

Summary Reduced entry requirements are attracting a

substantial number of teachers with strong qualifications.

Currently these teachers appear to be more effective in math than ELA and more effective in middle school grades than in upper elementary.

Alternate route teachers are not, on the whole, doing better than college recommended teachers, and are doing worse in some cases. However, few of the differences across pathways are statistically significant.

Need to look at the effects of characteristics of pathway on selection and preparation to figure out how to maximize effectiveness…

24

Not satisfying yet…

Methodological question about the social significance of effectsEstimating the size of teacher effects

(separating from schools)Measurement error

Variation in effects and experiences within pathways - trying to get at this with larger study

25

Childhood Program Math Value-Added(Grades 4-8, programs with at least 50 observations, with

school fixed effects and controls for teacher attributes)

-0.10 -0.05 0.00 0.05 0.10

Program Value Added (Proportion of Student Achievement Std Deviation)

26

Childhood Program ELA Value-Added (Grades 4-8, programs with at least 50 observations, with

school fixed effects and controls for teacher attributes)

-0.10 -0.05 0.00 0.05 0.10

Program Value Added (Proportion of Student Achievement Std Deviation)

27

Relationship of Program Features to Student Math Achievement

Constructed variables related to preparation, based on 1st year teacher survey responses

Variables/factors linked to 5 aspects of preparation Program structure Preparation for teaching math Preparation in learning/development, including preparation in

classroom management Preparation to teach linguistically and culturally diverse

students Field experiences

28

Content-Specific Preparation

General opportunities to learn about the teaching of math (Genmath): includes 12 items related to opportunities to: Learn typical difficulties students have with place

value Study, critique, or adapt math curriculum materials Study or analyze student math work

29

Field Experiences

Field-job congruence: My experiences in schools were similar to my current job in

terms of grade level My experiences in schools were similar to my current job in

terms of subject area

Field-program coherence: My program lacked a sense of coherence among courses and

between courses and field experiences What I learned in methods courses reflected what I observed in

my field experiences or in my own classroom

30

Learning About Learning

Learning factor: Includes 8 items related to learning about child

development and learners, including special education, classroom management.

Relative emphasis on: Develop strategies for handling student misbehavior Develop specific strategies for teaching English

language learners (those with limited English proficiency)

31

Opportunities Grounded in Practice

Relative emphasis on opportunities to Listen to an individual child read aloud for the

purpose of assessing his/her reading achievement Plan a guided reading lesson Study or analyze student math work

Relative opportunities to: Review New York City reading curriculum Review New York City mathematics curriculum

32

Value-Added of Preparation Program Features(Math, 2005 cohort)

Controlling for student prior scores a rich set of student characteristics a rich set of aggregate student characteristics

at the classroom level teacher attributes pathway

33

Effect of Program Attributes on 4th – 8th math student achievement

Variables All paths CR only

Opportunities grounded in practice 0.07** 0.061~

Opportunities to study NYC curriculum

0.041**

Field experience/job congruence 0.041** 0.045*

General Math 0.064*

Relative Emphasis on teaching ELL 0.03* 0.06**

** p<.01, * p<.05, ~ p<.10

34

With important caveats…

Analysis depends on significant variation in experiences. Doesn’t mean other factors aren’t important; Opportunities to learn to teach math, for example, may be very important but not enough variation in this factor to be significant.

Differences in quality of experience, rather than quantity, may matter most. Yet quality is hardest to measure in large scale studies.

We are early in the process of this analysis.

35

Challenge to teacher educators

The importance of creating opportunities for students to engage in practice-based activities (e.g., analyzing student work in math, conducting individual reading assessments, studying NYC curriculum).