““Value added” measures of Value added” measures of teacher quality: use and teacher quality: use and

policy validity policy validity

Sean P. Corcoran Sean P. Corcoran New York UniversityNew York University

NYU Abu Dhabi ConferenceNYU Abu Dhabi ConferenceJanuary 22, 2009January 22, 2009

Overview Overview

An introduction to the use of “value An introduction to the use of “value added” measures (VAM) of teacher added” measures (VAM) of teacher effectiveness – in both research and effectiveness – in both research and practice. practice.

A discussion of the policy validity of VAM A discussion of the policy validity of VAM – motivated by current work on “teacher – motivated by current work on “teacher effects” on multiple assessments of effects” on multiple assessments of similar skills. With:similar skills. With: Jennifer L. Jennings (Columbia U)Jennifer L. Jennings (Columbia U) Andrew A. Beveridge (Queens College)Andrew A. Beveridge (Queens College)

What are “value added” What are “value added” measures?measures?

Essentially, an Essentially, an indirect estimateindirect estimate of a of a teacher’s contribution to learning, teacher’s contribution to learning, measured using gains in students’ measured using gains in students’ standardized test score resultsstandardized test score results

What makes them “indirect?” What makes them “indirect?” Uses a statistical model to account for certain Uses a statistical model to account for certain

student characteristics (key: student characteristics (key: past past achievement), attributing remaining test achievement), attributing remaining test score gains to the teacherscore gains to the teacher

Clearly an improvement over test score Clearly an improvement over test score levelslevels

What are “value added” What are “value added” measures?measures?

Generally, “teacher effects” cannot be Generally, “teacher effects” cannot be separated from “classroom effects”separated from “classroom effects” E.g. two classrooms of similarly situated E.g. two classrooms of similarly situated

students where one has a particularly students where one has a particularly disruptive studentdisruptive student

May be able to improve VAM with multiple May be able to improve VAM with multiple years of results for teachersyears of results for teachers

This approach raises a range of This approach raises a range of additional issues and questions, some of additional issues and questions, some of which I will address in a momentwhich I will address in a moment

Growth in VAMGrowth in VAM

VAM of teacher effectiveness were VAM of teacher effectiveness were initially mostly of academic interestinitially mostly of academic interest Rivkin et al. (2005): effect size Rivkin et al. (2005): effect size

of .10/.11 SD for reading/mathof .10/.11 SD for reading/math Nye et al. (2004): 25-75Nye et al. (2004): 25-75thth percentile percentile

shift in teacher quality increased shift in teacher quality increased reading/math by .35/.48 SDreading/math by .35/.48 SD

Growth in VAMGrowth in VAM

Value added assessment of teachers Value added assessment of teachers is becoming widespread practice in is becoming widespread practice in the U.S.the U.S. Houston, Dallas, Denver, Minneapolis, , Dallas, Denver, Minneapolis,

CharlotteCharlotte EVASS New York City – for now a New York City – for now a

“development tool” only“development tool” only The The Teacher Data Tool Kit

Why the sudden interest?Why the sudden interest?

1.1. A logical extension of school A logical extension of school accountabilityaccountability

• Movement to collect, publicly report Movement to collect, publicly report student achievement measures at the student achievement measures at the school levelschool level

• In some cases, rewards and sanctions In some cases, rewards and sanctions (e.g. NCLB)(e.g. NCLB)

• Common sense appeal (both Obama Common sense appeal (both Obama and McCain supported “pay for and McCain supported “pay for performance” for teachers)performance” for teachers)

Why the sudden interest?Why the sudden interest?

2.2. Data availabilityData availability• Large longitudinal databases of Large longitudinal databases of

student performance enabled these student performance enabled these calculationscalculations

• Concurrent advancements in Concurrent advancements in methodologymethodology

Why the sudden interest?Why the sudden interest?

3.3. Improving our assessment and Improving our assessment and measurement of teacher qualitymeasurement of teacher quality

• Easily observed characteristics of Easily observed characteristics of teachers are often poor predictors of teachers are often poor predictors of classroom achievement (Hanushek classroom achievement (Hanushek and Rivkin 2006)and Rivkin 2006)

• Especially true of qualifications for Especially true of qualifications for which teachers are remunerated (e.g. which teachers are remunerated (e.g. education, certification, experience)education, certification, experience)

Issues with VAM (to name a Issues with VAM (to name a few…)few…)

1.1. Focus on a narrow measure of educational Focus on a narrow measure of educational outcomes: does “the test” adequately reflect outcomes: does “the test” adequately reflect our expectations of the educational system?our expectations of the educational system?

E.g. skill content, short-term vs. long-term E.g. skill content, short-term vs. long-term benefitsbenefits

2.2. Validity: assuming “the test” reflects Validity: assuming “the test” reflects outcomes we care about, is the instrument a outcomes we care about, is the instrument a valid one?valid one?

Teaching to the test and test inflation (Koretz Teaching to the test and test inflation (Koretz 2007) – even “good” tests lose validity over time2007) – even “good” tests lose validity over time

Issues with VAM (to name a Issues with VAM (to name a few…)few…)

3.3. Modeling for causal inference: how Modeling for causal inference: how can we be confident that our VAM can we be confident that our VAM are providing “good” estimates of are providing “good” estimates of the teachers true (i.e. causal) the teachers true (i.e. causal) contribution to student learning?contribution to student learning?

Students are not randomly assigned to Students are not randomly assigned to teachersteachers

Dynamic trackingDynamic tracking ““Teacher effects” may be context Teacher effects” may be context

dependentdependent

Issues with VAM (to name a Issues with VAM (to name a few…)few…)

4.4. PrecisionPrecision Estimates of teacher effects are just Estimates of teacher effects are just

that: estimatesthat: estimates Each student’s test score gain is a Each student’s test score gain is a

small—and noisy—indicator of teacher small—and noisy—indicator of teacher effectivenesseffectiveness

Are our estimates precise enough to Are our estimates precise enough to base personnel decisions on them?base personnel decisions on them?

Issues with VAM (to name a Issues with VAM (to name a few…)few…)

5.5. OtherOther Perverse incentives (gaming / Perverse incentives (gaming /

cheating)cheating) Subject dependencySubject dependency PersistencePersistence Scaling issues – e.g. ceiling effectsScaling issues – e.g. ceiling effects Missing data – e.g. absent or Missing data – e.g. absent or

exempted studentsexempted students

The “policy validity” of VAMThe “policy validity” of VAM

Do VAM of teacher effectiveness have Do VAM of teacher effectiveness have “policy validity?” That is, are they “policy validity?” That is, are they appropriate for practical implementation, appropriate for practical implementation, and for what purposes? (Harris 2007)and for what purposes? (Harris 2007)

If one were to make personnel decisions If one were to make personnel decisions based on VAM, at the very least these based on VAM, at the very least these measures should be:measures should be: Convincing as “causal” estimatesConvincing as “causal” estimates Relatively preciseRelatively precise

Our research questionOur research question

If VAM are meaningful indicators of If VAM are meaningful indicators of teacher effectiveness, they should be teacher effectiveness, they should be relatively consistent across alternative relatively consistent across alternative assessments of the same skills assessments of the same skills (especially for narrowly defined skills)(especially for narrowly defined skills)

In most cases we only observe one In most cases we only observe one assessment – the “high stakes” state assessment – the “high stakes” state assessment – upon which teacher assessment – upon which teacher effects are estimatedeffects are estimated

HoustonHouston

Houston is somewhat unique in that Houston is somewhat unique in that one can observe two measures of one can observe two measures of student achievement:student achievement: TAKS – a “high stakes” examTAKS – a “high stakes” exam Stanford 10 – a “low stakes” examStanford 10 – a “low stakes” exam Both test reading and math skillsBoth test reading and math skills

How consistent are VAM of How consistent are VAM of effectiveness on these two tests?effectiveness on these two tests?

Houston data and methodHouston data and method

Longitudinal student-level data on all Longitudinal student-level data on all students in the Houston ISD, 1998 – 2006 students in the Houston ISD, 1998 – 2006 (we use 2003-06)(we use 2003-06) Students are linked to their teachersStudents are linked to their teachers Student backgroundStudent background About 127,000 studentsAbout 127,000 students

We estimate teacher effects for 4We estimate teacher effects for 4thth and 5 and 5thth grade teachers on grade teachers on both both TAKS and TAKS and Stanford testsStanford tests Using 1 and 3 years of resultsUsing 1 and 3 years of results

Correlation across testsCorrelation across tests

Low- and Low- and high- high- stakes stakes reading reading

Low- and Low- and high-high-stakes stakes mathematmathematicsics

Correlation Correlation coefficientcoefficient

0.340.34 0.410.41

Teacher Fixed Effects on TAKS and Stanford Math Tests

0

10

20

30

40

50

60

High-Stakes 1 High-Stakes 3 High-Stakes 5

High-Stakes Quintile

Pe

rce

nt

of

Te

ac

he

rs i

n

Lo

w-S

tak

es

Qu

inti

le

Low-Stakes 1

Low-Stakes 2

Low-Stakes 3

Low-Stakes 4

Low-Stakes 5

Teacher effects on multiple Teacher effects on multiple teststests

Teacher effects on multiple Teacher effects on multiple tests (one year of data only)tests (one year of data only)

Teacher effects on multiple Teacher effects on multiple subjectssubjects

Teacher effect stabilityTeacher effect stability

ConclusionsConclusions Teachers who are good at promoting Teachers who are good at promoting

growth on a high-stakes test are not growth on a high-stakes test are not necessarily those who are good at necessarily those who are good at promoting growth on a low-stakes tests promoting growth on a low-stakes tests of the same subject.of the same subject.

Teacher effects vary significantly across Teacher effects vary significantly across years and subjectsyears and subjects

Useful for policy? Probably—but we Useful for policy? Probably—but we should resist relying too heavily on these should resist relying too heavily on these measuresmeasures Of course, more research is needed!Of course, more research is needed!