Using Growth Models to Monitor School Performance Over Time: Comparing NCE, Scale and Scores on NRTs and SBTs
American Educational Research AssociationAnnual Meeting
March, 2008
Pete Goldschmidt, Kilchan Choi, Felipe Martinez, and John Novak
Using Growth Model Value Added estimates, do inferences about school change
Examine the role of the metricNCE vs Scale Scores on a Vertically equated assessment.
Examine the role of switching Assessment type NRT vs SBT
Introduction
Summary Parameter Estimates Compared
4
Estimated Initial Status
Residual Initial Status
Estimated Growth
Value Added
Summary of Estimates Compared Using Rank Order Correlations
5
Also compare school ranks based on the residual Initial Status and Value Added estimates
SAT-9 Reading Achievement NCE SS NCE SS NCE SS
Mean Initial status (g000)
Student Predictors
Special Education (010) -0.47 -0.44 -0.47 -0.44 -0.47 -0.44
Low SES (020) -0.36 -0.4 -0.35 -0.4 -0.35 -0.39
LEP (030) -0.34 -0.35 -0.33 -0.34 -0.32 -0.33
Minority (040) -0.48 -0.54 -0.48 -0.54 -0.48 -0.53
Girl (050) 0.1 0.1 0.1 0.1 0.1 0.1
School Predictors
LAAMP Effect (001) 0.03 0.04 0.02 0.03 0.02 0.02
Minority (002) -0.01 -0.01 -0.01 -0.01 -0.01 -0.01
Low (003) 0.13 0.1 0.17 0.15 0.2 0.17
Mean Growth (g100) 0.07 0.64 0.07 0.63 0.07 0.63
Student Predictors
Special Education (110) 0 -0.03 0 -0.03 0 -0.03
Low SES (120) 0.05 0.06 0.05 0.06 0.05 0.06
LEP (130) 0.07 0.07 0.07 0.07 0.07 0.07
Minority (140) -0.03 -0.02 -0.03 -0.02 -0.03 -0.02
Girl (150) 0.01 0.01 0.01 0.01 0.01 0.01
School Predictors
LAAMP Effect (101) 0.01 0.01 0.01 0.01 0.01 0.01
Minority (102) 0.11 0.14 0.12 0.15 0.12 0.16
Low (103) -0.08 -0.08 -0.08 -0.08 -0.08 -0.08
25% 50% 75%
Summary of Results Describing SAT-9 Reading Achievement
6
Sample Test Type Initial Status Growth Growth
R25 Read 0.988 0.936 0.806
Math 0.987 0.963 0.863
R50 Read 0.99 0.932 0.798
Math 0.988 0.964 0.87
R75 Read 0.991 0.932 0.798
Math 0.989 0.964 0.871
0.931
0.929
0.935
0.932
Kendall (Tau) Correlation
Initial Status
0.925
0.925
Spearman Correlation
Correlations Between Value added estimates for NRT for models without student covariates
8
Sample Test Type Initial Status Growth Growth
R25 Read 0.964 0.914 0.779
Math 0.975 0.955 0.849
R50 Read 0.97 0.91 0.775
Math 0.978 0.956 0.857
R75 Read 0.974 0.908 0.776
Math 0.981 0.955 0.857
Spearman Correlation Kendall (Tau) Correlation
Initial Status
0.857
0.905
0.887
0.872
0.898
0.881
Correlations Between Value added estimates for NRT for models with student covariates
9
-0.89
-0.88
-0.87
-0.86
-0.85
-0.84
-0.83
-0.82
-0.81
-0.80
-0.79
0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.70 0.71 0.72 0.73
Efect Size of Growth (scale scores)
Rela
tive B
ias
in G
row
th
Comparison of Relative Bias to the Effect Size of Growth
13
Correlations between School Means by Year:
NRT1 and SBT2
Year
2002
(SAT-CST) 2003
(CAT-CST) 2004
(CAT-CST) Reading 0.971 0.967 0.960 Math 0.949 0.955 0.887
1 NRT consists of SAT9 and CAT6 2 SBT is California Standards Test
Estimated effects of student characteristics in initial Status and Growth
for NRT and SBT Reading Mathematics NRT SBT NRT SBT Mean Initial status (g000) 0.17 -0.13 0.14 0.07
Girl (g010) 0.07 0.15 -0.09 -0.14 Special Education (g020) -0.70 -0.72 -0.71 -0.74 Low SES (g030) -0.47 -0.60 -0.46 -0.54 LEP (g040) -0.14 -0.15 0.04 0.01 Minority (g050) -0.30 -0.31 -0.30 -0.37
Mean Growth (g100) -0.26 0.13 -0.20 -0.01 Girl (g110) 0.09 0.01 0.02 0.05 Special Education (g120) 0.07 0.05 0.03 0.10 Low SES (g130) -0.08 0.00 -0.03 -0.01 LEP (g140) 0.04 0.06 0.02 0.06 Minority (g150) -0.01 0.02 0.01 0.07
Correlations among Value Added estimates based on:
NRT1 and SBT2 Spearman Correlation Kendall (Tau) Correlation
Sample Test Type Initial Status Growth Initial Status Growth
Model 1 -Unconditional Growth
Read 0.979 0.548 0.880 0.340
Math 0.966 0.793 0.830 0.627
Model 2 – Growth with Covariates
Read 0.954 0.740 0.781 0.534
Math 0.933 0.808 0.772 0.659
1 NRT consists of SAT9 and CAT6 2 SBT is California Standards Test
Mathematics
School context and inferences
While individual student characteristics’ impact differ depending on assessment used (though not metric) -particularly for growth,
School enrollment characteristics have virtually no impact inferences between NRT and SBT.
Mathematics
Reading
Mathematics
Relationship between missing scores and school performance
Performance based on Percent MissingSchool Means
NRT -0.42 *CST -0.42 *
Value AddedNRT 0.2CST 0.05
* p < .05
Relationship between missing scores and school performance
Summary – the scale
Using a relative scale for monitoring individual achievement growth when the assessment is vertically equated – significantly under-estimates growth.
Using a relative scale for monitoring school performance based on growth when the assessment is vertically equated – yield very consistent results to using an absolute scale.
No patterns as to where deviations may occur.
Summary – the assessment
Individual results between NRT and CST highly correlated in each year.
Individual student characteristics affect relative performanceAttempting to become more egalitarian?
School results fairly consistent in Mathematics, but not in Language Arts
School characteristics have virtually no impact on changes in inferences or rankings of schools.
Summary – the method
Means highly correlated with student background
Means inversely correlated to misingness
VA added estimates based on individual growth substantively less related to student background
VA added estimates based on individual growth substantively less related to missingness