Upload
melinda-rowe
View
17
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Longitudinal Analysis of MAP Achievement Growth: Preliminary Estimates of School Effects Sept. 15-16, 2010 Mark Ehlert Cory Koedel Michael Podgursky Department of Economics, MU CALDER, NCPI Kansas City Area Education Research Consortium - PowerPoint PPT Presentation
Citation preview
1
Longitudinal Analysis of MAP Achievement Growth:Preliminary Estimates of School Effects
Sept. 15-16, 2010
Mark EhlertCory KoedelMichael PodgurskyDepartment of Economics, MU
CALDER, NCPIKansas City Area Education Research Consortium
Prepared for Missouri Technical Advisory Committee meeting. Kansas City, MO.September 15-16, 2010
2
Overview
• Examination of emerging MOSIS data system
• Patterns of Scale Score growth in MAP• A Simple VAM for School Effects
– Model and results– Covariates or not?
• Estimation of Teacher Program Effectiveness
• Future directions
3
• Missouri is developing a sophisticated P-20 data system
• IES State Longitudinal Data Grant• Ranks very favorably compared to other
states• Data quality is high
4
Data
• Matched Spring 2006-2009 student MAP scores using MOSIS ID
• Exclusions– Bad/duplicate values of MOSIS– Students retained in grade– Special districts
• Match rate for 4 years roughly 85% - 87%; match rate for 1 year regularly at 95%
5
Grade 2006 2007 2008 2009
3 a b c d
4 a a b c
5 a a a b
6 a a a a
7 a a a a
8 a a a a
10 a
EoC a
Math MAP Testing Regime
6
620
640
660
680
700
720
3 4 5 6 7 8
Aver
age
Scal
e Sc
ore
MAP Math: Average Performance Growthby Cohort
Grade06 = 3 Grade06 = 4 Grade06 = 5
MAP Math: Average Performance
By Cohort
7
620
640
660
680
700
720
3 4 5 6 7 8
Ave
rage
Sca
le S
core
MAP Com Arts: Average Performance Growthby Cohort
Grade06 = 3 Grade06 = 4 Grade06 = 5
MAP Com. Arts : Average Performance
By Cohort
8
MAP Math: 2008-2009 Average Gain Score
By Grade and Decile of 2008 Performance
9
MAP Com Arts: 2008-2009 Average Gain Score
By Grade and Decile of 2008 Performance
10
Value-added models
• Why “value-added?”• Traditional economic definition
– Business or firm value-added• Value of output – value of inputs• VAT
• Education analogy– Control for initial (pre-treatment) performance– Estimate the effect of contemporaneous inputs on education
outcomes
11
• We want to identify causal effect of inputs– “what works”
• Treatment and control / comparison groups– Example: teacher training programs and
teacher effectiveness– Class size– Teacher credentials
12
A i j t = f (A i t – k , S i , SCH j ) + ε i j t
Educational outcome (e.g., test score, graduation, college attendance)
Lagged or baseline performance
Student Characteristics
School / classroominputs or treatment
random error
A Simple VAM
i – th student
j –th school or classroom
t – th year or grade
13
A i g - A i g-1 = f (Ai g-1 (m, ca), student char, grade, year)
+ school effects + ε i t
Model estimated over all Missouri students, grades 3-8
Schools included if n > 20 student gain scores
3 gain scores x multiple grades per school
Gain Score Lagged Test Scores in both subjects
Average Effect by school (state mean = 0)
14
Appendix B – Regression Coefficients and Related Statistical Estimates
Dependent Variable
Math Com Arts
Coefficient T-Statistic Coefficient T-Statistic Past Scores:
First-Year Scale Score in Math - 0.34194** - 30.81 0.19145** 18.91 Squared First-Year Scale Score in Math
- 0.00003** - 3.37 0.00001 0.98
First-Year Scale Score in Com Arts
- 0.48245** - 38.19 - 0.54505** - 47.30
Squared First-Year Scale Score in Com Arts
0.00054** 55.42 0.00007** 7.84
Indicators for Student Characteristics:
American-Indian - 0.92858** - 2.72 0.08011 0.26 Asian/Pacific Islander 4.64526** 26.53 1.49822** 9.38 Black - 4.08531** - 47.43 - 1.64919** - 20.99 Hispanic - 0.89970** - 6.46 - 0.09035 - 0.71 Female - 2.16402** - 48.82 3.60541** 89.18 Special Education - 6.16846** - 86.27 - 8.07978** -123.90 Limited English Proficiency 0.24188 1.38 - 2.12545** - 13.32 Free/Reduced Price Lunch Eligibility
- 2.37397** - 46.78 - 2.38827** - 51.61
In the School Less Than a Full School Year
- 3.33210** - 29.97 - 2.63134** - 25.95
Indicators for Grade and Year: Terminal-year Grade 4 -19.46713** -157.16 - 8.14487** - 72.10 Terminal-year Grade 5 -18.22465** -161.81 - 6.14417** - 59.81 Terminal-year Grade 6 -13.95812** -170.22 -16.47484** -220.29 Terminal-year Grade 7 -17.73426** -254.94 -12.02678** -189.57 Dummy for 2006-2007 - 0.45133** - 8.40 - 1.45330** - 29.67 Dummy for 2007-2008 - 0.60767** - 11.47 - 0.24555** - 5.08
R2 = 0.251 R2 = 0.308 Number of School Effects = 1,773 (except the reference school) Sample Size(Number of Gainscores) = 926,358 ** denotes that coefficient is significant at 1% level.
15
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Eligible for Free/Reduced-Priced Lunch in School
Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free-Reduced Priced Lunch
Others District X: Significant District X: Insignificant
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Significant: Significantly different from the statewide average of school effects.
16
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Minorities
Standardized School Effects on MAP Math Performancevs. Percent Minorities
Others District X: Significant District X: Insignificant
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Significant: Significantly different from the statewide average of school effects.
17
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percentile Rank
Standardized School Effects on MAP Math PerformanceIn Rank Order
Others District X: Significant District X: Insignificant
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Significant: Significantly different from the statewide average of school effects.
18
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Eligible for Free/Reduced-Priced Lunch in School
Standardized School Effects on MAP Com Arts Performancevs. Percent Eligible for Free-Reduced Priced Lunch
Others District X: Significant District X: Insignificant
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Significant: Significantly different from the statewide average of school effects.
19
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Minorities
Standardized School Effects on MAP Com Arts Performancevs. Percent Minorities
Others District X: Significant District X: Insignificant
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Significant: Significantly different from the statewide average of school effects.
20
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percentile Rank
Standardized School Effects on MAP Com Arts PerformanceIn Rank Order
Others District X: Significant District X: Insignificant
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Significant: Significantly different from the statewide average of school effects.
21
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
Stan
dard
ized
Sch
ool E
ffec
t for
Com
Art
s*
Standardized School Effect for Math*
Standardized School Effects on MAP PerformanceCom Arts vs. Math
Others District X
Coefficients of Correlation(ComArts, Math):
Pearson = 0.75271Spearman = 0.70142
*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
22
Effect of Covariates(math results)
Model 1 = student covariates
Model 2 = no student covariates
23
-1
-0.5
0
0.5
1
-1 -0.5 0 0.5 1
Stan
dard
ized
Scho
ol Eff
ect*
from
Mod
el 2
Standardized School Effect* from Model 1
Standardized School Effects on MAP Math PerformanceModel 1 vs. Model 2
Model 1. WITH student covariatesModel 2. WITHOUT student covariates
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Coefficients of Correlation (Model 1, Model 2):
Pearson = 0.97989Spearman = 0.97581
24
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Eligible for Free/Reduced-Priced Lunch in School
Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free/Reduced-Priced Lunch
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1
2, School Dummies, Grade, Year, Student level covariates)
25
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Eligible for Free/Reduced-Priced Lunch in School
Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free/Reduced-Priced Lunch
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1
2, School Dummies, Grade, Year)
26
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Minorities in School
Standardized School Effects on MAP Math Performancevs. Percent Minorities
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1
2, School Dummies, Grade, Year, Student level covariates)
27
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Minorities in School
Standardized School Effects on MAP Math Performancevs. Percent Minorities
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)
Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1
2, School Dummies, Grade, Year)
28
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Eligible for Free/Reduced-Priced Lunch in School
Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free/Reduced-Priced Lunch
District X
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1
2, School Dummies, Grade, Year, Student level covariates)
29
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Eligible for Free/Reduced-Priced Lunch in School
Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free/Reduced-Priced Lunch
District X
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1
2, School Dummies, Grade, Year)
30
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Minorities in School
Standardized School Effects on MAP Math Performancevs. Percent Minorities
District X
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1
2, School Dummies, Grade, Year, Student level covariates)
31
-1
-0.5
0
0.5
1
0 20 40 60 80 100
Stan
dard
ized
Sch
ool E
ffec
t*
Percent Minorities in School
Standardized School Effects on MAP Math Performancevs. Percent Minorities
District X
*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)
Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1
2, School Dummies, Grade, Year)
32
Work Under Way
• Teacher training program effects– New teachers
• Retirement system effects– Effectiveness of teachers x retirement
behavior
33
A i g - A i g-1 = f (Ai g-1 (m, ca), student char, grade, year)
+ school effects + teacher effects + ε i t
Model estimated over all Missouri students, grades 3-8
Within school
34
A i g - A i g-1 = f (Ai g-1 (m, ca), student char, grade, year)
+ school effects + teacher char + ε i t
Model estimated over all Missouri students, grades 3-8
Within school
35
Comparative Effectiveness of Teacher Preparation Programs
36
37 Teacher Training Programs
37
Schools with at least one newteacher graduate:Fall, 2005 – Fall, 2009
38
39
40
41
42
43
44
37 x 37 cross placement of program grads x school, gr. 4-8
PSI1 591 156 122 45 91 96 90 30 51 192 46 139 40 33 28 34 10 13 11 23 143 82 102 19 18 10 12 10 6 11 4 10 13 9 4 11 0
PSI2 156 501 193 59 41 53 38 61 28 23 201 43 15 68 160 32 68 17 11 30 37 7 7 8 19 12 40 11 4 65 2 62 0 6 6 0 2
PSI3 122 193 352 53 139 130 60 64 34 12 80 14 50 29 37 77 15 20 26 57 14 2 3 49 14 22 21 24 16 17 6 19 12 3 21 0 0
PSI4 45 59 53 157 47 32 102 13 70 20 18 11 12 10 3 20 4 9 14 12 11 2 4 6 4 0 4 2 14 5 4 3 4 12 2 0 0
PSI5 91 41 139 47 365 222 124 61 109 4 12 2 107 9 4 10 3 2 75 2 2 0 8 70 0 4 12 22 4 2 4 2 44 5 24 0 0
PSI6 96 53 130 32 222 241 88 48 77 12 11 11 62 0 13 7 6 12 52 8 2 0 6 39 2 6 0 21 20 0 3 2 22 2 2 0 0
PSI7 90 38 60 102 124 88 338 15 86 10 4 14 40 0 10 5 6 2 26 6 0 6 7 24 6 0 2 2 37 2 7 0 23 2 2 0 0
PSI8 30 61 64 13 61 48 15 97 18 4 21 4 26 10 10 8 6 10 21 6 0 4 0 18 6 2 5 12 6 6 2 0 2 0 10 0 0
PSI9 51 28 34 70 109 77 86 18 124 4 0 5 25 5 0 2 4 6 30 4 7 2 3 20 2 6 0 8 12 2 4 0 11 4 0 0 0
PSI10 192 23 12 20 4 12 10 4 4 165 4 46 5 4 8 15 2 2 2 15 20 34 30 0 6 4 2 0 0 0 0 4 0 15 0 4 0
PSI11 46 201 80 18 12 11 4 21 0 4 273 17 5 113 76 9 29 18 2 6 21 4 5 0 10 9 33 0 11 34 2 26 0 0 0 0 0
PSI12 139 43 14 11 2 11 14 4 5 46 17 115 4 4 20 6 4 2 0 6 23 11 6 0 2 5 4 0 0 2 0 5 0 3 0 0 0
PSI13 40 15 50 12 107 62 40 26 25 5 5 4 79 9 4 6 0 4 30 2 0 5 2 18 0 4 0 10 6 0 0 0 10 2 7 0 0
PSI14 33 68 29 10 9 0 0 10 5 4 113 4 9 127 20 8 6 4 2 0 8 7 3 2 4 0 18 0 10 6 0 15 0 2 0 0 0
PSI15 28 160 37 3 4 13 10 10 0 8 76 20 4 20 113 7 29 0 2 2 9 4 2 0 2 3 15 0 0 26 0 22 0 0 0 0 2
PSI16 34 32 77 20 10 7 5 8 2 15 9 6 6 8 7 65 0 0 2 26 6 0 0 3 10 13 0 0 6 0 4 0 0 2 0 0 0
PSI17 10 68 15 4 3 6 6 6 4 2 29 4 0 6 29 0 40 0 2 4 2 0 0 2 2 0 2 0 0 12 0 9 0 0 0 0 0
PSI18 13 17 20 9 2 12 2 10 6 2 18 2 4 4 0 0 0 41 0 6 0 5 0 2 2 2 0 2 24 0 2 0 0 0 0 0 0
PSI19 11 11 26 14 75 52 26 21 30 2 2 0 30 2 2 2 2 0 63 0 2 0 0 10 0 0 0 5 0 0 2 0 8 0 5 0 0
PSI20 23 30 57 12 2 8 6 6 4 15 6 6 2 0 2 26 4 6 0 65 0 0 0 0 0 2 0 2 0 2 4 0 0 0 2 0 0
PSI21 143 37 14 11 2 2 0 0 7 20 21 23 0 8 9 6 2 0 2 0 161 11 10 2 6 2 7 0 0 0 0 0 0 0 0 0 0
PSI22 82 7 2 2 0 0 6 4 2 34 4 11 5 7 4 0 0 5 0 0 11 63 10 0 2 0 0 0 10 0 0 0 0 5 0 2 0
PSI23 102 7 3 4 8 6 7 0 3 30 5 6 2 3 2 0 0 0 0 0 10 10 48 2 3 0 4 2 0 0 0 0 0 0 0 0 0
PSI24 19 8 49 6 70 39 24 18 20 0 0 0 18 2 0 3 2 2 10 0 2 0 2 54 0 0 0 13 0 0 0 0 6 0 2 0 0
PSI25 18 19 14 4 0 2 6 6 2 6 10 2 0 4 2 10 2 2 0 0 6 2 3 0 39 0 2 0 0 0 0 0 0 0 0 0 0
PSI26 10 12 22 0 4 6 0 2 6 4 9 5 4 0 3 13 0 2 0 2 2 0 0 0 0 17 0 2 0 0 2 2 0 0 0 0 0
PSI27 12 40 21 4 12 0 2 5 0 2 33 4 0 18 15 0 2 0 0 0 7 0 4 0 2 0 43 0 0 4 0 16 0 0 0 0 0
PSI28 10 11 24 2 22 21 2 12 8 0 0 0 10 0 0 0 0 2 5 2 0 0 2 13 0 2 0 20 2 0 0 0 2 0 0 0 0
PSI29 6 4 16 14 4 20 37 6 12 0 11 0 6 10 0 6 0 24 0 0 0 10 0 0 0 0 0 2 49 0 0 0 2 0 0 0 0
PSI30 11 65 17 5 2 0 2 6 2 0 34 2 0 6 26 0 12 0 0 2 0 0 0 0 0 0 4 0 0 28 0 5 0 0 0 0 0
PSI31 4 2 6 4 4 3 7 2 4 0 2 0 0 0 0 4 0 2 2 4 0 0 0 0 0 2 0 0 0 0 10 0 2 0 0 0 0
PSI32 10 62 19 3 2 2 0 0 0 4 26 5 0 15 22 0 9 0 0 0 0 0 0 0 0 2 16 0 0 5 0 37 0 0 0 0 0
PSI33 13 0 12 4 44 22 23 2 11 0 0 0 10 0 0 0 0 0 8 0 0 0 0 6 0 0 0 2 2 0 2 0 32 0 0 0 0
PSI34 9 6 3 12 5 2 2 0 4 15 0 3 2 2 0 2 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0
PSI35 4 6 21 2 24 2 2 10 0 0 0 0 7 0 0 0 0 0 5 2 0 0 0 2 0 0 0 0 0 0 0 0 0 0 19 0 0
PSI36 11 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0
PSI37 0 2 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4
45
Other research
Teacher Pension Effects
How do pension rules affect workforce quality?