Upload
edolie
View
37
Download
2
Tags:
Embed Size (px)
DESCRIPTION
School-level Correlates of Achievement: Linking NAEP, State Assessments, and SASS NAEP State Analysis Project. Sami Kitmitto. CCSSO National Conference on Large-Scale Assessment June 2006. Overview of the Study. - PowerPoint PPT Presentation
Citation preview
School-level Correlates of Achievement: Linking NAEP,
State Assessments, and SASS
NAEP State Analysis Project
Sami Kitmitto
CCSSO National Conference on Large-Scale Assessment
June 2006
Create a valuable data set for policy analysis by adding achievement scores to a comprehensive school survey
School and Staffing Survey (SASS) Extensive information from a national survey of
schools, but no achievement scores National Assessment of Educational Progress (NAEP)
Nationally representative scores comparable between states
State Assessment Database (NLSLSASD) Collection of all available school-level state
assessment data Scores comparable within states
Overview of the Study Overview of the Study
What are the important school characteristics that correlate with achievement?
Do the results of Don McLaughlin and Gili Drori (2000) compare to the results from a larger and more recent set of data? 2000 SASS vs. 1994 SASS
36-38 states vs. 20 states
Research QuestionsResearch Questions
NAEP 1998, 2000 and 2002
Used 2000 Math Grades 4 & 8 and 1998 & 2002 Reading scores for Grades 4 & 8
Used full population estimates
Mean and standard deviation at the school level
Mean and standard deviation at the state level
Replicate weights used
Data AssemblyData AssemblyNAEP DataNAEP Data
NLSLSASD 2000
Selected two scores for each grade/subject:
Grade 4 Math, Grade 4 Reading
Grade 8 Math, Grade 8 Reading
Remove between state variation
Create standard score within each state:
Data AssemblyData AssemblyNLSLSASD 2000 DataNLSLSASD 2000 Data
s
sisis StateStd
StateMeanSchoolMeanX
NAEP and NLSLSASD Correlation
Using only schools in both NAEP and NLSLSASD:
Calculated correlation between NAEP and NLSLSASD scores at the state level for matched schools
Data AssemblyData AssemblyNAEP and NLSLSASD School-LevelNAEP and NLSLSASD School-Level
Used NAEP to introduce between state differences and variation to standardized scores
Rescaled to mean of 50 and standard deviation of 10
Data AssemblyData AssemblyNAEP State-Level and NLSLSASDNAEP State-Level and NLSLSASD
sssisis NaepMeanNaepSaCorrNaepStdXY )(
)(
10))((50
YStd
YMeanYACHIEVE is
is
School Level Information
From school, principal, teacher and district surveys
Social Background
Organizational Characteristics
School Behavioral Climate
Teacher Characteristics
Data Preparation Step 2Data Preparation Step 2SASS 2000SASS 2000
Analysis Sample Dropped schools with less than 50 students
Did not include schools that were combinations of elementary, middles and or high schools
Missing values: list-wise deletion of observations
Teacher Qualifications Dropped Teacher sample is not random or representative at
the school level
High percent of variation was within schools not between schools
Results indicated that these measures were mostly noise
Data Set Used for AnalysisData Set Used for Analysis
Number of Schools With Two Valid Scores
Number of Schools in Analysis Sample
Data NumbersData Numbers
NAEP/
NLSLSASD SASS Schools
# Schools # Schools # States
Math 34,106 2,287 38 Elementary School Reading 34,099 2,273 37
Math 17,524 1,414 38 Middle School
Reading 15,707 1,333 36
# Schools
Math 1,885 Elementary School Reading 1,883
Math 723 Middle School
Reading 698
Structural Equation Modeling
Similar to multiple regression analysis
Allows for multiple measures of concepts
Models measurement error
Observed variables = Measures
Conceptual factors = Latent Variables
Analysis MethodologyAnalysis Methodology
Path Model Relating Latent Variables
ModelModel
Poverty
Limited English
Proficiency
Race
Normative Cohesion
Student Behavioral
Climate
Class Size
Teacher Influence
Student Academic
Achievement
School Size
Measurement Model
ModelModel
Poverty
Limited English
Proficiency
Race
% Free Lunch Eligible
Poverty a Problem
% LEP
% Non-White
Teacher Attitudes and
Opinions
School Size
Enrollment
Class Size
Average Class Size
Student/ Teacher Ratio
Class Size a Problem
Normative Cohesion
Clear Norms Parcel
Cooperation Parcel
Teacher Influence
Control of Classroom
Parcel
Influence on School Policies Parcel
Student Behavioral
Climate
Climate Problems Parcel #1
Climate Problems Parcel #2
Score #2
Score #1
Student
Academic Achievement
Fit Statistics
Replication ResultsReplication Results
Elementary School Middle School
Math Reading Math Reading
GFI 0.976 0.973 0.973 0.973
AGFI 0.948 0.941 0.942 0.942
RMR 0.029 0.032 0.030 0.030
Chi-Square 381 446 156 149
Chi-Square DF 63 63 63 63
RMSEA Estimate 0.052 0.057 0.045 0.044
90% Lower Limit 0.047 0.052 0.037 0.035
90% Upper Limit 0.057 0.062 0.054 0.053
Bentler's CFI 0.979 0.978 0.986 0.988
Estimated Coefficients for Achievement Equation
Replication Results (cont)Replication Results (cont)
Elementary School Middle School
Math Reading Math Reading
Class Size -0.230 * -0.242 * -0.181 * -0.487 *
School Climate Problems -0.257 * -0.085 -0.743 0.082
Normative Cohesion 0.149 * 0.039 0.153 -0.125
Teacher Influence -0.017 0.022 0.038 0.120
School Size 0.011 -0.052 * 0.042 0.033
Poverty -0.473 * -0.346 * -0.324 -0.415 *
Race -0.184 * -0.347 * -0.106 -0.455 *
Limited English 0.068 * -0.032 0.106 * 0.082 *
R-squared 0.625 0.533 0.738 0.637
Latent variables are scaled to one of their measures
‘Class Size’ is scaled to student/teacher ratio
Coefficients are standardized
A one standard deviation increase in ‘Class Size’ is correlated with a -.23 standard deviation difference in math achievement in elementary schools
Standard deviation of student/teacher ratio in the sample is ~ 4 students/teacher
Mean is 15.5 students/teacher
Interpretation of CoefficientsInterpretation of Coefficients
Reported Estimated Effects of Student/Teacher Ratio and Class Size
Literature on ‘Class Size’Literature on ‘Class Size’
Variable No Effect or Not Significant Small Effect Sizeable Effect
Student/Teacher Ratio
Hanusheck 1986 Prais 1996 Hedges and Greenwald 1996
Eide & Showalter 1998
Todd and Wolpin 2004
Class Size Hoxby 2000 Nye, Hedges & Konstantopoulos 2000
Ferguson 1991
Krueger and Whitmore 2001 Boozer & Rouse 1995
Coates 2003 Krueger 1999
Angrist & Lavy 1999
Fertig & Wright 2005
Add principal responses to school climate questions
Add additional controls: urbanicity, % IEP, magnet school indicator
‘Principal Leadership’
‘Resources’
Per pupil expenditures (district level)
Number of computers
‘Parent Involvement’
Teacher and principal reports of parent involvement being a problem
School programs to involve parents
Avenues for Future ResearchAvenues for Future Research
Linking NAEP, NLSLSASD and SASS provides a powerful national sample of schools matched to achievement scores
SASS provide multiple measures of key conceptual factors
SEM provides a methodology to take advantage of the depth of SASS information
Class size found to be correlated with achievement
In middle schools, more important for reading than math
Results on achievement are similar to McLaughlin and Drori 2000 with improved fit
ConclusionsConclusions