Upload
stuart-carter
View
221
Download
3
Tags:
Embed Size (px)
Citation preview
Accountability and Students with Disabilities: Assuring Valid Inferences about Teachers and Schools
CCSSO National Conference on Student AssessmentNew Orleans, LAJune 27, 2014Strand 11A
Heather Buzick
Research ScientistEducational Testing
ServiceCenter for Validity
Research
Ann Schulte
Research ProfessorArizona State University
James Ysseldyke
Birkmaier Professor of Educational LeadershipUniversity of Minnesota
Daniel Wiener
Administrator of Inclusive Assessment
Massachusetts Department of Elementary and
Secondary Education
IES-funded 5-state longitudinal study on
student growth
Research on teacher evaluation
Research supported by IES-funded National
Research and Development Center on Assessment and Accountability for Special
Education
Observations from Massachusetts and discussant remarks
National advisory panel member for NCAASE
NCEO research
Assuring valid inferences about teachers and schools…
What do we know about the academic growth of students with disabilities?
What choices do we have for modeling,
reporting, and evaluating?
How can we make valid inferences
about teachers and schools educating
students with disabilities?
Copyright © 2014 by Educational Testing Service. 4
Describing and Using Growth from Students with Disabilities on Summative Assessments
Heather BuzickEducational Testing ServicePrinceton, NJ
A portion of this research was funded by the Institute for Education Sciences (Award #R324A120224)
Copyright © 2014 by Educational Testing Service. 5
Current research•Describi
ng Summative Academic Growth for Students with Disabilities: A Summary from 5 States
Study 1
•Using Test Scores from Students with Disabilities in Teacher Evaluation
Study 2
Copyright © 2014 by Educational Testing Service. 6
Motivation and importance
• Approximately 14% of students have a diagnosed disability• The majority spend most of instructional time in general
education classroom*• Approximately 80% of teachers have at least one student
with a disability in their classroom**• At least 75% of students with disabilities take the general
assessment* • Students’ disabilities can have an impact on access to test
content, student may require testing accommodations, teaching and learning may differ from other students
• How should students’ test scores be included in accountability systems?
*from Historical State-Level IDEA Data Files (http://tadnet.public.tadnet.org/pages/712**Estimate. Sources available from author
Copyright © 2014 by Educational Testing Service. 7
Study 1 Research Questions
What does growth on state summative
assessments look like for students with
disabilities?
Does growth differ by grade, content area,
and disability subtype?
How does the growth of students with
disabilities compare to students without a
disability?
Where is growth for students with
disabilities occurring relative to the
proficiency cutscore?
Copyright © 2014 by Educational Testing Service. 8
Some definitions of growth within individual students
• Differences in vertically scaled scores (gains)– E.g., 300 scaled score in grade 3, 320 scaled
score in grade 4
• Transitions across proficiency levels– E.g., “basic” in grade 3, proficient in grade 4
• Student growth percentiles – From grade 3 to grade 4, the student grew as
much as or more than 70 percent of other students in the state peers who had similar grade 3 test scores
3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8Grew below proficiency Grew across proficiency Grew above proficiency
ELA
0
5
10
15
20
25Transitions across proficiency levels
NoneAutismEmotional ImpairmentOther health impairmentSpecific Learning DisabilitiesSpeech or language impairment
Pe
rce
nt
3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8Grew below proficiency Grew across proficiency Grew above proficiency
Math
0
5
10
15
20
25Transitions across proficiency levels
NoneAutismEmotional impairmentOther heath impairmentSpecific learning disabilitiesSpeech or language impairment
Pe
rce
nt
3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8Low Moderate High
ELA
0
10
20
30
40
50
60
70Student growth percentiles
NoneAutismEmotional impairmentOther health impairmentSpecific learning disabilitiesSpeech or language impairment
Pe
rce
nt
3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8Low Moderate High
MATH
0
10
20
30
40
50
60
70Student growth percentiles
NoneAutismEmotional impairmentOther health impairmentSpecific learning disabilitiesSpeech or language impairment
Pe
rce
nt
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3Difference in 5th Grade Math Teachers’ Value-added Scores
Classrooms with no students with disabilitiesMixed classroomsSpecial education-only classrooms
Percent of students with disabilities in the classroom
[Mo
de
l w
ith
dis
ab
ilit
y-r
ela
ted
va
ria
ble
s]-
[Mo
de
l w
ith
ou
t d
isa
bil
ity
-re
late
d v
ari
ab
les
]
Copyright © 2014 by Educational Testing Service.
Conclusions
• The model matters for the inferences we make about schools, teachers, and individual students (sometime is subtle, but important ways)– Policy makers: identify the claims they wish to make– Measurement experts: help policymakers understand the
meaning derived from a particular model
• How much growth is enough?– Norms based on accumulated growth data from multiple
sources– Prediction associated with college- and career-ready standards
Accountability Dilemmas for Students with Disabilities and Policy Alternatives
Ann SchulteArizona State University
Natalie MurrNorth Carolina State University
Joseph StevensUniversity of Oregon
17
National Center on Assessment & Accountability for Special Education
• NCAASE www.ncaase.com
• Institute of Education Sciences, 2011-2016• Co-PI’s
– Stephen Elliott & Ann Schulte, Arizona State Univ– Joseph Stevens & Gerald Tindal (Project Director), Univ of
Oregon
This work is supported by the Institute of Education Sciences, U.S. Department of Education, through grant R32C110004 awarded to the University of Oregon. The opinions expressed are those of the authors and do not necessarily represent views of the Institute or the U.S. Department of Education.
19
NCAASE 2011-2016:Our Key Research Questions
1. What is the natural developmental progress in achievement for students with disabilities?
2. What models best characterize achievement growth for students with disabilities who are participating in general achievement tests?
3. How do various growth models represent school effects for students with and without disabilities, and how do results compare to those derived from the status models now in use?
4. How do results from different types of interim assessments of students’ achievement meaningfully contribute to a model of academic growth for students with disabilities?
5. How can information about opportunity to learn and achievement growth be used to enhance academic outcomes for students with disabilities?
Persistent Accountability Dilemmas
• Bias introduced by including only current students with disabilities in students with disabilities (SWD) subgroup (Ysseldyke & Bielinski, 2002)
• “One shot” model of assessing proficiency and SWD performance variability—retests to assure assessment fairness (Wei, 2012)
• Students start at differing levels, status measures do not consider student progress relative to starting point—importance of looking at growth to assess school and teacher effects (Dunn & Allen, 2009; Stevens, 2005)
Data Sources for Presentation
• North Carolina test data (NCAASE also looking at AZ, OR, PA)
• Cross sectional-2010– Allowed retests for non-proficient student, inclusion of
students who had exited special education for two years or less
– State-level growth metric—residual gain score using two prior years’ test scores, z-score score based on mean gain and sd in standard setting year
• Longitudinal—Math 2001-2005 cohort, Reading 2003-2007 cohort
Impact of Two Specific Policies
• Including students who have exited special education
• Allowing retesting for students who do not reach proficiency
23
3 4 5 6 7-1.20
-1.00
-0.80
-0.60
-0.40
-0.20
0.00
Special Education in Current Year SWD
Special Education at Wave 1 SWD
Grade
Eff
ect
Siz
e
Mathematics Achievement Gap
Stable Subgroup Membership Matters
Change in Mean Number of Students Reaching Proficiency
Rdg/LA Mathematics
Mean SD Mean SD
Current Participation in Special Education Only 40 21 59 20
Including Students Two Years Post Dismissal 47 19 65 18
Net Change in Percent Proficient
+7 +6
24
Change in School-level Percent Proficient for SWD w/ Exiters Included
Mathematics Reading0
10
20
30
40
50
60
70
80
90
100
Current onlyWith 2-Yr Exited
SWD’s Reaching Math Proficiency With and Without Retest
Grade 3 Grade 4 Grade 50
10
20
30
40
50
60
70
80
90
100
Single TestW/ Retest
SWD’s Reaching Reading Proficiency With and Without Retest
Grade 3 Grade 4 Grade 50
10
20
30
40
50
60
70
80
90
100
Single TestW/ Retest
Growth vs. Proficiency
• What does growth across grades look like for specific exceptionalities?
• Relationship between status and growth for students with and without disabilities
Mathematics Growth by Exceptionality
3 4 5 6 7235
240
245
250
255
260
265
270
275
General Education
Speech-language Impairment
NC Proficiency Cutpoint
Grade
Mat
hem
atic
s S
cale
Sco
re
Mathematics Growth by Exceptionality
3 4 5 6 7235
240
245
250
255
260
265
270
275
General Education
Speech-language Impairment
Autism
Hearing Impairment
NC Proficiency Cutpoint
Grade
Mat
hem
atic
s S
cale
Sco
re
Mathematics Growth by Exceptionality
3 4 5 6 7235
240
245
250
255
260
265
270
275
General Education
Speech-language Impairment
Autism
Other Health Im-pairment
Hearing Impairment
Specific Learning Disability
NC Proficiency Cutpoint
Grade
Mat
hem
atic
s S
cale
Sco
re
Mathematics Growth by Exceptionality
3 4 5 6 7235
240
245
250
255
260
265
270
275
General Education
Speech-language Impairment
Autism
Other Health Im-pairment
Hearing Impairment
Emotional Dis-turbance
Specific Learning Disability
Intellectual Disability
NC Proficiency Cutpoint
Grade
Mat
hem
atic
s S
cale
Sco
re
Reading Growth by Exceptionality
3 4 5 6 7230
235
240
245
250
255
260
265
General Education
Speech-language Impairment
NC Proficiency Cutpoint
Grade
Rea
ding
Sca
le S
core
Reading Growth by Exceptionality
3 4 5 6 7230
235
240
245
250
255
260
265
General Education
Speech-language Impairment
Autism
Hearing Impairment
NC Proficiency Cutpoint
Grade
Rea
ding
Sca
le S
core
Reading Growth by Exceptionality
3 4 5 6 7230
235
240
245
250
255
260
265
General Education
Speech-language Impairment
Autism
Other Health Im-pairment
Hearing Impairment
Specific Learning Disability
NC Proficiency Cutpoint
Grade
Rea
ding
Sca
le S
core
Reading Growth by Exceptionality
3 4 5 6 7230
235
240
245
250
255
260
265
General Education
Speech-language Impairment
Autism
Other Health Im-pairment
Hearing Impairment
Emotional Dis-turbance
Specific Learning Disability
Intellectual Disability
NC Proficiency Cutpoint
Grade
Rea
ding
Sca
le S
core
Growth by Starting Proficiency Level-Math
General Ed SWD
Growth by Starting Proficiency Level-Rdg
General Ed SWD
Conclusions
• SWD subgroup is not stable and policy changes allowing longer time to “count” in subgroup improve school SWD outcomes
• Retesting benefits SWDs and may also be likely to benefit other groups characterized by large achievement gaps
• SWDs show growth mathematics and reading achievement across grades, although improvement may not be reflected in changes in status (Non-proficient/proficient)
• Large differences in starting point achievement skills within SWD group, smaller differences in growth
Accountability and Students with Disabilities: Assuring valid inferences about teachers and schools
Jim Ysseldyke
Purposes of Monitoring Student Growth
• District/State Accountability• Individual Progress
Monitoring/instructional planning• Teacher evaluation (value added)
Typical Accountability Models for SWD
• Cross-sectional• Cohort Static• Cohort Dynamic
Typical Scores
• Scaled Scores• Proficiency Levels• Effect Sizes• More recently Student Growth
Percentiles (ala Betebenner) or Student Deciles in some of our work
Main Issues
• Reducing achievement gap (GE v SE)• Nobody wants SWD in their
accountability profile• How long should SWD count?• What model should be used?• What scores should be used?
Major Points I Heard
• Students start at differing levels, so status measures do not consider student progress relative to starting point
• Use of cross sectional dangerous• SWD are growing, but many may not
meet proficiency standards
Major Points I Heard
• We have limited data on growth norms for SWD (small Ns)
• Much concern about how long to count SWD (at district or state level).
Achievement Gap Using 3 Analytic Methods over 6 Years
21.1
27.3
32.0
40.646.2
47.4
22.0
28.4 31.0
36.3
47.0
45.4
22.6 22.725.1 26.2
21.7
0
10
20
30
40
50
60
3 4 5 6 7 8
Per
cent
age
of S
tude
nts
abov
e P
rofic
ienc
y Le
vel
Grade
Cross Sectional Cohort Dynamic Cohort Static
Spring-Fall SGP Growth by Category v National Norms
STAR Reading Growth for Grade 10 SED Students
Spring to Fall SGP for Students with SED in Differing Programs
Big Ideas
• We can identify expectations for SWD, and we can build growth norms for SWD, but it will take large databases to do so. State tests change too often. Suggest large databases such as Renaissance Learning STAR databases, AIMSweb, Easy CBM. FAST or Oregon Behavioral Research & Teaching Institute
Big Ideas
• Use Cohort Static Model and Effect Sizes• Include SWD in SWD group for two years
after exit as a policy rule• SWD should count in teacher evaluation
systems• More research on how accommodation use
should be factored into growth models• The promise of Goal Setting Tool
Accountability and Students with Disabilities:
Assuring Valid Inferences About Teachers and Schools
Dan Wiener
Administrator of Inclusive Assessment
CCSSO NCSA
June 27, 2014
MASSACHUSETTS:Student Growth Percentiles (SGP)+ District-Determined Measures
(DDM)= Student Impact Rating
Students with disabilities (SWD) may demonstrate high growth, but are more likely to have a lower SGP.
Therefore, teachers of SWD would be more likely to meet the definition for low growth, if student growth was the only measure.
Determine Student Impact Rating for those educators using professional judgment of multiple measures over multiple years.
DDMs should reflect the different ways SWD show learning and growth.
Massachusetts Department of Elementary and Secondary Education
54
Multiple Measures
Districts are required to use median SGPs as one measure for evaluating teacher effectiveness
One or more DDMs must also be used as a measure.
Consider the learning setting, shared responsibilities of teachers, creative use by districts of composite measures
Educators should be involved in setting these parameters.
Massachusetts Department of Elementary and Secondary Education
55
Massachusetts Department of Elementary and Secondary Education
Statewide MCAS ResultsStudents with Disabilities
Change in Performance 2009 to 2013 (5-Year Trends)
Percent of Students with Disabilities Scoring Needs Improvement and Higher
English Language Arts
Mathematics Science & Tech/Eng.
2009 2013 Change 2009 2013 Change 2009 2013 Change
Grade 3 65 68 +3 59 57 -2
Grade 4 58 56 -2 62 61 -1
Grade 5 70 62 -8 51 50 -1 66 63 -3
Grade 6 70 63 -7 47 51 +4
Grade 7 70 70 0 38 46 +8
Grade 8 72 76 +4 38 43 +5 47 50 +3
Grade 10 81 88 +7 68 73 +5 70 79 +9
Massachusetts Department of Elementary and Secondary Education
57
ELA
% of Students Scoring Needs Improvement or
Higher
Between-Group Gap:Percentage Point Difference
Students with
Disabilities
Non-disabledStudents
Students with Disabilitiesvs. Non-disabled Students
Grade 3 69 97 28Grade 4 54 94 40Grade 5 63 96 33Grade 6 64 96 32Grade 7 72 98 26Grade 8 70 97 27Grade 10 88 99 11
Spring 2013 MCAS ELA:Achievement Gap between Students with Disabilities and Non-disabled
Students:
Massachusetts Department of Elementary and Secondary Education
58
Mathematics
% of Students Scoring Needs Improvement or
Higher
Between-Group Gap:Percentage Point Difference
Students with
Disabilities
Non-disabledStudents
Students with Disabilitiesvs. Non-disabled Students
Grade 3 63 94 31Grade 4 65 96 31Grade 5 54 93 39Grade 6 50 92 42Grade 7 39 87 48Grade 8 40 88 48Grade 10 70 97 27
Spring 2013 MCAS Mathematics:Achievement Gap between Students with Disabilities and Non-disabled
Students:
End of Gr. 10 (2011)
End of Gr. 11 (2012)
End of Gr. 12 (2013)
0
20
40
60
80
100
9397
99
61
7681
43
58
82
Percent of Students in Class of 2013 Scoring Needs Improvement or
Higher on MCAS ELA, Math, and STE
Non-Disabled Students Students with Disabil-itiesEnglish Language Learners
Perc
en
t
59
Massachusetts Department of Elementary and Secondary Education