View
217
Download
0
Category
Tags:
Preview:
Citation preview
Designing Accessible Reading Assessments
Examining Test Items for Differential Distractor Functioning
Among Students with Learning Disabilities
Kyndra Middleton
The University of Iowa
kyndra-middleton@uiowa.edu
April 10, 2007
Designing Accessible Reading Assessments
Purpose of the Study
• To examine whether different distractor choices functioned differentially for students with learning disabilities who did not receive an accommodation, students with learning disabilities who received a read-aloud accommodation, and students with learning disabilities who received an accommodation other than a read-aloud
• To help determine whether a test can be modified for students with learning disabilities by removing a distractor choice while maintaining adequate test validity and information
Designing Accessible Reading Assessments
Instrument Used
• 4th grade English Language Arts assessment from a criterion-referenced statewide test– Operational test data
– Reading (42 MC items)– Writing (33 MC items)
Designing Accessible Reading Assessments
Sample Used
• 30,000 non-LD students sampled from 298,622 students
• 9,056 LD students who did not receive an accommodation
• 4,727 LD students who received an accommodation based on their IEP/504 plan
• 1,371 LD students who received an accommodation based on their IEP/504 plus a read aloud accommodation
Designing Accessible Reading Assessments
Sample Used cont’d
Subgroup Far Below Basic
Below Basic
Basic Proficient Advanced Total Number of Students
No Disability 4% 11% 31% 30% 25% 30,000
Learning Disability—no
accommodation
32% 34% 25% 7% 2% 9,056
Learning Disability—IEP/5
04
35% 38% 23% 4% 1% 4,727
Learning Disability—IEP/504 & read aloud
26% 36% 30% 6% 1% 1,371
Percentage of Students at Each Proficiency Level
Designing Accessible Reading Assessments
Sample Used cont’dReference-Focal Comparisons
Reference Group Focal Group
No Disability (Group 0) Learning Disability—no accommodation (Group 20)
No Disability (Group 0) Learning Disability—IEP/504 (Group 21)
No Disability (Group 0) Learning Disability—IEP/504 & read aloud (Group 22)
Learning Disability—no accommodation (Group 20)
Learning Disability—IEP/504* (Group 21)
Learning Disability—no accommodation (Group 20)
Learning Disability—IEP/504 & read aloud (Group 22)
Note: IEP = Individualized Education Plan* = comparison did not show DIF so was not included in the DDF analyses
Designing Accessible Reading Assessments
Procedure
• Examine items that previously displayed DIF for DDF– DDF: when two groups that have been matched on
ability have different probabilities of selecting a distractor
• Standardized Distractor Analysis (SDA)
– Distinguishes between distractors
– Identifies uniformly and nonuniformly biased distractors
– An extension of standardized p-difference
Designing Accessible Reading Assessments
Procedure Used cont’d
• Equation used to test for DDF:
STD(i) =
• : negligible DDF
• : moderate DDF
• : large DDF
10.SDA05.
05.SDA
10.SDA
s
rsfss
W(i)P(i)PW
Designing Accessible Reading Assessments
Results
• 70% of the items that displayed DIF also displayed DDF
• 100% of DDF occurred with a comparison between the read aloud and some other group
• 64% of the distractors that displayed DDF were in favor of the read aloud group
Designing Accessible Reading Assessments
Option D Difference*(moderate DIF)
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Diffe
ren
ce in
Perc
en t
F-R
Designing Accessible Reading Assessments
Option B*: large DIF
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Dif
fere
nc
e in
Pe
rce
nt
F-R
Designing Accessible Reading Assessments
Option B Difference(no DDF)
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Diffe
ren
ce in
Perc
ent
F-R
Designing Accessible Reading Assessments
Option B: moderate DDF
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Dif
fere
nc
e in
Pe
rce
nt
F-R
Designing Accessible Reading Assessments
Option C Difference(large DDF)
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Diffe
ren
ce in
Perc
en
t
F-R
Designing Accessible Reading Assessments
Results cont’dComparison Groups
0-20 0-21 0-22 20-21 20-22 Ite
m
A B C D A B C D A B C D A B C D A B C D 3 *R
R
*R
R
10 + *F
F
+ *F
F 13 + + *F
F
- 25 ++ *F 32 *R
R
- 33 *F
F
+ 34 + *F 45 *R
R
56 *R 64 *R
R
- *R
R
- *R
R
Note: +: moderate DDF in favor of the focal group++: large DDF in favor of the focal group-: moderate DDF in favor of the reference group*R: DIF in favor of the reference group*F: DIF in favor of the focal groupShaded box: Items that did not exhibit DIF
Designing Accessible Reading Assessments
Results cont’d
• 17% that assessed reading standards showed DDF
• 9% that assessed writing standards showed DDF
• No observed pattern across content or cognitive area between groups
Designing Accessible Reading Assessments
Results cont’d
• Item that displayed large DDF was the most difficult item that displayed DIF
• One item displayed DDF in each of the distractors (two favoring the read aloud group and one favoring the non-LD group)
• Item that displayed DDF in two of its distractors was a spelling item– Both were homophones– Additional difficulty caused by read aloud
Designing Accessible Reading Assessments
Conclusions/Future Research
• Measurement dissimilarity between read aloud group and other groups
• Exploratory study: More research needed to determine whether read aloud actually alters test’s validity
• Matched on ability to provide more information at extremes
Recommended