27
Designing Accessible Reading Assessments A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis ETS

A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

  • Upload
    ashton

  • View
    21

  • Download
    0

Embed Size (px)

DESCRIPTION

A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis ETS. Branden Hart Teresa King* Skip Livingston Pavan Pillarisetti Kitty Sheehan Elizabeth Stone* Klaus Zechner. ETS Contributors. Linda Cook* Kelly Bruce Jennifer Dean Dan Eignor - PowerPoint PPT Presentation

Citation preview

Page 1: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

A Multistage Adaptive and Accessible Reading Assessment for

Accountability

Cara Cahalan LaitusisETS

Page 2: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

ETS Contributors

• Linda Cook*

• Kelly Bruce

• Jennifer Dean

• Dan Eignor

• Lois Frankel

• Gena Gourley

• Eric Hansen

• Branden Hart• Teresa King*• Skip Livingston• Pavan Pillarisetti• Kitty Sheehan• Elizabeth Stone*• Klaus Zechner

Page 3: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

DARA Goal 4

• Field test a multi-stage component-based reading assessment.– Reduce number of students performing at

“chance level” – Allow students to show what they know– Push instructional to include both

comprehension and reading fluently for students with reading-based LD

Page 4: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

DARA Test Design

Page 5: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Accessibility Elements• Students with disabilities included in pilot test• “Higher” interest passages selected based on

student ratings• Single column question format (increased white

space and reduced wrapping of text)• Included “context” sentence• Panel of disability experts reviewed items and

made suggested revisions (simplified language)

Page 6: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Page 7: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Page 8: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Page 9: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Page 10: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Data Collection Design

Page 11: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Primary Research Questions• For accountability purposes, is it possible to

combine scores from the two different routes on the component test (i.e., average scores from Test 1 and Test 2)?

• Is the Component test more accessible than the state assessment

– Do RLD students do better on the Component test than the state assessment while students without disabilities (NLD) perform similarly on both assessments?

Page 12: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Other Research Questions• Can we reduce the number of students scoring at

chance level?• Can we use automated scoring technology

(SpeechRater) to score oral reading fluency measure?

• Can we accurately route students based on 7, 14, 21, and 28 items?

• What is the best measure of oral reading fluency? • How do we combine fluency and comprehension

test scores (50/50, 25/75, 75/25)?

Page 13: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Sample

• 8th Grade Students

• 26 Middle Schools

• 294 RLD (final sample=275)

• 194 LP (not include in this presentation)

• 500 Non-Disabled (final sample=486)

Page 14: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Description of Sample by NLD/RLD

• Race, Gender, and cut score impact

Group

SEX RACE

% M % F % A.I. % A % B % H % M % P.I. % W

NLD 1 46.18 53.82 0.00 2.82 2.42 8.47 2.02 3.63 80.65

NLD 2 46.84 53.16 0.42 2.97 4.24 8.05 2.97 3.81 77.54

RLD 1 63.12 36.88 0.00 1.42 4.96 22.70 6.38 2.84 61.70

RLD 2 62.69 37.31 0.75 0.75 5.22 8.96 5.22 4.48 74.63

Page 15: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Page 16: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Page 17: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Test Score Summaries: Route 1

NMean(Std Dev)

Criterion(48 items)

Component Comprehension(42 items)

Component Fluency(obs max=222.75)

Component Total: Scaled(max=48)

RLD (Route 1) 14114.92(3.96)

14119.14(6.16)

14171.44(32.37)

14120.61(5.60)

NLD (Route 1) 24936.34 (7.95)

24933.71(6.12)

249145.49(30.18)

24937.34(6.27)

Page 18: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Test Score Summaries: Route 2

NMean(Std Dev)

Criterion(48 items)

Component Comprehension(42 items)

Component Total:Scaled(max=48)

RLD (Route 2) 13427.21(6.01)

13422.66(7.43)

13425.90(8.49)

NLD (Route 2) 23734.29(8.05)

23730.65(7.41)

23735.03(8.47)

Page 19: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Primary Research Questions• For accountability purposes, is it possible to

combine scores from the two different routes on the component test (i.e., average scores from Test 1 and Test 2)? YES

• Is the Component test more accessible than the state assessment

– Do RLD students do better on the Component test than the state assessment while students without disabilities (NLD) perform similarly on both assessments? YES, for Route 1

Page 20: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Can we reduce the number of students scoring at chance level?

Page 21: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Routing decision

 

Best passage

(2)

Passage 1

Passages 1, 2

Passages 1, 2, 3

Full routing

(8 items; Route 1:

<= 3)

(8 items; Route 1:

<= 3)

(16 items; Route 1:

<= 6)

(24 items; Route 1: <= 10)

(32 items; Route 1: <=

13)

Reliability 0.63 0.60 0.74 0.77 0.79

% (N) students assigned to Route 1

40.73% (112)

41.45% (114)

34.55% (95)

43.64% (120)

51.27% (141)

% (N) students assigned to Route 1 on this test

and full routing test

70.21% (99)

72.34% (102)

65.96% (93)

82.98% (117)

100% (141)

Can we accurately route students based on 7, 14, 21, and 28 items?

Page 22: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Fluency TestHuman vs. Automated Scoring

ALL

Passage

1

Passage

2

Passage

3

Passage

4

N 547 126 151 148 122

Pearson r 0.68 0.81 0.71 0.60 0.76

Can we use automated scoring technology (SpeechRater) to score oral reading fluency measure?

Page 23: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Future Questions for Study and Policy

Q: What is the best measure of oral reading fluency?

• Corrected words per minute in 1st minute• Words per minute, corrected words per minute, percent correct, rating

Q: How do we combine comprehension and fluency scores

• 25% fluency + 75% comprehension• 50/50, 75/25

Page 24: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Contact information

Cara Cahalan Laitusis

Senior Research Scientist

Educational Testing Service

Mailstop 09R

Princeton, NJ 08541

[email protected]

Page 25: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Extra Slides

Page 26: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Test Score Correlations: Route 1

  CriterionComponent

ComprehensionComponent

FluencyComponent

Total (Scaled)

Criterion 1.00 0.80 0.55 0.83

Component Comprehension

0.30 1.00 0.46 0.97

Component Fluency

0.27 -0.02 1.00 0.67

Component Total (Scaled)

0.38 0.94 0.31 1.00

NLD

RLD

Page 27: A Multistage Adaptive and Accessible Reading Assessment for Accountability Cara Cahalan Laitusis

Designing Accessible Reading Assessments

Test Score Correlations: Route 2

  CriterionComponent Total

(Scaled)

Criterion

1.00 0.82

Component Total (Scaled)

0.76 1.00

NLD

RLD