Guidelines for Accessible A ssessment P roject: Research R esults

Guidelines for Accessible Assessment Project: Research Results Laurene Christensen, Ph.D., Research Associate, National Center on Educational Outcomes

Vitaliy Shyyan, Ph.D., Research Associate, National Center on Educational Outcomes

Project Goal

• To develop research-based guidelines for the representation of Common Core State Standards assessment content in audio and sign

Methodology• Cognitive labs with elementary, middle, and high school students

• Students selected based on the following criteria:• Print disabilities• Low vision• English language learners• Deaf, hard of hearing, and ASL users

• Participating students were from states in PARCC, Smarter Balanced, and unaffiliated states

• Students took a short math assessment with isomorph pairs

• Research teams observed the student during the assessment and asked follow-up questions

Research Instruments

• Cognitive lab protocols• Demographic/login information• Audio/sign item sets • Post-test questions

• Computer-based test items

Demographic/Login Information

• School #

• Student #

• Grade

• Need (deaf/hard of hearing; reading-based learning disability; English language learner; low vision)

• Student gender

• Test login ID

• Note taker

• Date

Audio and Sign Item Sets

Grades 3-5 Grades 6-8 Grades 9-12

Audio 12 items (6 sets)

16 items (8 sets)

16 items (8 sets)

Sign 12 items (6 sets)

12 items (6 sets)

9 items (4 sets)

Post-test QuestionsAudio Sign

• When there were answer options, the read aloud said, “answer” before each answer option (like “Answer A, Answer B”). Was it clear what the answer options in each question were? Did you like having the word “answer” read before the answer option, or would you prefer “Answer Option A, Answer Option B,” or just “A, B,” or nothing at all?

• For how many years have you received a read aloud accommodation during testing?

• Do you use a read aloud accommodation in the classroom (outside of testing)? Does someone read aloud to you at home? Do you use read aloud computer programs?

• Do you believe that using a read aloud accommodation helps you understand test questions better?

• What did you think about the voice of the read aloud we used today (quality, speed, etc.)?

• For how many years have you communicated in ASL?

• Are members of your family deaf or hard of hearing?

• At what age was the onset of your hearing loss?• Do you view ASL videos in the classroom (outside

of testing)?• When taking this practice test, did you usually

read the item first or view the ASL video first?• Do you believe that viewing ASL videos helps you

understand test questions better?

Grade 3-5 Audio: Set 1, Item 1

For the first question, the places of each house were read aloud.

Grade 3-5 Audio: Set 1, Item 2

For the second question, places of each classroom were not read aloud.

Grade 3-5 Audio: Set 2, Item 1

For the first question, the student told when the student dropped a soybean in the field and how many fields were planted.

Grade 3-5 Audio: Set 2, Item 2

For the second question, the student was not told anything when the student dropped a corn kernel on a field.

Grade 3-5 Sign: Set 3, Item 1

In the first question, the equations were not signed.

Grade 3-5 Sign: Set 3, Item 2

In the second question, the equations were signed.

Grade 3-5 Sign: Set 4, Item 1

In the first question, the signer used the ASL signs for right angle, symmetry, and length.

Grade 3-5 Sign: Set 4, Item 2

In the second question, the signer used ASL signs and finger-spelled those terms.

Grade 6-8 Audio: Set 4, Item 1

For the first question, each room was highlighted as it was read aloud.

Grade 6-8 Audio: Set 4, Item 2

In the second question, the room names were only read aloud when the student clicked on them.

Grade 6-8 Sign: Set 5, Item 1

In the first question, the signer used the ASL sign for estimates.

Grade 6-8 Sign: Set 5, Item 2

In the second question, the signer used ASL signs and fingerspells estimates.

Grade 9-12 Audio: Set 7, Item 1

For the first question, the picture was described.

Grade 9-12 Audio: Set 7, Item 2

For the second question, the picture was not described.

Grade 9-12 Sign: Set 3, Item 1

In the first question, the signer presented the question twice, once at the beginning of the item and again after the money information.

Grade 9-12 Sign: Set 3, Item 2

In the second question, the signer presented the information in the order it appears in English.

Student Demographics: Gender

52 (56%)40 (44%)

Gender

MaleFemale

Student Demographics: Accommodation

46 (50%)46 (50%)

Accommodation

AudioSign

Student Demographics: Grade

Elementary School Middle School High School

3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th

9 12 7 8 10 13 11 9 8 5

28 31 33

Student Demographics: Need

Elementary School Middle School High School

Need Number Percentage Number Percentage Number Percentage

Low Vision 3 11% 4 13% 10 30%

Deaf/HH 16 57% 14 45% 16 48%

LD-Reading 5 18% 10 32% 7 21%

ELL 4 14% 3 10% 0 --

Student Preferences: Elementary AudioItem

Set Item Description Need First Item Second Item No Preference

1 Coordinate Grids

Low Vision3 0 0

100% -- --

LD-Reading2 3 0

40% 60%

ELL3 0 1

75% -- 25%

2 Drag and Drop

Low Vision3 0 0

100% -- --

LD-Reading3 2 0

60% 40% --

ELL2 2 0

50% 50% --

3 Equations

Low Vision3 0 0

100% -- --

LD-Reading2 3 0

40% 60% --

ELL1 1 2

25% 25% 50%

Student Preferences: Elementary AudioItem

Set Item Description Need First Item Second Item No Preference

4 Diagram Description

Low Vision2 1 0

67% 33% --

LD-Reading2 2 1

40% 40% 20%

ELL3 1 0

75% 25% --

5 Answer Options with Numerals

Low Vision3 0 0

100% -- --

LD-Reading2 3 0

40% 60% --

ELL3 1 0

75% 25% --

6 Large Numbers

Low Vision3 0 0

100% -- --

LD-Reading3 2 0

60% 40% --

ELL4 0 0

100% -- --

Student Preferences: Middle School AudioItem

Set Item Description Need First Item Second Item No Preference

1 Line Graphs

Low Vision3 1 0

75% 25% --

LD-Reading8 0 2

80% -- 20%

ELL2 1 0

67% 33% --

2 Drag and Drop

Low Vision1 3 0

25% 75% --

LD-Reading2 4 4

20% 40% 40%

ELL0 3 0-- 100% --

3 Equations

Low Vision4 0 0

100% -- --

LD-Reading6 2 2

60% 20% 20%

ELL3 0 0

100% -- --

Student Preferences: Middle School AudioItem

Set Item Description Need First Item Second Item No Preference

4 Diagram Description

Low Vision3 1 0

75% 25% --

LD-Reading4 3 3

40% 30% 30%

ELL2 0 1

67% -- 33%

5 Fractional Ratios

Low Vision3 1 0

75% 25% --

LD-Reading8 2 0

80% 20% --

ELL2 1 0

67% 33% --

6 Answer Options with Numerals

Low Vision4 0 0

100% -- --

LD-Reading8 1 1

80% 10% 10%

ELL3 0 0

100% -- --

Student Preferences: Middle School Audio

Item Set Item Description Need First Item Second Item No Preference

7 Large Numbers

Low Vision3 0 1

75% -- 25%

LD-Reading 6 2 075% 25% --

ELL2 0 0

100% -- --

8 Decimals

Low Vision 3 1 075% 25% --

LD-Reading 4 3 057% 43% --

ELL2 0 0

100% -- --

Student Preferences: High School Audio

Item Set Item Description Need First Item Second Item No Preference

1 DecimalsLow Vision

8 2 0

80% 20% --

LD-Reading3 3 1

43% 43% 14%

2 EquationsLow Vision

9 1 0

90% 10% --

LD-Reading6 1 0

86% 14% --

3 Fractional RatiosLow Vision

8 0 280% -- 20%

LD-Reading7 0 0

100% -- --

Student Preferences: High School Audio

Item Set Item Description Need First Item Second Item No Preference

4 Large NumbersLow Vision

10 0 0

100% -- --

LD-Reading7 0 0

100% -- --

5 Drag and DropLow Vision

3 6 1

30% 60% 10%

LD-Reading4 3 0

57% 43% --

6 Line GraphsLow Vision

6 0 186% -- 14%

LD-Reading4 3 0

57% 43% --

Student Preferences: High School Audio

Item Set Item Description Need First Item Second Item No Preference

7 Political CartoonsLow Vision

5 1 0

83% 17% --

LD-Reading 4 2 067% 33% --

8 MapsLow Vision 5 1 0

83% 17% --

LD-Reading 2 3 040% 60% --

Insights from Students: Audio

• It’s easier to break down a problem when you are listening. (205)

• I liked how it actually told me what the equation was. (317)

• The read aloud accommodation is helpful sometimes because sometimes I can’t really focus on what I’m reading. (210)

• Yes, [helpful], it tells you the question, and you can better think about the answer and what’s the problem. … [equations read aloud] You know better the problem – you can think about it faster if the computer tells you the problem. (112)

• [when shapes are described] You don’t have to think that hard. It helped me answer better. (106)

• It’s easier to break numbers down when you are listening and taking notes. (205)

• [usefulness of read aloud] Sometimes there is a word, and I’m not sure what it is, so it helps when I am told what the word is. (216)

• Helpful. Sometimes I can’t read stuff correctly or don’t understand by myself. If they read for me, it’s easy to know. (213)

• Preferred the one with the numbers read – it is unnatural not to have the numbers read. … Yes, a whole lot [helpful] because I really don’t struggle on words, it tells me what it is, it helps me better when somebody reads to me. (209)

• I like it when I read in my head. I get more focused. Another voice is distracting. (303)

Insights from Students: Audio Quality

• Yes, I liked the voice, it was a little slow. (201)

• The voice is clearer than some I heard before. (109)

• It sounded like a robot. (303)

• It sounded pretty much like Siri from my phone. A bit too fast. (102)

Student Preferences: Elementary Sign

Item Set Item Description First Item Second Item No

Preference

1 Coordinate Grids/Use of SpaceNumber 8 8 0

Percent 50% 50% --

2 Drag and DropNumber 5 9 2

Percent 31% 56% 13%

3 EquationsNumber 2 10 2

Percent 14% 71% 14%

4 Finger SpellingNumber 8 6 1

Percent 53% 40% 7%

5 Plurality and RepetitionNumber 7 7 0

Percent 50% 50% --

6 Signing Answer OptionsNumber 6 5 3

Percent 43% 36% 21%

Student Preferences: Middle School Sign

Item Set Item Description First Item Second Item No

Preference

1 Coordinate Grids/Diamond Presentation

Number 5 8 0

Percent 38% 62% --

2 Drag and Drop/Presentation of Number Line

Number 7 6 0

Percent 54% 46% --

3 EquationsNumber 3 8 1

Percent 25% 67% 8%

4 Floor Diagram/Point of ViewNumber 4 7 1

Percent 33.3% 58.3% 8.3%

5 Finger SpellingNumber 5 8 0

Percent 38% 62% --

Student Preferences: High School Sign

Item Set Item Description First Item Second

ItemThird Item

No Preference

1 Finger SpellingNumber 0 8 8 0

Percent -- 50% 50% --

2 EquationsNumber 12 2 0

Percent 86% 14% --

3 Diamond PresentationNumber 11 0 2

Percent 85% -- 15%

4 Political CartoonsNumber 13 2 0

Percent 87% 13% --

Insights from Students:Sign

• Some of the English words I don’t know so I can watch the sign. (262)

• I liked the one with the picture describe. In deaf culture we do describe photos. (353)

• Watching videos is helpful – if I don’t understand something, I can get more information. (362)

• I can understand questions better when they are signed, not finger-spelled. (156)

• ASL is our language. English is not bad but ASL is my language. … When you read the English, it is hard to understand. But the signs help. (167)

• We don’t have laptops. But if I had a test on a computer – I’m not sure about videos. You have to learn English. (365)

Questions and Comments