Technology Assisted Reading Assessment Exploring Assistive Technology Issues in Reading Assessment ATIA 2010 Orlando: Jason Altman, NCEO

Technology Assisted Reading Assessment

Exploring Assistive Technology Issues in Reading

Assessment

ATIA 2010 Orlando:

Jason Altman, NCEO


Statement of Problem• Little evidence that technology improves

validity of tests for students w/ VI• One reason is that students’ experience

and exposure to AT varies• TARA seeks to develop a test of student

proficiency with AT • For use prior to using AT on large-scale

reading assessments


Preliminary Studies: The State of AT Use with VI

Students

• Study #1: TVI survey (presented at ATIA ’09)

• Study #2: TVI follow-up interview

• Study #3: Student Observational Interview


Revisiting Teacher Survey

• AT use is a large part of instruction for students with VI

• TVIs take a “blended” approach to teaching reading (e.g., using a variety of modalities)

• Standardization of assessments may be a challenge because students use different technology for different purposes


More on Teacher Survey

• Reading and technology are intertwined activities for students with VI

• The larger the caseload, the less likely students will have individualized AT experiences

• Teachers who focus on foundational reading skills tend to provide fewer technology opportunities for students.


Overview: Results of Teacher Interview Study (#2)

• 1st study showed “accommodations” or “modifications” are common for students w/ VI

• Both high- and low-tech approaches to teaching reading are used by TVIs– Most likely for different purposes and at different

times– Such flexible approaches to reading and

situational use of technologies create challenges for standardization of assessments


Method• Interviewed 27 TVIs• At least two TVIs were interviewed in the

following categories:– TVIs who work in Schools for the Blind– TVIs who are itinerant teachers– TVIs who teach students with low-vision– TVIs who teach students who are completely

blind– TVIs who do not teach braille– TVIs who teach braille


Instrument and Procedures• All interviews were conducted by telephone• In most cases, there were two (and up to

four) researchers present during telephone interviews

• Research participants were asked to answer a series of structured interview questions

• Interviews generally lasted 30 minutes, although some interviews lasted as long as one hour


Results• All 27 TVIs indicated that they would use a

technology-based assessment if one were readily available

• All TVIs who taught students with low-vision used at least some form of magnification equipment

• TVIs who taught students who use braille readers also reported using a variety of technologies such as audio devices and braille note takers


Specific Products• Alpha Smart• Bookport• Braille and Speak• DAISY Readers• Extreme Reader• Freedom Box• IPod• JAWS• Kurzweil• MAGIC• OpenBook• OutSpoken• PacMate• Zoomtext


Technology and reading• Some TVIs used technology fluidly in instruction• Those who had wide-ranging knowledge of the

tools available (and had access to such tools) match instructional tasks with technology tools.

• 5 TVIs spent most of their time teaching students how to use CCTVs and were hesitant to engage in teaching students using computer-based tools (especially those that added voicing to files)

• In all cases, however, TVIs used some form of technology to assist teaching


Formal AT assessment protocols

• About 1 of 4 teachers were using formal assessment protocols/tools to assess students’ assistive technology use

• Among those most frequently used are the SETT Protocol and the Texas School for the Blind Assistive Technology Assessment Summary for Students with Visual Impairments

• One teacher who used the Texas assessment also mentioned using the Georgia Project for Assistive Technology’s (n.d.) Assistive Technology Evaluation and the Wisconsin Assistive Technology Initiatives (2004) WATI Assistive Technology Checklist


Assessment: Proficient Reader Who Uses AT

• Teachers recognized that “advanced proficiency” in technology-assisted reading meant the ability to move fluidly between technologies in order to optimize the potential of various technologies

• A proficient user of technology-assisted reading would be a reader who read and understood grade level material– Most TVIs were concerned that their students be

assessed on grade-level, but with a fair opportunity to demonstrate grade-level skills

– TVIs were concerned about the validity of tests


Discussion

• There is a desire for a formal assessment or evaluation of student skills

• Should discriminate between various levels of proficiency

• There are a small number of assessments/frameworks in use


Activity

TVIs in this room: Had you been participants would have findings changed


Student Observational Interview: Study #3


Method: Sample• Students with visual impairments in grades 6-

10• In total, we interviewed 18 students for this

study from 5 states across the US• We sampled students from both general

education school systems (n=9) and state schools (n=5) for the blind

• Four additional students were educated at state schools through general education classes in a nearby public school


Method: Instrument and Procedures

• Students participated in “observational interviews” facilitated by three researchers on the project– Observational interviews were a hybrid between

verbal interviews, where respondents describe phenomenon (Bogdan & Biklen, 1998) and cognitive interviews, where interviewees participate in an activity and describe their thoughts and actions (Ericsson & Simon, 1993)


Procedures (cont.)• During these interviews, students were asked

to complete reading tasks using AT– Afterward, students were asked how they

use AT in the reading process, including how to download files, retrieve information from printed material, and to explain preferences

• Each interview lasted between 30 and 60 minutes


Results of Student Study• Demands of a rigorous high school

curriculum at times left students without enough time to explore and become proficient in new and possibly more efficient technology strategies

• For students with visual impairments reading large print or Braille, the time it takes to complete a task is often far greater than that for their peers with full functioning vision


Results• 11 of the 18 students could read large

print – 3 of them regular print as well, and many were audio book users.

• The remaining 7 students all read braille – and 3 of the large print readers also read braille.

• 13 students used audio books to access print no matter what their primary method of print reading was


Results• 8 students used technology for reading

Braille – 5 used Braille Note• For magnification, students used a variety of

technologies, from simple handheld magnifiers to computer-based products

• Zoomtext (most often also with speech), CCTV

• Technology use for 7 depended on the reading situation


Results

• Technologies are sometimes unreliable (e.g., not all formats work for internet-based text), take time to learn, and do not always accompany the student beyond the school walls (e.g., many students cannot bring AT devices home to assist with homework)


Student characteristics and independence

• Students varied from completely relying on help from sighted teachers, para-professionals, and peers to being assertively independent

• Most students described their dependence on others as situational

• One student mentioned that he is not afraid to ask his friends or family to read things for him

• Another student, said that help is not needed


Technology Choices• Depended on availability of technology in student

location• For example, students at a state school typically

would have more access to technology than a student in a rural public school that did not serve a large population of students with visual impairments

• Access at home may differ from access at school– One student, who was on the academic honor roll at

school, often stayed after school and came in before school every morning to finish homework


Choice of specific technology• The student’s TVI or para-professional

generally was very involved in the decision• Some students attended technology

conferences• It is important to note that another person

(teacher, parent, or para-professional) needed to learn the technology along with the student

• TVIs had to be careful not to over-extend themselves with multiple new technologies


Role of expected change in situation

• Decisions were impacted by the nature of the visual impairments

• Changes in student vision and forecasts for future change often called for a change in products– For example, one student used a CCTV up

until 6th grade but stopped using it because there was a decrease in his field of vision and it was no longer practical


Difficulty with AT• The amount of time that must be dedicated to

training and practice in using the equipment sometimes outweighed the potential benefit of the technology.

• Another complication was consistent access to technology

• Students were sometimes forced to share technology with several other students, including CCTVs


Other Hangups• There was also difficulty in transporting, setting

up, and stowing equipment• Upon learning a particular technology platform,

students were hesitant to move to other platforms

• Equipment malfunction, or general hang-ups and glitches also were sometimes troublesome for students, teachers, and paraprofessionals alike.

• One student stated that the Web sites that contained many graphics can be most problematic


Testing and Large-Scale Assessment

• Multiple students reported using enlarged text to access the items and then having the paraprofessional fill in the bubble answer sheet

• At least one student was able to take a math test using a computer; however, he was required to take the reading test using a paper and pencil version

• Other students used standard question and answer forms, accessing them via CCTVs


Discussion• What do these findings mean for TVIs?

• Does the situation play out similar in your classroom?

• Would you agree that your student’s participation in large-scale assessment could be more meaningful?


Conclusions• As statewide assessment policy is still in

flux (Thurlow et al., 2006), it is important for research to continue in the area of technology-assisted reading to provide stakeholders with a clearer picture of the avenues that students with visual impairments take to access reading anddemonstrate their reading skills


Also at this Conference• Also see today from 2:45-3:45

Bonaire 8 EDU-15

Development of an Accountability Test to Measure a Student’s Ability to Access Text with AT

Presenting: Laitusis, Hall


Next Steps• Field Testing the new Technology

Based Reading Assessment in school classrooms

• We need your help!– Administrators of the field test– Students who can participate


Thank you! For more information…National Center on Educational OutcomesUniversity of Minnesota612-626-1530 http://www.nceo.info

Jason Altman [email protected]

35

Documents

Technology Assisted Reading Assessment Exploring Assistive Technology Issues in Reading Assessment ATIA 2010 Orlando: Jason Altman, NCEO