Embed Size (px)
Citation preview
Item Accessibility for the Next Generation of Assessments:
Lessons Learned From Development of AA-MAS
Sue Bechard NCSA 2011
Computer adaptive scaffolding strategies
Montana investigated the use of interactive computer-based assessments for reading and mathematics tailored to the unique characteristics of AA-MAS eligible students.
NCSA 2011 2
NCSA 2011 3
Research Conducted to Describe Eligible Students and Determine
Test Adaptations □ Literature review □ IEP analysis (N=200) □ Exploratory cognitive interviews (N=21) □ Teacher interviews/focus groups □ Beta test (N=145) □ Follow-up cognitive interviews during beta (N=17) □ General test performance analysis □ Pilot Test (N=1354)
NCSA 2011 4
Teachers’ Descriptions of Students’ Characteristics (Draft Eligibility Guidelines)
They Need Additional Instructional Supports: o individualized curriculum and instruction, differentiated content and
expectations. o information organized in smaller “chunks”, directions given step-by-step. o guidance to sort out irrelevant or less important information. o concrete learning aids, such as demonstrations, visuals, graphic organizers,
manipulatives, calculators, or fact sheets. o less dense text and vocabulary simplification due to poor reading skills.
They Experience Cognitive Challenges: o abstract, inferential, or symbolic thinking. o self-monitoring and self-evaluation. o transferring what is learned from one situation to another. o maintaining attention. o grade level vocabulary, especially words with multiple meanings and
technical terms. o receptive and/or expressive language skills. o short-term or working memory capacities.
NCSA 2011 5
MT GSEG Assessment Approach • Computer-based and interactive • Original CRT MC items with 4 answer options • All students see original items in same order • Student receives feedback with scaffolded item
after incorrect response to original item • One scaffolded item is administered per original
item • Student has one opportunity to correct a mistake • Assesses reading and mathematics for grades 7
and 8. □ Beta test: Spring 2009 (145 students, 49 items, 1 session) □ Pilot test: Winter 2010 (1354 students, 142 items, 3 sessions)
NCSA 2011 6
Scaffolding
A dynamic assessment strategy that includes structural assistance introduced to organize information or guide responses embedded in the presentation of the item or task.
The assistance is expected to affect the knowledge, skills, and abilities (KSAs) required to respond to a task, but are not intended to change the construct being measured.
Scaffolding Example
Reading Inference
Which statement best summarizes paragraph 5?
Travelers’ stories aided mapmakers. Natural forces aided mapmakers.
New discoveries aided mapmakers. Knowledge of winds aided mapmakers.
Submit Answer 8 NCSA 2011
Original question
Which statement best summarizes paragraph 5?
Here Be Dragons Natalie M. Rosinsky
An old story from Norway says that the explorer Floki used ravens to find the distant land of Iceland. Floki sailed in an open boat, without a map. After many days at sea, Floki freed one bird at a time from the cage. The first bird flew back the way Floki had come. He knew that wasn’t the way to Iceland. Finally, some of the birds flew off ahead of the ship. Floki followed them and found Iceland.
9 NCSA 2011
When you summarize, you try to include all the important points. As you read this paragraph, try to focus on all of the details instead of noticing just one.
Early mapmakers used the sun and the stars to find the location of a faraway place. Was it in the east, where the sun rises? Or in the west where the sun sets? Sailors also knew a lot about the winds. In their stories of discovery, they might tell of winds that blew from the north, pushing the ships toward an island in the south. They wrote of winds from the east that brought them to a distant shore.
When you finish reading this part of the passage, click on the green arrow that says “Go To Question.” 10 NCSA 2011
Scaffolded question
Which statement best summarizes paragraph 5?
Travelers’ stories aided mapmakers. Natural forces aided mapmakers.
New discoveries aided mapmakers. Knowledge of winds aided mapmakers.
Submit Answer 11 NCSA 2011
NCSA 2011 12
Frequency of Scaffolding Strategies: Reading
NCSA 2011 13
Frequency of Scaffolding Strategies: Mathematics
NCSA 2011 14
Pilot Test Each student was assigned to one pilot test. All students took the Spring 2010 general CRT.
□ Students with IEPs (IEP): selected by school level IEP teams, using Draft Eligibility Guidelines.
□ Students without IEPs (GEN): randomly selected from across the state to reflect same proportions of student performance comparable to Students with IEPs.
NCSA 2011 15
Pilot Test Student Sample
NCSA 2011 16
Four Lessons Learned from Pilot Test: Research Questions
1. What do students and test administrators say about the pilot test?
2. Do students do better with scaffolding? 3. Do eligible students with IEPs do better on
the 2% pilot test the than students without IEPs?
4. Given the scaffolding strategies used on the 2% pilot test for each content area, which strategies work?
NCSA 2011 17
1. What did the students say about the pilot test?
(N=1263/1354, 93%) □ They prefer to take tests on computer. □ They thought the pilot test was easier
than other tests they take. □ The scaffolding helped them answer the
questions. □ They really liked or really did not like the
read-aloud feature, but most liked.
NCSA 2011 18
1. What did the test administrators say about the pilot test?
19/29 Test Administrators (66%) observed 472/1354 students (35%) • Students preferred taking test by
computer vs. paper. • Students seemed more engaged. • Students had different reactions to the
read-aloud feature. • The scaffolding seemed to motivate the
students to try harder to answer questions.
NCSA 2011 19
2. Do students do better with scaffolding? Math 8 = YES
NCSA 2011 20
3. Do eligible students with IEPs do better on the 2% pilot test the than students without IEPs?
Reading 7 = NO
NCSA 2011 21
4. Given the scaffolding strategies used on the 2% pilot test for each content area, which strategies work?
ALMOST ALL
Effectiveness of Scaffolding Strategies by Items per Test □ Math 7 82% of items □ Reading 7 90% of items □ Math 8 90% of items
NCSA 2011 22
Observations and Recommendations Teacher Advisory Panel (TAP)
• Tighten the Eligibility Guidelines: severity of disability, intensity of services, prior CRT performances, and best match of test to student characteristics.
• Benefits of the CRT-M: • students have greater control resulting in higher
self-esteem and increased motivation, • presentation of one problem at a time, and • easier administration of read-aloud.
• Attention to opportunity to learn by teachers has increased due to project.
NCSA 2011 23
Technical Advisory Committee (TAC) What type of technical evidence is needed to
pursue this type of assessment operationally & to report results?
The CRT-MAS is a better design for the eligible students.
The project resulted in learning that is transferrable to the general test.
Continue to investigate the instructional approaches used to help the students.
Keep collecting data on effects of scaffolding. There is an opportunity for teachers to change
instruction based on scaffolding strategies used.
NCSA 2011 24
TAC: What technical evidence should be gathered to validate how the items are scored
operationally?
Options for how items may be scored: A) Report the original item score, and show how the
score changes once students receive the scaffold. Take the original score and judge that against the grade level
Performance Level Descriptors (PLDs), Show how scaffolding scores relate to Modified PLDs. The report could show how far students are from grade level
expectations.
B) Take the correct items, scaffolded or not, and determine where you would put the scores based on CRT PLDs. Allow full credit for a scaffolded item answered correctly.
Implications for Policy, Feasibility, and Implementation for Montana
1. Involved a tremendous amount of work and commitment by Montana educators--considering the size of the student population, participation was high.
2. While the online format was positively received: a. The items were tied to the software with which they were
developed making them difficult and expensive to share and distribute. Interoperability standards for any online items should be up front in the development.
b. For Montana, a state with limited resources, the costs of developing the online assessment are prohibitive. A consortium appears to be a feasible answer.
NCSA 2011 25
NCSA 2011 26
Summary of Lessons Learned from MT GSEG
There appears to be a group of students with disabilities whose characteristics have implications for item and test design.
All students benefitted from effective scaffolding strategies across the performance spectrum.
A computer-based interactive assessment is a model that engages and motivates students.
Read-aloud needs to be studied further. Options available for scoring and reporting could improve
information for instruction. What works for eligible students can be applied to general
test design. Online assessments need to be developed using
interoperability standards. Given the resources required, this type of test is best
developed by consortia.
