Modifying Achievement Test Items: A Theory-Guided & Data-Based Approach

Modifying Achievement Test Items:Modifying Achievement Test Items:A Theory-Guided & Data-Based ApproachA Theory-Guided & Data-Based Approach

Stephen N. Elliott Learning Sciences Institute

andDepartment of Special Education

Vanderbilt University

CCSSO 2009 National Conference on Student Assessment

CCSSO NSA Conference / Elliott 2009

2

Projects & Projects & PartnersPartners

• CAAVES: Consortium for Alternate Assessment Validity and Experimental Studies – USDE funded; 2006-2009 – Partners: AZ, HI, ID, & IN + Vanderbilt Measurement

Group + Discovery Education Assessment

• CMAADI: Consortium for Modified Alternate Assessment Development and Implementation– USDE funded; 2007-2010– Arizona Dept. of Education– Indiana Dept. of Education

• Visit Websites for Resources Discussed Today

– http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml– http://peabody.vanderbilt.edu/LSI_Projects/C-MAADI_Project_Home.xml– http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home/TAMI_Project.xml


3

Key TermsKey Terms• AccessAccess: the opportunity for test-takers to demonstrate

proficiency on the target construct of a test or a test item. In essence, complete access is manifest when a test-taker is able show the degree to which he/she knows the tested content. Access, therefore, must be understood as an interaction between individual test-taker characteristics and features of the test itself.

• AccommodationAccommodation: widely recognized in state testing guidelines as individualized changes to the setting, scheduling, presentation format, or response format of an assessment.

• ModificationModification: alterations or adjustments of test items to facilitate access for virtually all test takers. Appropriate modifications ….– Remove extraneous material, – Maintain the same depth of knowledge (DOK),– Do NOT change the grade-level construct being measured, and– Increase the validity of the inference from the test score.


4

Evolving Modification Evolving Modification ParadigmParadigm

Step 1. Evaluate original item accessibility.

Step 2. Reduce sources of construct-irrelevant variance in items.

Step 3. Document changes to items.

Step 4. Pilot test with student cognitive labs & post-assessment focus groups.

Step 5. Field test with large sample of students.

Step 6. Conduct psychometric & related analyses.


5

• Goal #2 of the CAAVES Project is to “investigate feasibility of item modification strategies for future alternate assessments.”

• This goal was accomplished by:(a) developing a common set of test items

from existing reading and mathematics tests using modification principles that facilitate reading access and valid responses and

(b) using a computer-based delivery system to experimentally examine student preferences, score comparability, and item statistics of the modified items for students with and without disabilities.

CAAVES Project Goal #2CAAVES Project Goal #2


6

Motivating QuestionsMotivating Questions

We wanted to answer the following questions

about item modifications:

1. Will modifications in testing conditions change the skill (construct) being measured?

2. Will taking the test under modified conditions change the resulting scores?

3. Will non-disabled examinees benefit if allowed the same modifications?


7

Procedures for Addressing Procedures for Addressing QuestionsQuestions

We completed the following…..

• Modified a common set of existing reading and math items to create items designed to be more accessible and still measure the same grade-level content as the original items.

• Conducted a cognitive lab study with a small sample of students with and without disabilities to gain their insights into which item modifications are preferred and most likely to improve test access for students whose disability involves reading difficulties.

• Conducted a cross-state experimental study to compare the effects of tests with and without modified items on students’ test performances and test score comparability.

• Conducted post-assessment survey of all students concerning their perceptions of item types and cognitive ease.


8

Participation Criteria forParticipation Criteria for Students with Disabilities Students with Disabilities


9

Anatomy of an ItemAnatomy of an Item


10

10

Item Summary Reports: An Item Summary Reports: An ExampleExample


11

Overview of ResultsOverview of Results

Elliott, et al. (in press)Exceptional Children


12

Tools to Analyze Items & Guide Tools to Analyze Items & Guide ModificationsModifications

The Test Accessibility and Modification Inventory (TAMI; 2008) and

TAMI Accessibility Rating Matrix (ARM; 2009) were developed asdecision-making tools to facilitate the analysis of new and existingtests and test items with the purpose of enhancing their accessibility.


13

TAMI Item Accessibility TAMI Item Accessibility RatingsRatings


14

TAMI Item Accessibility TAMI Item Accessibility RatingsRatings


15

TAMI Item Accessibility TAMI Item Accessibility AuditsAudits

IN CMAADI Item Review,2009


16

Method for Documenting Method for Documenting OTLOTL

AZ Cog LabStudy, 2008


17

Cognitive LabsCognitive Labs

Excerpted from Kettler, Elliott, & Beddow, in press Peabody Journal of Education


18

Post-Assessment Focus Post-Assessment Focus GroupsGroups

AZ CMAADI Pilot Study, 2009


19

Theory-Guided & Data-Theory-Guided & Data-Based ModificationsBased Modifications


20

Review:Review: Evolving Modification Evolving Modification ParadigmParadigm

Step 1. Evaluate original item accessibility.

Step 2. Reduce sources of construct-irrelevant variance in items.

Step 3. Document changes to items.

Step 4. Pilot test with student cognitive labs & post-assessment focus groups.

Step 5. Field test with large sample of students.

Step 6. Conduct psychometric & related analyses.


21

Thanks!Thanks!

• Thank you very much for your time and joining us for this session.

• Please provide follow-up questions and suggestions in writing to:

[email protected]@vanderbilt.edu

http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml

Documents

Modifying Achievement Test Items: A Theory-Guided & Data-Based Approach