Designing Math and ELA Tasks for AA-AAS Using ECD and UDL

This material is based on work supported by two Enhanced Assessment Grants from the U.S. Department of Education: Alternate Assessment Design—Mathematics and English Language Arts. The PADI online system was developed through a grant from the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. Department of Education or the National Science Foundation.

Designing Math and ELA Tasks for AA-AAS Using ECD and UDL

National Conference on Student AssessmentNational Conference on Student AssessmentJune 19, 2011June 19, 2011

Renée Cameto and Geneva Haertel – SRI InternationalWendy Carver – Utah State Office of EducationKaren Denbroeder – Florida Department of EducationCarol Scholtz – Idaho State Department of EducationDeborah Matthews – Kansas State Department of Education

Shawnee Wakeman – ModeratorEdward Roeber – Discussant

Introduction Based on work from two Enhanced

Assessment Grants funded by the U.S. Department of Education

Each project involved a consortium of states and SRI International: Alternate Assessment Design–Mathematics

Utah, Idaho, Florida Alternate Assessment Design–English

Language Arts Idaho, Utah, Kansas

2

Agenda ECD Framework Co-Design Process Example Design Patterns and Tasks Task Tryouts

Teacher training and materials Videos of task administration Results

Panel Discussion with State Representatives Q & A Discussant

3

Need Alternate Assessments need to be well

designed The design and development processes

employed need to be systematic, rigorous, and reflective of industry standards

NCLB/ESEA require that students with disabilities, including SWSCD, be assessed on grade-level academic content

Assessment continues to be a lever of educational change (assessment drives instruction and school reform)

4

Innovative Approach Evidence-centered design (ECD) is an

innovative assessment design process Initial work on ECD conducted at ETS by Mislevy,

Steinberg, and Almond (2003) Has been used for more than 15 years; in these EAG

projects, ECD is being extended to the population of students with significant cognitive disabilities

Is a framework and set of processes Meets criteria for developing well-designed

assessments Supports the design of items that are aligned to the

focal constructs of interest

5

Innovative Approach ECD can be applied to:

All subject areas All grade levels All types of assessments (large scale, summative,

formative, technology-enabled, pencil/paper, advanced placement, workforce, etc.)

All types of alternate assessments (portfolio, performance task, checklist, etc.)

All item/task formats (multiple choice, constructed response, performance task, etc.)

Integrating UDL into the ECD framework promotes accessibility of items through consideration of student needs and abilities during initial design and throughout the design process

6

What is Evidence-Centered Design?

Critical Question: How do we judge what students know and what they can do? Make explicit what knowledge or skills are the target Identify non-target but required knowledge and skills Generate the kinds of observations that provide

evidence of a student having the target knowledge Determine the kinds of stimuli and work products

needed to gather the evidence These attributes form the basis of the assessment

argument - an argument based on evidence

7

Co-Design Process ECD process makes

use of co-design at all layers

Co-design team typically includes: Special educators

(experience with SWSCD)

Content experts Assessment specialists State Department of

Education administrators

8

Alternate Assessment Design

A multistep process using ECD co-design

9

Select Standards

Common Core

Domain Analysis

Create Design Patterns

Domain Modeling

Create Summary Task

TemplatesConceptu

al Assessme

nt Framewo

rk

Author Assessment

Tasks

Assessment Implementati

on

Pilot and Refine Tasks

Assessment Delivery

Select StandardsGOAL: In Domain Analysis, content relevant to the assessment is organized and selectedIdentify standards to meet state AA-AAS needs, for example, by identifying commonalities among the consortium states’ extended standards and comparison standards (e.g., NCTM Standards and Expectations, Common Core State Standards)Develop list of standards common to all participating states

10

Select Standards

Common Core

Create Design PatternsGOAL: In Domain Modeling, a narrative description of the assessment argument structure (Design Pattern) is produced to guide task developmentDesign Patterns for selected standards are developedDesign Patterns are reusable and improve efficiency of task developmentDesign Patterns can improve content validityDesign Patterns create a design space for assessment developers

11

Create Design

Patterns

Elements of Design Patterns

Student Model What knowledge, skills or abilities

(KSAs) should be assessed?

What construct (complex of student attributes) should be assessed?

Focal KSAs

Primary Knowledge, Skills, and Abilities (KSAs) targeted by this DP

Additional KSAs

Other KSAs that may be required by tasks from this DP, some of which can be supported by UDL and accommodations

Educational Standards

Associations with Educational Standards from different states as well as national standards, if desired

12


Student Model What knowledge, skills or abilities

(KSAs) should be assessed?

What construct (complex of student attributes) should be assessed?

Focal KSAs Primary Knowledge, Skills, and Abilities (KSAs) targeted by this DP

Additional KSAs

Other KSAs that may be required by tasks from this DP, some of which can be supported by UDL and accommodations

Educational Standards

Associations with Educational Standards from different states as well as national standards, if desired

13


Evidence Model What behaviors or performances should

reveal the knowledge, skills, and abilities?

What behaviors should reveal the construct?

Potential Observations

Observed behaviors of students that can provide evidence of Focal KSAs

Potential Work Products

What students say, do, or make that provides evidence about the Focal KSAs

Potential Rubrics Some evaluation techniques that may apply

14


Evidence Model What behaviors or performances should

reveal the knowledge, skills, and abilities?

What behaviors should reveal the construct?

Potential Observations

Observed behaviors of students that can provide evidence of Focal KSAs

Potential Work Products

What students say, do, or make that provides evidence about the Focal KSAs

Potential Rubrics

Some evaluation techniques that may apply

15


Task Model What tasks, situations, or stimuli should

elicit those behaviors and performances?

What tasks should elicit those behaviors?

Characteristic Features

Aspects of assessment situations likely to evoke the desired evidence

Variable Features Aspects of assessment situations that can be varied in order to control difficulty or target emphasis on various aspects of KSAs

16

Create Design Patterns Task Model

What tasks, situations, or stimuli should elicit those behaviors and performances?

What tasks should elicit those behaviors?

Characteristic Features

Aspects of assessment situations likely to evoke the desired evidence

Variable Features

Aspects of assessment situations that can be varied in order to control difficulty, target emphasis, or provide support for Additional KSAs

17

Create Design Patterns Variable Features are the aspects of the

task that are varied to support the Additional KSAsUDL Category

Example Additional KSA

Example Variable Feature

Perceptual (Receptive)

Ability to perceive linguistic components of the stimulus material and question

Delivery mechanisms by which question is perceived (e.g., read aloud, concrete objects, Braille)

Skill and Fluency (Expressive)

Ability to communicate response (e.g., respond verbally, by using pictures, by making a selection from a group)

Response mode options (e.g., pointing, speech and verbalization, assistive device/augmentative communication)

Language and Symbols

Ability to decode text, symbols, or images

Level of abstraction required of student (e.g., concrete objects, images, text)

18

Create Design Patterns Variable Features are the aspects of the

task that are varied to support the Additional KSAs

UDL Category Example Additional KSA

Example Variable Feature

Cognitive Ability to process multistep problems

Provide graphic organizers

Executive Ability to plan and sequence

Prompts and scaffolds to estimate effort, resources, and difficulty

Affective Ability to engage (e.g., task-specific motivation)

Enhance relevance, value, and authenticity of tasks

19

Source: Center for Applied Special Technology (CAST), http://CAST.org

Author Assessment Tasks

GOAL: In Assessment Implementation, tasks are authoredInformation pre-populates from the Design Pattern to the Task TemplateTemplates are used to author a family of items aligned with a Focal KSA

Complexity is varied within the family of items

UDL is incorporatedItems individually administered

20

Author Assessment

Tasks


Step 1. Pre-populate Task Template with information from the Design Pattern Focal KSAs and Additional KSAs Potential Observations and Work Product Variable Features for Cognitive Background and UDL Characteristic Features

Step 2. Select attributes for authoring items Focal KSAs and Additional KSAs Potential Observations and Work Products Variable Features for Cognitive Background and UDL

21


Step 3. Select Variable Features to be used in the item family Indicate how Variable Features will be

applied (e.g., limit numbers to those with two or fewer digits, include multiple representations of stimulus materials)

Consider ways to vary complexity Consider ways to incorporate UDL

22


Step 4. Create Item DirectiveItem 1 Item 2 Item 3a/3b

Examiner shows student a picture of three pizzas and says, Here are three pictures of parts of a pizza. Examiner then presents student with a note card with the numeric fraction “¾” and says, Which picture shows three fourths of a pizza?

Examiner presents students with two photos, one with a whole pizza and the other with a half of a pizza and says, Here are two photos of pizzas. Examiner then presents student with a note card with the numeric fraction “½” and says, Show me which photo shows half of a pizza.

3a) Examiner presents to student an illustration of two unlabeled pizzas and says, Here are two photos of pizza, a whole pizza and part of a pizza. Show me the photo that is a part of a pizza.3b) If student does not respond or responds incorrectly, examiner removes all stimuli but the photo of part of a pizza and says, [Look at/touch] the photo of part of the pizza.23

Mathematics Item 1 Examiner shows student a picture of three pizzas

and says, Here are three pictures of parts of a pizza.

Examiner then presents student with a note card with the numeric fraction “¾” and says, Which picture shows three fourths of a pizza?

24

¾






Mathematics Item 2 Examiner presents student with two photos, one

with a whole pizza and the other with a half of a pizza and says, Here are two photos of pizzas.

Examiner then presents student with a note card with the numeric fraction “½” and says, Show me which photo shows half of a pizza.

26½






Mathematics Item 3a Examiner presents to student an illustration of

two unlabeled pizzas and says, Here are two photos of pizza, a whole pizza and part of a pizza. Show me the photo that is a part of a pizza.

28

Mathematics Item 3b If student does not respond to item 3a or

responds incorrectly, examiner removes the first illustration (of a whole pizza) and shows student only the second illustration. Examiner says, [Look at/touch] the photo of part of a pizza.

29


Step 5. Document correct answerItem 1 Item 2 Item 3a/3b

Student indicates the picture of ¾ of a pizza on the worksheet

Student indicates the picture of ½ of a pizza

3a) Student indicates the picture that is not a whole pizza

3b) Student looks at/touches the picture of half of pizza

30


Step 6. Describe stimulus materials

31

Item 1 Item 2 Item 3a/3bThree unlabeled drawings of pizzas presented in a row (these are bird’s eye views of the pizzas, not side views with perspective). Each pizza is divided into four slices: one has 2 of the four slices remaining, one has one of the four remaining, and the third has three of the four remaining. Every pizza shows the quarters outlined with a dotted line. Every pizza has four section outlines even if they are missing. Card with ¾ printed on it.

Two unlabeled photos of pizzas presented in a row (these are bird’s eye views of the pizzas, not side views with perspective). One photo is of a whole pizza and the other photo is of a half of a pizza. Card with ½ printed on it.

Two unlabeled photos of pizzas presented in a row (these are bird’s eye views of the pizzas, not side views with perspective). One photo is of a whole pizza, the second photo is of half of a pizza.


Step 7. Describe materials for the examiner

32

Item 1 Item 2 Item 3a/3b3 pictures of pizzas 1 card with numeric ¾Recording sheet for teacher to complete

2 photographs of pizzas

1 card with numeric ½

Recording sheet for teacher to complete

2 photographs of pizzas

Recording sheet for teacher to complete

ELA Item 1 Examiner says, This passage is from Roll of Thunder,

Hear My Cry. Examiner reads passage to student:

Cassie tells Mama that a mob is going to come after a boy. Papa runs outside with a gun. Soon, Mama sees that their cotton field is on fire. She thinks it started by lightning. Then Cassie goes back to her room and wonders if people can live in peace.Everyone goes out to the cotton field. Papa and Mama's neighbors are helping them put out the fire. The people in the mob are helping put out the fire too. Cassie thinks that her Papa started the fire.

33

ELA Item 1 Examiner lays out a card with the following

question and reads the question to the student: How did Papa starting the fire change the story? Examiner then lays out a card with each of the following sentences and points to each sentence as it is read.

1. The fire made the mob more angry and they came after the boy

2. The mob helped put out the fire and didn't hurt the boy

3. Cassie got scared of the fire and ran away from home

34

ELA Item 2 Examiner says, This passage is from Roll of

Thunder, Hear My Cry. Examiner reads passage to student:

A mob is going to come after a boy. His friends are warned that the mob is coming. One of the friends sets his own cotton field on fire to lead the mob away. All the neighbors and the mob come together to put out the fire. The boy is not hurt.

35

ELA Item 2 Examiner lays out a card with the following question

and reads the question to the student: What happened next after the friend set the cotton field on fire? Examiner then lays out a card with each of the following sentences and points to each sentence as it is read.

1.All the neighbors and the mob put out the fire2.The boy’s friends are warned that the mob is

coming3.A mob decides to come after the boy

36

ELA Item 3a Examiner places a picture card in front of student

and reads the first sentence from the passage: Cassie goes to the store.

Examiner places the second picture card in front of student and reads the second sentence from the passage: The storekeeper yells at Cassie.

Examiner places the third picture card in front of student and reads the third sentence from the passage: The storekeeper makes Cassie get out of the store.

37

ELA Item 3a Examiner removes all pictures and then presents

student with two pictures (Picture 1 and Picture 3) and says, Look at these pictures. Which picture shows what happened after the storekeeper yelled at Cassie?

ELA Item 3b If student cannot or does not respond, examiner

removes Picture 1 leaving only Picture 3 in front of student and says, [Look at/touch] the picture that shows what happened after the storekeeper yelled at Cassie.

38

Design Patterns Library of Design Patterns and Tasks

AAD-Mathematics and AAD-ELA projects, when completed, will have produced approximately 50 Design Patterns and Task families–200 items

Spans grades 3–8 and high school Math Design Patterns organized around NCTM

Standards and Expectations ELA Design Patterns organized around Common

Core State Standards Supports design of multiple-choice, scenario-

based, and portfolio assessment formats

39

Benefits of Using ECD with UDL

What states gain from applying this approach in assessment design: Content-valid assessment tasks designed to

align to state or national standards Increased efficiency through systematic task

design and development-reusable design patterns and task templates

Built-in documentation of design decisions using PADI (NSF and SRI development funding)

Variation in DOK and complexity Attention to UDL throughout design process

40

Pilot and Refine TasksGOAL: In Assessment Delivery, the newly developed assessment tasks are pilot tested and refined based on empirical resultsAll newly developed assessment tasks must be empirically studied to establish their feasibility, reliability, and validityMethodologies may include cognitive labs, teacher surveys, field tests of tasks, and student observations

41

Pilot and Refine Tasks

Math Task Try-outsResearch Questions: Task and Item Viability

Can the 3 items within the exemplar task be administered as designed?

Is the task clear to the student and to the teacher?

Appropriateness for a Range of Student Performance Levels Can any of the students do the most difficult item? Can most students do the least difficult item?

42

Math Task Try-outs Detailed Task Materials and Instructions Data Collection Booklet: 85 questions

Score for each item administered Item related information

Student received instruction related to the item Item characteristics – directions, graphics,

manipulatives and materials, language, complexity

Student information Communication level, disability, grade-level

Teacher information Years of experience, familiarity with academic

standards43

AAD-M Idaho - One State’s Experience with PADI & ECD

PADI and ECD Pluses!Thorough Stepwise Methodology – for AA ECO Based Item Development; based upon DOK, UDL fits with LALStakeholder Participation – Assessment, Curriculum & Ed SRI – Pioneering Methodology – AA Crosswalk with Multiple State ECOs (precursor to Common Core)Increased # Tools in Educators’ Toolbox for formative classroom assessmentIncreased Educator Awareness of AA and Ties to Instruction

44

Flow Chart

There are two paths: Student responds

correctly to A1 Student responds

incorrectly to A1

45

Administer

Item A1

Administer

Item A2

Stop administration

Stop administration

Correct response

Administer

Item B

Administer

Item C

Stop administration

Correct, incorrect, no response, or refused

Incorrect or no response

Correct, incorrect, no response, or refused

Note: Be sure to gain the student’s attention before presenting EACH item.

Refused

How to Administer Task Items

Sequence of Items within a Task: General Case When Item A1 is Correct

46

Sequence of Items within a Task: Special Case When Item A1 is Incorrect

47

Idaho AAD-M Task Try-out Incentives!

Stipend for Teachers

Assessment and Instructional Materials

Student Rewards Shopping &Budgeting withDouble DigitSubtraction

48

Math Task Try-outs Number of student participants: 192 Number of teacher participants: 55 Number of tasks administered by

state: 16 9 tasks/suites of items common across

all states 7 tasks/suites of items unique to each

state

49

Math Task Try-outs Fidelity of Implementation

Of the 1,547 item administrations:98% (1,514) items were administered as intended

2% (29) items were not administered as intended

50

Math Task Try-outs

51

Math Task Try-outs

52

Math Task Try-outs

53

Percent of Items Correct by Student Communication Level

Task Try-outs Preliminary Results

54


55


56

Idaho Task Tryout VideoVideo Demonstration Illustrates Sequence of Items Within a Task and Ties to Instruction.

57

8th Grade Student Mathematics Number and Operations A1

Idaho – AAD-M Next Steps

Ties to Instruction Outcomes Workshop and Learning Community 68 Educators 2 day hands-on Materials 4 live sites (split screen) Tanberg Video Teleconferencing Interactive Capacity to view:

Demonstration/Lecture Websites Documents (via doc camera) Presenters PowerPoint

58

Idaho AA Learning Community Lesson Plan & Resources

Alternate Assessment Learning Community AA Mathematics Lesson Plan Library

http://itcnew.idahotc.com/dnn/iaa/AADocuments.aspx

59

Task Refinement Individual tasks can be refined on the

basis of item level results from the try-outs: Reword prompts and answer choices Revise item format (e.g., multiple choice to

constructed response) Adjust stimulus materials Refine scoring rubrics Increase or decrease use of supports for

Additional KSAs Recommend changes in modes of

perception and expression used in tasks60

Summary ECD is well-suited to developing alternate

assessment tasks; supports integration of UDL ECD allows the systematic documentation of

assessment tasks to support efficiency of task development (re-usability)

ECD supports the design/development of a range of items with varying levels of complexity and DOK

The co-design approach actualizes the value of the special educators, content specialists and assessment specialists

61

Panel Questions1. Describe the co-design process in your

state.2. What were the benefits/challenges of using

evidence centered design/universal design for learning as the frameworks for task development?

3. How are the tasks being used in your state now?

4. What is the future of this approach in your state/consortium?

62

Contact Us http://alternateassessmentdesign.sri.com

[email protected] [email protected] [email protected] [email protected] [email protected] [email protected]

63