So. . . What Makes an Innovative Item an Innovative Item?

So. . . What Makes an Innovative Item an Innovative Item?

National Conference for Student Assessment

June 26, 2014

Innovation ExaminedInnovative ELP Standards – The State Perspective

Innovative vs Technology-enhanced Items

Innovative Measurement for Innovative Standards

Discussion, Q & A

Questions to Consider

•What constructs might require innovative items?

• Is it possible to have innovative items without technology enhancements (TE)?

•How do innovative items differ from TE items?

•Do we need innovative measurement models?

Innovative ELP Standards –The State PerspectiveMargaret Ho

ELPA21 Former Executive Board Chair & English Language Proficiency Assessment (WELPA) Coordinator, Washington Office of Superintendent of Public Instruction

Seminal pieces for ELP standards redefined

When English language learners (ELLs) use language effectively as they progress toward independent participation in grade-appropriate activities to attain college and career readiness, they can…

1construct meaning from oral presentations and literary and informational text through grade-appropriate listening, reading, and viewing

2participate in grade-appropriate oral and written exchanges of information, ideas, and analyses, responding to peer, audience, or reader comments and questions

3 speak and write about grade-appropriate complex literary and informational texts and topics

4 construct grade-appropriate oral and written claims and support them with reasoning and evidence

5 conduct research and evaluate and communicate findings to answer questions or solve problems

6 analyze and critique the arguments of others orally and in writing

7 adapt language choices to purpose, task, and audience when speaking and writing

8 determine the meaning of words and phrases in oral presentations and literary and informational text

9 create clear and coherent grade-appropriate speech and text

10 make accurate use of standard English to communicate in grade-appropriate speech and writing

ELPA21 Consortium States

Collaboration with Partners

Oregon Dept. of Education

CRESSTUniversity of California

Los Angeles

National Center on Educational Outcomes (NCEO) University of

Minnesota

Council of Chief State School Officers (CCSSO)

Understanding Language Initiative

(Stanford University)

What is an innovative test item?

Emilie Pooler

Assessment Specialist, Education Testing Service

in·no·va·tiveˈinəˌvātiv/

Adjective

1. (of a product, item, idea, etc.) featuring new methods;

advanced and original.

“Innovative test item” - About 13,700 results

How have others described“Innovative” items?

•“Innovative items contain content or functionality that is not possible in a text-based, multiple-choice question (item). Wendt & Harmes, 1999

•Innovative item types, defined broadly, are those items in a CBT that make use of features and functions of the computer to do things not easily done in traditional paper-and-pencil assessments.

Parshall & Harmes, 2007

•“Innovative items—computer-delivered items with interactivity that makes them unlike traditional, paper-and-pencil-delivered items…” Parshall, Davey, and Pashley,

2000

•Innovative Item Types Require Innovative Analysis Thompson, N. A., Ro, S., Smith, L., & Santos, J,

2009

•“…innovative item types include new elements that are not fully addressed in the existing item writing guidelines” and “As assessments become more innovative, more elaborate scoring methods might be needed.”

Parshall & Harmes, 2007

Types of innovative items identified in the literature

•Audio Items

•Hot Spot Items

•Technology enhanced items

•Simulated environments

•Situated tasks

•Respond to dynamic visual stimuli

•Multiple-choice with new media distractors

•Yes/No with justification

•Categorizing

•Drag and connect

•Drag and drop

•Select and classify

•Ranking and sequencing

•Assembling proof

•Insert text

•Correction and substitutions

•Graphical modeling

•Problem-solving vignette

•Matrix completion

•Sorting

•Concept map

•Open-ended multiple choice

•Bug/fault/error correction

•Multiple answer

•Alternate choice

•Observe and report (speaking)

•Interactive presentation (listening)

•Follow instructions (listening)

Characteristics of innovative items

“Innovative” items…

• Show great promise for improving the quality, validity, and reliability of state large-scale assessments

• Elicit positive reactions from students

• Are more engaging to students

• Bring everyday technology into assessment

• Can test students to a greater depth of knowledge

• Align more closely with curricular approaches

• Allow for more advanced applications of universal design and accessibility

• Can be adaptive within an item

• Are computer-delivered items with interactivity

Traditional Item

Fixed:

Read

or

Listen

Stem/Prompt(fixed: text, graphic, or

sound)

Response options

(fixed: text, graphic, or

sound)

Fixed:

Write,

Speak,

or Select

option(s)

Item Format Response ActionPresentation

Under-

standing

of what

language

students

know

Innovative items = Technology enhanced

items.

Innovative Measurement for Innovative StandardsPhoebe Winter

ELPA21 Item Acquisition and Development Team Lead

Language is Interactive

ELD Standards c. 2005

•Listening: Understand basic structures, expressions, and vocabulary such as school environment, greetings, questions, and directions

•Speaking: Express and defend a point of view using precise language and appropriate detail

•Reading: Identify and use knowledge of synonyms, antonyms, homophones, and homographs to understand text

•Writing: Use the writing process to write brief narratives and stories with a few standard grammatical forms

•Some interactive: Write brief responses to selected literature with factual understanding of the text using simple sentences

Standards Measured by ELPA21

•Participate in grade-appropriate oral and written exchanges of information, ideas, and analyses, responding to peer, audience, or reader comments and questions.• Participate in conversations, discussions, and written exchanges about a variety of topics, texts, and issues; building on the ideas of others and expressing his or her own; asking and answering relevant questions; adding relevant information and evidence; and paraphrasing the key ideas expressed.

•Construct meaning from oral presentations and literary and informational text through grade-appropriate listening, reading, and viewing.•Use an increasing range of strategies to determine two or more central ideas or themes in oral presentations or written text, and explain how they are supported by specific textual details, and summarize a simple text.

Sample Item 1

Grade 4-5 Listening

Listen to a student giving a presentation about an author.

I did my presentation on my favorite author, Laura Ingalls Wilder. Laura Ingalls Wilder is from a state in the USA called Wisconsin.You can see Wisconsin on this map.

Laura Ingalls Wilder was born here in 1867. I made a timeline to show you some important events in her life.

Laura Ingalls Wilder wasn’t always a writer. She became a teacher after she finished school.

She started a successful farm where she raised animals and grew apples.

Her first book was about her childhood in the woods of Wisconsin.

Her first book was a great success. She became famous and wrote more popular books about her life in the countryside. Together, the books are called the Little House series. They are still well known.

You said Laura Ingalls Wilder started a farm and then started writing. What inspired her to start writing?

Ms. Wilder was inspired by her daughter. After she started her farm, her daughter encouraged her to start writing.

Are all of Ms. Wilder’s books about her childhood?

I read that Ms. Wilder’s ideas came from her childhood in the woods and her life as a farmer.

A. To point out where an author was born

B. To discuss why authors enjoy traveling

C. To show the best area to build a farm

D. To explain why settings are important for stories

Why does the student use a map in her presentation?

Now, answer the questions:

Move the four events to the correct places on the timeline.






A. She was encouraged by her daughter.

B. She loved writing essays in school.

C. She was friends with many writers.

D. She wanted to teach writing to others.

Why did Laura Ingalls Wilder start writing?


In grade appropriate contexts…

1. construct meaning from oral presentations and literary and informational text through grade-appropriate listening, reading, and viewing.

5. conduct research and evaluate and communicate findings to answer questions or solve problems.

8. determine the meaning of words and phrases in oral presentations and literary and informational text.

Sample Item 2

Kindergarten Listening

Hi. My name is Jason. I need your help. Can you help me find some things in my bedroom?

Click on the robot.

Click on the robot.

Click on the books.

Click on the books.

Click on the kite.

Click on the kite.

Click on something you can wear.

Click on something you can wear.

Click on something with wheels.

Click on something with wheels.

Thank you! Now, help me put some things away.


Put the shoes under the bed.

Put the shoes under the bed.


Put the jeans in the closet.







Put the lamp on the table.





Now help me choose a pillow. Choose the pillow with stripes. Put it on the bed.



ELP Standard 1

Listening Subclaim: The English language learner constructs meaning from oral presentations of literary and informational text. The learner can:

Level 1 Level 2 Level 3 Level 4 Level 5with prompting and support (including context and visual aids), use a very limited set of strategies to: identify a

few key words in read-aloudsand oral presentations of information or stories

with prompting and support (including context and visual aids), use an emerging set of strategies to: identify some

key words and phrases in read-aloudsand oral presentations

with prompting and support (including context and visual aids), use a developing set of strategies to: identify main

topics ask and answer

questions about key details in read-alouds and oral presentations

with prompting and support (including context and visual aids), use an increasing range of strategies to: identify main

topics answer

questions about key details or parts of stories

retell events from read-alouds, picture books, and oral presentations

with prompting and support (including context and visual aids), use a wide range of strategies to: identify main

topics answer questions

about key details retell familiar

stories from read-alouds, picture books, and oral presentations

ELP Standard 8

Listening Subclaim: The English language learner determines the meaning of words and phrases in oral presentations. The learner can:

Level 1 Level 2 Level 3 Level 4 Level 5with prompting and support (including context and visual aids), recognize the

meaning of a few frequently occurring words in simple oral presentations and read-aloudsabout familiar topics, experiences, or events.

with prompting and support (including context and visual aids), recognize the

meaning of some frequently occurring words and phrases in simple oral presentations and read-aloudsabout familiar topics, experiences, or events.

with prompting and support (including context and visual aids), answer questions

to help determine the meaning of some words and phrases in simple oral presentations and read-aloudsabout familiar topics, experiences, or events.

with prompting and support (including context and visual aids), answer and

sometimes ask questions about the meaning of words and phrases in simple oral presentations and read-aloudsabout a variety of topics, experiences, or events.

with prompting and support (including context and visual aids), answer and ask

questions about the meaning of words and phrases in simple oral presentations and read-aloudsabout a variety of topics, experiences, or events.

Sample Item 3

Grade1 Speaking

Today, we are going to see what happens when we put a white flower in a cup of colored water. Now watch.You need these materials: a bowl of water, purple food coloring, and a white flowerFirst, put the food coloring in the water.Second, put the flower in the bowl.Third, wait overnight.Look, the white flower is now purple!Today, you learned how to make a colored flower. Tell your friend what the teacher did.

Please use the tool below to record your response.



You may now play back,re-record, or submit your answer.

ELP Standard 3

Speaking Subclaim: The English language learner speaks about complex literary and informational texts and topics. The learner can:

Level 1 Level 2 Level 3 Level 4 Level 5 communicate

simple information or feelings about familiar topics or experiences

communicate simple messages about familiar topics, experiences, events, or objects in the environment

deliver short simple oral presentations about familiar topics, stories, experiences, or events

using simple sentences and drawings or illustrations,

deliver short simple oral presentations about a variety of texts, topics, experiences, or events

including a few descriptive details,

deliver oral presentations about a variety of texts, topics, experiences, or events

Standard 9

Speaking Subclaim: The English language learner creates clear and coherent speech. The learner can:Level 1 Level 2 Level 3 Level 4 Level 5

[Standard introduced at Level 2.]

with support (including visual aids and modeled sentences),

retell an event

present simple information with emerging control of some frequently occurring linking words

with support (including modeled sentences),

retell (in speech) a simple sequence of events in the correct order

present simple information with developing control of some frequently occurring linking words (e.g., and, so) and temporal words (e.g., first, then)

recount two or three events in sequence

present simple information about a topic with increasingly independent control of some temporal words (e.g., next, after),and some frequently occurring linking words (and, so)

recount a more complex sequence of events in the correct order

introduce a topic

provide some facts about a topic using temporal words to signal event order and using frequently occurring conjunctions (linking words or phrases)

Sample Item 4

Grade 6-8 Reading

Page 1

Assignment: Build a HovercraftIn this assignment, you will construct a model hovercraft and then use a balloon inflated with different amounts of air to measure the duration traveled by the model hovercraft.

A hovercraft boat can be seen in the picture above skimming along the water. Notice the boat’s “skirt,” the large black tube that encircles it. The skirt traps air as large fans create enormous pressure under the boat, which in turn creates an air cushion between the boat and the water. The air cushion reduces friction between the boat and the water, facilitating movement along the water. Hovercraft travel best over calm water, such as still lakes, or even smooth ground, such as paved roads. They cannot travel quickly over rough, choppy water or over rocky, uneven ground.

BackgroundA full-sized hovercraft is a vehicle that glides above the water or the ground on a layer of air called an air cushion. This air cushion is created when air is forced out of the bottom of the vehicle at extremely high pressure.

Read the Assignment and the Student Report below. Then answer questions X – Y.


Question: How does the amount of air cushioning a hovercraft affect how long it can travel?

A complete list of materials and procedures are provided below.

Model Hovercraft ExperimentIn this experiment you will create a model hovercraft by gluing an old drinking bottle pop-up top to a CD or DVD and then attaching a balloon to force air out the bottom, which creates an air cushion.

Materials1. Model hovercraft2. Balloon3. Stopwatch4. Large flat surface for testing the hovercraft5. Notebook

Page 2



Procedures1. Completely blow up the balloon, twisting the end tightly to prevent air from leaking. Then pull the end of the balloon over the closed pop-up top. The picture below illustrates what this should look like:

Page 3

2. Place the model hovercraft on a flat surface and open the pop-up top to release the air. The air will flow through the lid and the hole in the center of the CD or DVD, creating an air cushion.



3. Nudge the edge of the hovercraft so that it begins to travel across the surface of the table. Using a stopwatch, measure the amount of time the hovercraft floats before it ceases moving completely.4. Follow steps 1-3 two additional times (for a total of three trials). Record the time for each trial on the chart.

Page 4

Student Report



5. For the next set of trials, blow up the balloon only about halfway and repeat steps 3 and 4.6. For the last set of trials, fill the balloon with only a small amount of air and repeat steps 3 and 4.

Page 5

Student Report



7. Sum up the times of the three trials for each size balloon and divide each total by three to get the average time of travel for each amount of air in the balloon.8. Compare the average time for each amount of air in the balloon and answer the question: How does the amount of air cushioning a hovercraft affect the length of time it can travel?

Page 6

Student Report




Page 7

Student Report

DescriptionWhen the balloon was filled with a large amount of air, it traveled for a longer time on average than when it was filled with a medium amount of air. Also, when the balloon was filled with a medium amount of air, it traveled longer on average than when it was filled with a small amount of air. As the amount of air in the balloon decreased, the balloon traveled for a shorter amount of time.


Question:What is the main purpose of the assignment?


Student Report

A. To build a model hovercraft that can travel for more than 30 seconds

B. To find out the length of time a model hovercraft will travel using different amounts of air

C. To decide what is the best surface for a model hovercraft to travel over

D. To learn the distance a hovercraft travels when pushed with different amounts of force


Question: The experiment discusses an air cushion. What does an air cushion do?


Student Report

A. Allows a vehicle to travel with less friction

B. Gives people on a vehicle a soft place to sit

C. Stops a person from getting hurt in an accident

D. Moves a vehicle forward with force


Select the part of the picture that shows where the air cushion is.








Select the part of the diagram of the model hovercraft that shows where the air cushion is.








Hovercraft probably do not travel well over rough or rocky ground because:

A.Hovercraft do not have wheels for steering

B.Hovercraft engines are not powerful enough

C.The hard skirt would break against rocks

D.The skirt would not retain a good air cushion


Look at the word “trial” in Step 2 of the Procedures. Choose the phrase that is closest in meaning to how the word “trial” is used in the passage.

A.Going to a court to get a decision

B.Trying something difficult

C.Repeating part of a process

D.Finding out if something works

3. Nudge the edge of the hovercraft so that it begins to travel across the surface of the table. Using a stopwatch, measure the amount of time the hovercraft floats before it ceases moving completely.4. Follow steps 1-3 two additional times (for a total of three trials). Record the time for each trial on the chart.

Student Report


Look at the results chart. Four cells in the chart are highlighted. Select the cell that shows the longest time that a hovercraft with a large balloon traveled.






Select the bar graph that accurately represents the information in the results chart.

Student Report A.

B.

C.

Innovation in Measurement Models

•We have:

•Innovative standards

•Innovative items addressing those standards

•Do we need

•Innovative measurement models?

www.elpa21.org

Margaret HoWashington [email protected]\

Emilie PoolerAssessment [email protected]

Phoebe WinterIAD Team [email protected]

The contents of this presentation were developed under a grant from the U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education and you should not assume endorsement by the Federal government.

mailto:[email protected]/

mailto:[email protected]

mailto:[email protected]

Documents

So. . . What Makes an Innovative Item an Innovative Item?