Designinga Pool of Items

Workshop Flow

• The construct of MKT– Gain familiarity with the construct of MKT– Examine available MKT instruments in the field

• Assessment Design– Gain familiarity with the Evidence-Centered Design approach– Begin to design a framework for your own assessment

• Assessment Development– Begin to create your own assessment items in line with your

framework• Assessment Validation

– Learn basic tools for how to refine and validate an assessment • Plan next steps for using assessments

Domain Modeling (Design Pattern)

(Define Test

Specs)

Dom

ain

Ana

lysi

s

Define item

Template

Define item

Specs

Develop Pool of items

Collect/ Analyze ValidityData

Refineitems

Refineitems

Assemble Test

Document Technical

Info

Assessment Development Process

From Design to Development

• Use information from the Design Pattern to generate items• (Use the process of generating items to refine the Design

Pattern)

What the test-taker should know

• Focal Knowledge, Skills, and Abilities

Evidence we can collect to show they know it

• Potential observations• Potential work products• Potential rubrics

Kinds of situations that can evoke this evidence

• Characteristic features• Variable features

Item Development Steps

Based on the foundational design work…1. Content experts develop draft items

2. Refine draft items – “hygiene”

3. Collect validity data on draft items

4. Use validity data to refine items

5. Assemble and document the assessment

Assessment Item Anatomy

Essential parts of a butterfly are…A. Wings, legs, teethB. Eyes, wings, legsC. Nose, knees, earsD. Fingers, frontal lobes, hair

Butterflies are members of the _______ phylum.

What are the closest relatives of the butterfly, and why?

Stem

Choices

Distracters

Prompt

Constructed Response (Closed-Ended)

Constructed Response (Open-Ended)

Multiple Choice

Conducting Hygiene Review

is essential• Clarity and precision• Accuracy• Grammar• No ambiguity• No unintended cues• Distracters believable, appropriate, same length• Placement of correct response varied

Not a Trivial Process!SimCalc

Other Resources forHunters and Gatherers

• Standardized tests– International

• PISA (http://nces.ed.gov/surveys/pisa/ )• TIMSS (http://timss.bc.edu/ )

– National• NAEP (http://nces.ed.gov/nationsreportcard/ )

– State• TAKS released items

Other Resources forHunters and Gatherers

• Other resources– Established curricular materials

• Approved textbooks• TPD materials• Supplemental teacher resources

– SRI’s online resources• Performance Assessment Links in Math (PALM): http://palm.sri.com• Integrated Performance Assessments in Technology: http://ipat.sri.com • Online Evaluation Resource Library (OERL): http://oerl.sri.com • GLOBE assessment tools: http://globeassessment.sri.com

– Research literature (e.g., theoretical papers that provide example items)– Other research-based assessments– Mental Measurements Yearbook from the Buros Institute of Mental

Measurements

Welcome to(mini) Item Camp!

Goals

• Develop 1 to 3 draft items for your (or your neighbor’s) assessment

• Have an experience of participating in an item camp

Starting with Item Templates

• What are templates?– Templates are industry standard in assessment development– Templates outline the structure of an item, you fill in the content– Guidelines to systematically produce high-quality items

• What are these templates?– Created based on the SimCalc design pattern to help seed

thinking about the SimCalc items– Templates might be quite different for your design pattern– They are based on our observations about the structure of items

that fit into types and “worked” in field testing– Heuristic tool, not theoretical framework

• In this context, recommended but not required

Template Types Based onDomain Analysis of MKT

Unconventional forms or representations Choosing problems and examples that can

illustrate key curricular ideas Differentiating between colloquial and

mathematical uses of language Linking precise aspects of representations Understanding implications of models and

representations Evaluating mathematical statements

Structure of the Templates

All multiple choice, with the following elements:– Context

Sets up a teaching task Presents information necessary to answer the question

– Question/Prompt– Distracters

Review Templates

Guidelines for distracters

• Usually 4 or 5 distracters (may be more)• May use “Choose one” or “Choose all that apply.”• Responses may include “All of the above” and “None of the

above.”• Whenever possible, distracters will reflect common errors or

misunderstandings, naïve pre-conceptions, or other misconceptions.

• Teachers should not be able to rule out a distracter or identify the answer simply because of superficial or trivial characteristics, syntactic complexity, or concept complexity.

• Distracters will not be partially correct responses nor will they be designed to “trick” teachers in responding incorrectly.

Bejudicious!

Other Item Types

• True/false• Matching• Open-ended

–Calculations–Short responses–Generating representations–Generating explanations, stories, etc.

Require a rubric for scoring

Filling Out the Item Form

• Name• Connection to standard or curriculum• KSA assessed• Item• Correct answer and/or rubric• Notes

“Item Hygiene”

• Consider this a brainstorm, so do not be overly concerned about IH

• However, keep in mind that this is the next part of the pipeline

• IH includes:– Clarity, precision,

correctness– Grammar– Non-ambiguity– No unintended cues– Distracters believable,

appropriate, same length– Placement of correct

response varied

Draw on Resources Available

• Your Design Pattern• Your curriculum materials• MKT resources (e.g., prior assessments)• NCTM and Texas standards

Activity #3Mini-Item Camp

• Find Activity #3 in Tab 5• Work on your own, with a partner, or with a small group• Develop some MKT items (1 to 3)• Prepare 1 or 2 items for the Item Hygiene Review (see page 7)• On the item form, include:

– Name– Connection to standard or curriculum– KSA assessed– Item– Correct answer and/or rubric– Notes

• Time for Activity:9:30-10:15

• On the item form, include:–Name–Connection to standard or curriculum–KSA assessed–Item–Correct answer and/or rubric–Notes

Activity #4

• Find Activity #4 on Tab 5• Sit in groups of 3 for Hygiene Review• Use the Item Hygiene Guide to edit your teammates items• Fix your 1 or 2 items and prepare a clean copy for this

afternoon’s activity• Discussion to follow

– Show-and-tell of 2 or 3 items– Your insights, questions, challenges

Activity #4Conduct Item Hygiene

Review• Conduct item hygiene on your set of items,

using the Item Hygiene Guide. We suggest writing corrections directly on the item.

• Following, we will discuss– Insights about development of assessment

items– Questions and challenges

Some Useful Resources

• Anderson, J. R. (2000). Cognitive psychology and its implications (3rd Ed.). New York: W. H. Freeman and Company.

• Baxter, G. P., & Glaser, R. (1998). The cognitive complexity of science performance assessments. Educational Measurement: Issues and Practice, 17, 3, 37-45.

• Pellegrino, J., Chudowsky, N., Glaser, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.

• Quellmalz, E., Hinojosa, T., & Rosenquist, A. (2001). Design of student assessment tools for the Global Learning and Observations to Benefit the Environment (GLOBE) program. Presentation at the annual GLOBE International Conference, Blaine, WA.

• Snow, R. E., Federico, P. A., & Montague, W. E. (Eds.). (1980). Aptitude, learning, and instruction: Volume 2. Cognitive process analyses of learning and problem-solving. Hillsdale, NJ: Erlbaum.