Question DevelopmentHow do I create good multiple choice items?

Agenda

Black BoxWhy we are here

Black Box

Back to the Classroom

Black Box

What happens inside…

Black Box

Does Formative Assessment raise standards?

(I prefer “achievement” to standards.)

1. enhances communication between student and teacher.

2. actively involves students

3. yields results that are then used to adapt teaching

4. affects the motivation and self-esteem of students

Black Box

Is there room for improvement in the way we assess?

Issue #1 cultivate understanding, not rote learning; work

collaboratively with other teachers; emphasize quality, not

quantity.

Issue #2 emphasize learning, not grading; encourage each student’s

personal improvement; convey your belief that students can

learn

Issue #3 analyze student work to discern their learning needs; make

feedback meaningful

Black BoxHow can we improve formative

assessment?

Feedback to students should be about the particular qualities of their work.

Students should be trained in self-assessment so that they can understand the main purposes of their learning and what they need to do to achieve.

Effective teaching includes opportunities for students to express their understanding.

Black BoxSo…if I am in the black box, how do

I respond?1. I learn from professional development.

2. I start in low gear and patiently find my own way of incorporating new learning.

3. While remaining aware of external tests, I take leading responsibility for assessing ongoing work to monitor and improve progress.

4. Continually reflect on my work.

Research

Black & Wiliam Research Questions:1. Does better formative assessment

equal higher test scores?2. Does formative assessment need

improving?3. If so, what improvement is needed?

Research FindingsStudy Standard Deviation GainsBloom (1984) 1.0 to 2.0Black and William (1998) .5 to 1.0Meisels, et al. (2003) .7 to 1.5Rodriguez (2004) .5 to 1..8Harrie and Timperley (2007) .5 to 1.0

Why are we here?

What’s coming with Common Core Curriculum?

EOCT vs. GHSGTContinuous Improvement

Our Element

The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.

EQ? How can well designed assessments evaluate student performance on the GPS and provide information about student strengths and weaknesses?

Determine and define vocabulary Identify key terms within the

standard(s)/element(s). Define each term. In your group:

▪ Underline key terms in the element.▪ Define key terms

The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.

Study the standard(s)/element(s)Generate key questions related to

the standard(s) and/or element(s).Answer each question.

The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.

Identify strong and weak workCollect and rate strong and weak

samples of work related to the standard(s) and/or element(s). List examples of strong and weak work Use information on handout to help

The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.

Scaffold understanding of the standard(s) and/or element(s)Paraphrase the standard(s) and/or

element(s).

The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.

Develop “I Can” statementsDescribe the standard(s) and/or

element(s) as statements of intended learning (e.g., I can use information from what I read to draw conclusions (make inferences), I can use mathematical vocabulary to describe how I solved a problem, etc.).

The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.

Depth of Knowledge (DOK) Jigsaw Activity

1. Count off at your table 1, 2, 3, 4, 1, 2, …

2. All 1’s meet; All 2’s meet; All 3’s meet; All 4’s meet

3. In “Expert Groups” read and discuss everything about your DOK Level

4. Return to your “Home Group”5. Experts report out to the “Home Group”

Common Verbs

Describe Explain IdentifyIllustrate Trace CompareContrast Predict ApplySort Justify AnalyzeEvaluate Summarize DefineList Differentiate DistinguishDefend Conclude Categorize

Rules are made for breaking?Basic rules for creating multiple

choice itemsReduce systematic errors Increase reliability of test items

Basically – design the questions to determine what the student really knows or does not know… Not whether or not the student has “test

taking skills”

“Misguided” Test Taking TipsGuess if you are not sureLook for answer choices that

grammatically complete the question If one answer is much longer than

the others – it is most likely the correct answer

If “All of the Above” is provided and you are reasonably sure of two answer choices , “All of the Above” is most likely correct.

Multiple Choice Items

Who was president during World War II?A. Franklin RooseveltB. George WashingtonC. William ClintonD. Ronald Reagan

Item StemKey

Distracters

Rule #1

Make the problem clear to the student

Poor Example:1. Antigens

A. Attack memory cellsB. Attack killer T Cells

C. Attack pathogens D. Attack helper T cells

Rule #1

Better Example:

1. Which of these do antigens attack?

A. PathogensB. Helper T cellsC. Killer T cellsD. Memory cells

Rule #2 State the item stem in the positive

whenever possible.Poor Example:1. Which of these is not a reason why President

Roosevelt decided not to involve the United States in World War II until the bombing of Pearl Harbor?

A. The US was upset with Great Britain over the sinking of merchant ships.B. The US was involved in another war at the time.C. The US was already allied with Germany at the time of Pearl Harbor.D. The US had not been attacked directly until Pearl Harbor.

Rule #2Better Example1. Which of these best describes why the

United States became involved militarily in World War II?

A. The US was directly attacked.B. Great Britain asked for assistance

against German submarines.

C. French resistance needed military arms after Germany invaded.

D. The US was obligated to become involved by its membership in NATO

Rule #3

Make sure the item stem does not give away the correct answer.

Poor Example:1. What led to the formation of the States'

Rights Party? A. The level of federal taxation B. The demand of states for the right to

make their own laws C. The industrialization of the South D. The corruption of federal legislators

on the issue of state taxation

Rule #3

Better Example:

1. What led to the formation of the States' Rights Party?

A. The level of federal taxation B. The demand by states to make

their own laws C. The industrialization of the South D. The corruption of federal

legislators on the issue of state taxation

Rule #4

Emphasize qualifiers such as most likely and best in the stem.

Poor Example:1. Which measurement unit below

would be best to use if you wanted to measure the mass of a bumblebee?

A. gram B. kilogramC. millimeterD. centimeter

Rule #4

Better Example

1. Which measurement unit below would be best to use if you wanted to measure the mass of a bumblebee?

A. gram B. kilogramC. millimeterD. centimeter

Rule #5

Make sure the answer choices are plausible.

Poor Example:1. Who was president during World

War II?A. Franklin RooseveltB. George WashingtonC. George W. BushD. Jesse James

Rule #5

Better Example1. Who was president during World

War II?

A. Dwight EisenhowerB. Lyndon JohnsonC. Franklin RooseveltD. Woodrow Wilson

Rule #6 Develop answer choices that are parallel in

grammar and in length.Poor Example:1. In English class, Cynthia must write a descriptive

essay about the person she admires the most. She wants to write about her father. Which of these would be the best way she could begin to write her essay?

A. Look through old picture of her and her fatherB. She could call her best friend to talk about himC. By making a list of activities she has done with her father.D. Asking her father to come to her class

Rule #6

Better Example:

1. In English class, Cynthia must write a descriptive essay about the person she admires the most. She wants to write about her father. Which of these would be the best way she could begin to write her essay?

A. By looking at her baby picturesB. By asking her father to come to her

classC. By calling her best friend to talk about

her fatherD. By listing all of the things he has done

for her and others

Rule #7

Avoid using “All of the above” and “None of the above.”

Poor Example:1. Which of these should be used to

measure in millimeters?

A. A yard stickB. A metric rulerC. A digital scaleD. None of the above

Rule #7

Better Example:1. Which of these should be used to

measure in millimeters?

A. A yard stickB. A metric rulerC. A digital scaleD. A graduated cylinder

Rule #8 Place the answer choices in logical order.

Numbers in numerical order One-word answer choices in alphabetical order Order sentences from shortest to longest

Poor Example:1. A teacher is planning a field trip and will need

school buses to transport students A school bus holds 36 students. If 1,128 students will be transported, how many buses are needed?A. 32B. 31.33C. 36D. 31

Rule #8

Better Example1. A teacher is planning a field trip and

will need school buses to transport students A school bus holds 36 students. If 1,128 students will be transported, how many buses are needed?

A. 31B. 31.33C. 32D. 32.33

Rule #9

Avoid clues in the answer choices that give away the correct answer.

Poor Example1. Which of the following revolutions

resulted in increased business by speeding travel?A. industrialB. technologyC. textileD. transportation

Rule #9

Better Example

1. Which of the following best describes a result of the transportation revolution in the 1800s?

A. Suburban areas grewB. Demands for slave labor decreasedC. Manufacturing and production increasedD. People relied less on mass transportation

Rule #10 Make sure that the correct response is

the only correct response.Poor example1. Chicago Bears: football::

_________:baseball

A. Anaheim AngelsB. Los Angeles GalaxyC. Atlanta BravesD. Tampa Bay Buccaneers

Rule #10

Better example

1. large: big :: triumph : ___________

A. lossB. ruleC. smallD. success

Seinfeld teaches History - Cooperative Learning

The Work

Look at what you will be teaching from Jan. 6th to Feb. 1st

Focus on questions that address the standards

Assign questions: DOK Tag Standard/Element Tag

Create a bank of 30-35 questions (80% at DOK 2 or 3)

Question FormatDOK1 GPS Type Question Stem Here

A. Type Answer Choices Here

B. Type Answer Choices Here

C. Type Answer Choices Here

D. Type Answer Choices Here

1. Indicate DOK and Standard/Element

2. Leave Column Blank 3. Leave

Row Blank

Resources

www.edinformatics.com Released questions from standardized

state tests across the nationhttp://vhsprofessionallearning.wikisp

aces.com/ PowerPoint 5 Step Protocol Template for Question Banks