© Courseware Interactive Ltd., 2003 Presentation May 22nd 2003 Designing Effective Multiple Choice Questions ILTA, Waterford Niall Watts Courseware Interactive

© Courseware Interactive Ltd., 2003

PresentationMay 22nd 2003

Designing Effective Multiple Choice Questions

ILTA,Waterford

Niall Watts

Courseware Interactive Ltd


Evaluation - content & teaching

Retention

Reinforcement

Diagnosis - feedback

Assessment

Engagement - Interaction

Why ask questions?


Interactivity

Students interacting with content

Make them thinkCheck their knowledgeCognitive activity

Not necessarily:FlashMenusDatabases


Types of Question

Choosing - Multiple Choice, Matching, Selecting

Open - Text Entry, Short Answer, Fill in the Blank, Completion


Choosing

Advantages Easy to Mark Usability

Disadvantages Hard to set Guessing answer Recall of factual information

Popular with teachers & designers


Short Answer

Advantages Easy to Set No guessing

DisadvantagesHard to Mark

Less Popular


Multiple Choice

Multiple-choice questions are often criticised for being easy to guess and for emphasising factual information. We contend that well-written multiple-choice questions can do a good job of assessing comprehension, problem solving and other higher order skills.

Alessi & Trollip, Multimedia for Learning 3rd Ed.,2001


Multiple Choice Terminology

Anatomy StemOptionsDistractors (Foils)


Multiple Choice Clarity

Most text in stem

Easy to ReadFocused on questionNo additional or supplementary

informationAvoid scrolling



Clear InstructionsNo ambiguityTell them what is expected

e.g Number of Correct OptionsNo tricks

Just checking the student’s knowledge and understanding of the course



AvoidNegativesConditionals


Multiple Choice Distractors

Hard to setThe secret of good multiple choice question

design is well-written distractorsA distractor must never suggest the correct

answer


Multiple Choice Issues

Plausible distractors

more

Option order

Number of options

Accidental hints


Distractors - Plausibility Sensible

All options should sound plausible No ‘Mickey Mouse’ unless it’s a Disney Quiz

Hint Write question as open Use popular wrong answers as distractors


Testing Comprehension

Avoid repeating lesson text verbatim students may recognise key words students may recall context

Paraphrase for comprehension Use different examples or case studies Ask student to apply a rule or principle


Higher Order Learning

Problem solvingGive several answers to a problem

AnalysisChoose between interpretations Select reasons for choice


Distractors – Accidental Hints

ConditionsGrammatical & Linguistic ConsistencyOption LengthRepeating TextPrecisionIntersecting OptionsOverlapping OptionsAll of the AboveNone of the Above


Distractors – Accidental Hints

Correct answers must always be correct Distractors must always be incorrect Negatives can be confusing Conditionals can obscure the answer or give it away

Exercise



Avoid ambiguityCorrect answers must always be correctDistractors must always be incorrectE.g. Word Processing Question

– Complete the following sentence correctly. A Word processing program is best defined as software used to:

– .Publish newsletters– .Produce documents– .Design graphics– .Send mailshots

– Option 2 is probably ‘the best definition’ but options 1 and 4 could also be

correct. more



Avoid ambiguityE.g. Word Processing Question

– The question could be better written as:

– Complete the following sentence correctly. A Word processing program is best defined as software used to:

– 1. Send e-mail messages– 2. Produce documents– 3. Produce graphics– 4. Produce accounts

more



Avoid ambiguityAvoid Negatives. They can confuse a student.

E.g.– Which of the following is not a fraction?

– 0.5– 1/2

• This could be better written as:– Which of the following is a fraction?

– 0.5– 1/2

more



Avoid ambiguity

Negatives in true/false questions can be particularly confusing. E.g.– London is not the capital of England. True or False.

• In particular avoid negatives in both the stem and the options

more



Avoid Ambiguity - Conditions

Avoid conditions such as ‘often’, ‘sometimes’,’usually’,’always’,’never’ or ‘all’.

Avoid the use of the verb ‘may’. Conditions may make the answer unclear or give it away altogether.

True or False. Metals are usually solids at room temperature.

The word usually should be avoided as it hints that the correct answer is true.

True or False. All Metals are solids at room temperature.

The word all should be avoided as it hints that the correct answer is false.



Grammatical & Linguistic Consistency

Sentence structure reveals answer where language flows from stem to correct answer. Example:Second option below

more

Which option completes the following sentence? In 1492 Christopher Columbus…

is the discoverer of North Americalanded in the New Worldis buried in Seville cathedralwas beaten by the Vikings by 500 years



Length of Options

Where one answer is much longer than the others it may well be correct

more

Which of the following best defines a bird?

A feathered animalA warm-blooded animal with feathersA warm-blooded animal with feathers that lays eggs



Repeating Text Where text in the stem is repeated in one of the options

that option is likely to be correct. Example

more

Why was the Financial Services Authority founded?

To prevent money launderingTo regulate financial servicesTo provide consumer informationTo control pension sales



Precision Where one option is more precise it hints that it may be

correct

more

The height of Ben Nevis is:

1000 metres1343 metres1500 metres4000 metres



Intersecting Options An option covering more than one point is more likely to be

correct. Here, option 2 covers both eating & time.

more

Medication to lower blood pressure should be taken once a day. Should it be taken?

At a mealtimeBefore breakfastIn the morning



Overlapping Options Be careful that you do not choose overlapping distractors. At

first glance they may seem correct but potentially there are multiple correct answers

more

Due to cutbacks the budget was reduced to less than:

£10,000 £8,000 £5,000

Suppose the budget was reduced to £4,500!



All of the Above Avoid ‘all of the above’

If one distractor is wrong it cannot be the answer If two options are correct it must be the answer (1 correct only)

more



None of the Above ‘None of the above’ can be used safely provided

Sometimes it is the correct answer Provided ‘the best answer’ is not required

more



Ambiguity Negatives Conditions Grammatical & Linguistic Consistency Option Length Repeating Text Precision Intersecting Options Overlapping Options All of the Above None of the Above


Multiple Choice Options

How many options?

4 is popularHow many correct?Higher number of options reduces guessing riskThree options may be appropriate e.g.

– At constant temperature with increasing pressure the volume of a fixed mass of gas

– 1. Decreases– 2. Increases

– 3. Stays the Same


Multiple Choice Options

Option Order

3rd position correct is popular1st is unpopular!Position choice:

Correct answer must be randomly positioned Or options in order e.g alphabetical

Position must NOT suggest a correct choice


True/False Questions

Multiple Choice with 2 options 50% chance of correct guess Easy to Mark Hard to Set Can be disguised e.g

select an object on a diagram


CourseBuilder

Used to create learning interactions Extension for Dreamweaver Free Register with Macromedia Download from www.macromedia.com/ Follow instructions on screen


CourseBuilder

Make a multiple choice question

Insert - CourseBuilder Interaction

Multi Choice Radios with Defaults

Red button to edit


CourseBuilder

Customising multiple choice question

Multi Choice Radios with Set Text of Layer

Multi Choice Checkboxes with Layers


ResourcesInstructional Design

IFETS - ifets.ieee.orgITFORUM - University of GeorgiaERICAssessment – Question Mark

Web developmentMacromediaCoursebuilder ListServRapidIntake – www.rapidintake.comJakob Nielsen Usability - www.useit.com


Contact Niall Watts Courseware Interactive Ltd

[email protected] www.courseware.ie (353-1) 295 2484

Documents

© Courseware Interactive Ltd., 2003 Presentation May 22nd 2003 Designing Effective Multiple Choice Questions ILTA, Waterford Niall Watts Courseware Interactive