“There’s no confidence in multiple-choice testing …”

Phil Davies

School of Computing

University of Glamorgan

South Wales

Who has NO confidence?

• STAFF– “They can just guess the answer”– “It’s not the way it is in the real world”– “there are varying degree of knowledge”

(Echternacht, 1972)

– “don’t worry it’s easy, it’s multiple choice”

• Possibly more importantly -- STUDENTS

The First Test

• 40 Questions / Formative

• Time Consuming / Difficult to create sensible distracters / Rocket Science

• I thought … Average 1 minute per question

• Actual … all finished in ten minutes …

apart from one who took 50 minutes• Why? No reason to think

On-Line Assessment &

Learning … OLAL• Level Two Degree .. Computer

Communications & Networks

• Failures High … Programming to Demonstrate .. Technical Subject .. Essay-type answers in examinations

• Assess .. “Knowledge” “Understanding” “Learning Outcomes” …. Directed

On Line Assessment & LEARNING

• Four Tests

• Weeks 3,6,9,12 = 5%, 10%, 15% 20%

• Cumulative

• Two Passes of SAME test

• Learning .. Short term?

• Slowed Down .. Thought about WHEN WRONG .

What decides the QUALITY of a question?

• In the recent World cup qualifier between England and Greece, who scored the all important equalizing goal?

• David Beckham

• Victoria Beckham

• Brooklyn Beckham

Rewards?

• Know it

• On the tip of my tongue

• If I see the answers I’ll know it

• Deduction

• Guess

• Know it … oh no I’ve got it wrong .. misinformation

Allocation of Marks

• Logarithmic +3, +2, +1, -3, -2, -1

• This is not the same!! -> not seeing answer

• Students were involved / Provisional– “You can try this out but it must not have a

negative affect upon our results compared with last year”

• Reward knowledge +4, +2, +1, -2, -1, 0

• If we don’t know it, then why take off 1

Results 2000-2001 (NO CONFIDENCE TESTING)Test Number Average % Standard

DeviationOLAL TEST 1 56.49 16.11OLAL TEST 3 49.23 13.25OLAL TEST 4 59.18 14.25

Results 2001-2002 (WITH CONDFIDENCE TESTING)OLAL TEST 1 63.10 15.27OLAL TEST 3 55.55 13.41OLAL TEST 4

+4, +2, +1, -2, -1, 058.60 15.42

OLAL TEST 4+3, +2, +1, -3, -2, 0

51.33 17.32

OLAL TEST 4+3, +2, +1, -3, -2, -1

50.13 18.08

OLAL TEST 4+5, +3, +1, -3, -2, -1

56.45 16.43

OLAL TEST 4+5, +3, +1, -2, -1, 0

61.35 14.60

2002 OLAL 1/3 .. Better by approx. 6%

+5, +3, +1, -2, -1, 0

2000 +2 -1

05

1015202530

90to99

80to89

70to79

60to69

50to59

40to49

30to39

20to29

10to19

0to9

2000 +2 -1

2001 +4 +2 +1 -2 -1 0

05

1015202530

2001 +4 +2 +1 -2-1 0

2000 +2 -1

05

1015202530

90to99

80to89

70to79

60to69

50to59

40to49

30to39

20to29

10to19

0to9

2000 +2 -1

2001 +5 +3 +1 -2 -1 0

05

1015202530

2001 +5 +3 +1 -2-1 0

Ma

rks

20

00

-20

01

No

co

nfi

de

nc

e

%

20

01

-20

02

Co

nfi

de

nc

e

5,3

,1,2

,1,0

%

90-99 0 380-89 6 670-79 23 1960-69 25 2450-59 24 2740-49 12 1430-39 7 320-29 3 310-19 0 1

0-9 0 0

0

5

10

15

20

25

30

90 -99

80 -89

70 -79

60 -69

50 -59

40 -49

30 -39

20 -29

10 -19

0 - 9

No confidence

Confidence

Percentile Frequency Distribution

Average Mark Improvement, Tests 3 to 4Mark Rangevia OLAL 3

2000 (no confidence) 2001 Confidence(5,3,1,-2,-1,0)

80-89 4.3 7.0070-79 3.85 3.5960-69 7.32 3.6350-59 6.37 5.0740-49 10.52 5.8030-39 12.44 11.1020-29 20.71 19.5510-19 35.05 26.11

0-9 35.39Quartiles

75+ 5.98 4.2350+ 6.62 3.2525+ 10.78 5.260+ 16.29 10.49

AverageImprovement

9.95 11.00

Number of Questions correct (out of 60)MARKSTest 3

2000No Confidence

2001Confidence

80-89 55.00 56.0070-79 50.00 50.8660-69 48.71 47.7450-59 44.23 44.6640-49 42.52 41.5830-39 39.23 40.0020-29 38.66 39.6710-19 37.50 37.00

0-9 37.00Quartiles

75+ 48.48 50.5850+ 44.33 46.8025+ 42.93 43.550+ 39.10 40.27

Proportion of Questions Correct

• High Confidence => 84% correct

• Fairly Confident => 58% correct

• Not confident => 47% correct

• 72% of all questions answered with high confidence

Based on Test 3, who were the most confident (60 Questions)

• 75+– 52 7 1

• 50+– 44 13 3

• 25+– 42 15 3

• 0+– 33 19 8

Student Feedback Thoughts on Multiple Choice

• “useful learning tool”

• “second test is a really useful way of learning”

• “immediate feedback helps me to learn”

• “doesn’t reflect my lack of knowledge”

• “not fair, I found it difficult to tell the difference between the answers”

• “negative marking made me panic”

Student thoughts on Confidence

• “Eliminates guesswork”

• “I was waiting for a leggy blond to bring on my prize at the end”

• “separated the lucky students from me”

• “at least I didn’t lose marks for selecting no confidence”

• “are you testing my confidence or knowledge”

Quality of a multiple choice question?

• In the recent World cup qualifier between England and Greece, who scored the all important equalizing goal?

• Teddy Sheringham• Michael Owen• Robbie Fowler• QUALITY OF QUESTION NOT

DISTRACTERS

Question Format• “One or more questions were impossible to

judge”

• “I wish you’d put more detail in the question”

• “Perhaps it would have been better to show the answers, ask for confidence, and then show the question”

Conclusions?• Assess subject area not individual’s

confidence

• Quality of questions => question NOT distracters

• Allocation of Marks (based on past?)

• Student Accept / Staff Accept

• Time for tests increases “really had to think”