The Continual Assessment of The Continual Assessment of Confidence or Knowledge with Confidence or Knowledge with

Hidden MCQ?Hidden MCQ?

Short Paper W.I.P.Short Paper W.I.P.

Phil DaviesPhil Davies

School of ComputingSchool of Computing

University of GlamorganUniversity of Glamorgan

Conference 1999Conference 1999Learning Through AssessmentLearning Through Assessment

Phil DaviesPhil Davies

School of ComputingSchool of Computing

University of GlamorganUniversity of Glamorgan

Implementation of OLALImplementation of OLAL

Tests Weeks 3,6,8,12 (60% Subject Mark)Tests Weeks 3,6,8,12 (60% Subject Mark) Percentage Marks 5%,15%,15%,25%Percentage Marks 5%,15%,15%,25% ““Beep Test”Beep Test” +2 Correct -1 Incorrect(Really -3)+2 Correct -1 Incorrect(Really -3)

Student ConcentrationStudent Concentration

GuessingGuessing– Why should the student care if incorrect?Why should the student care if incorrect?

Allow TWO PASSES of the testAllow TWO PASSES of the test By concentrating on first passBy concentrating on first pass

– will get reward on second passwill get reward on second pass– formativeformative

Results of The TestsResults of The Tests

Improvement of Test Results Pass One to Two

Who benefited from the test?Who benefited from the test?

<10 10-19

20-29

30-39

40-49

50-59

60-69

70-79

80-89

90>

0 1 5 5 20 23 27 17 10 0

Student FeedbackStudent Feedback

Excellent way of gaining knowledgeExcellent way of gaining knowledge 2nd Test Allowed us to learn2nd Test Allowed us to learn Good assistant to learningGood assistant to learning Encourages RevisionEncourages Revision Very good, promotes learning of a subject Very good, promotes learning of a subject

early onearly on Better than sex!!!!Better than sex!!!!

““There’s no confidence in There’s no confidence in multiple-choice testing multiple-choice testing

…”…”

CAA Conference 2002CAA Conference 2002

Phil DaviesPhil Davies

What decides the QUALITY of a What decides the QUALITY of a question?question?

In the 2001 World cup qualifier between In the 2001 World cup qualifier between England and Greece, who scored the all England and Greece, who scored the all important equalizing goal?important equalizing goal?

David BeckhamDavid Beckham Victoria BeckhamVictoria Beckham Brooklyn BeckhamBrooklyn Beckham Rebecca LoosRebecca Loos

Rewards?Rewards?

Know itKnow it On the tip of my tongueOn the tip of my tongue If I see the answers I’ll know itIf I see the answers I’ll know it DeductionDeduction GuessGuess Know it … oh no I’ve got it wrong .. Know it … oh no I’ve got it wrong ..

misinformationmisinformation

Proportion of Questions CorrectProportion of Questions Correct

High Confidence => 84% correctHigh Confidence => 84% correct Fairly Confident => 58% correctFairly Confident => 58% correct Not confident => 47% correctNot confident => 47% correct

72% of all questions answered with high 72% of all questions answered with high confidenceconfidence

Student thoughts on ConfidenceStudent thoughts on Confidence

““Eliminates guesswork”Eliminates guesswork” ““I was waiting for a leggy blond to bring I was waiting for a leggy blond to bring

on my prize at the end”on my prize at the end” ““separated the lucky students from me”separated the lucky students from me” ““at least I didn’t lose marks for selecting at least I didn’t lose marks for selecting

no confidence”no confidence” ““are you testing my confidence or are you testing my confidence or

knowledge”knowledge”

Quality of a multiple choice Quality of a multiple choice question?question?

In the 2001 World cup qualifier between In the 2001 World cup qualifier between England and Greece, who scored the all England and Greece, who scored the all important equalizing goal?important equalizing goal?

Teddy SheringhamTeddy Sheringham Michael OwenMichael Owen Robbie FowlerRobbie Fowler QUALITY OF QUESTION NOT QUALITY OF QUESTION NOT

DISTRACTERSDISTRACTERS

““Bite The Bullet”Bite The Bullet”

Use Confidence Testing for all four tests (10%, Use Confidence Testing for all four tests (10%, 15%, 20% & 25%)15%, 20% & 25%)

Weighting of Passes (50/50, 60/40, 70/30 & 80/20)Weighting of Passes (50/50, 60/40, 70/30 & 80/20) How to judge if the students had ‘really’ learned?How to judge if the students had ‘really’ learned? Identify improvement > Identify improvement > Select five questions that were answered ‘poorest’ Select five questions that were answered ‘poorest’

in test one & pass through to test twoin test one & pass through to test two Select five questions that were answered ‘poorest’ Select five questions that were answered ‘poorest’

in test two & pass through to test three ( + original in test two & pass through to test three ( + original five) …. Etc.five) …. Etc.

How to identify ‘Poorest’How to identify ‘Poorest’

Using confidence, degree of correctness of Using confidence, degree of correctness of answeringanswering

Using confidence scores to provide a Using confidence scores to provide a degree of ‘poorness’ degree of ‘poorness’ i.e. +4, +2, +1, -6, -5, -4i.e. +4, +2, +1, -6, -5, -4

High degree of ‘poorness’ could just be a High degree of ‘poorness’ could just be a ‘BAD’ question‘BAD’ question

Swing between 1Swing between 1stst and 2 and 2ndnd pass of degree of pass of degree of correctness should be ‘Reasonable’correctness should be ‘Reasonable’

80

>

75

-79

70

-74

65

-69

60

-64

55

-59

50

-54

45

-49

40

-44

35

-39

30

-34

25

-29

1 2 5 6 11 13 15 12 7 7 6 3

Test # Average Pass One Average Pass Two Ratio First/Second Average Total

Test One10%

42.56% 72.74% 50 / 50 57.65%

Test Two15%

26.80% 66.62% 60 / 40 42.73%

Test Three20%

41.14% 76.40% 70 / 30 51.72%

Test Four25%

51.00% 81.50% 80 / 20 57.10%

Test 2 - 1 Test 3 - 2 Test 4 - 3 Test 4 - 1

80> -2 4 23 25.00

75-79 10.5 9 45.5 65.00

70-74 -31.8 43 9.6 20.80

65-69 -34.83 43.83 29.67 38.67

60-64 -45.27 53.27 16.55 24.55

55-59 -43.15 36.08 20.23 13.15

50-54 -36.2 48.8 21.53 34.13

45-49 -7.0 27.92 38.83 59.75

40-44 -45.43 30.00 28.43 13.00

35-39 -32.86 27.00 25.71 19.86

30-35 -47.33 44.5 31.33 28.50

25-29 -47.67 37.00 -13.00 -23.67

Average -30.25 +33.70 +23.12 +26.56

Identifying Poor Questions Test Identifying Poor Questions Test OneOne

Qu

estio

n N

um

be

r

# C

orre

ct Pa

ss 1

# W

ron

g P

ass 1

We

igh

ting

Fa

ctor

# C

orre

ct Pa

ss 2

# W

ron

g P

ass 2

We

igh

ting

Fa

ctor

We

igh

ting

Diffe

ren

ceP

ass 2

-1

Q5 27 61 -251 56 32 +9 260

Q14 30 58 -218 60 28 +67 285

Q15 34 54 -181 55 33 +4 185

Q19 8 80 -356 76 12 +223 579

Q20 33 55 -187 58 30 +16 203

Q27 11 77 -373 58 30 +21 394

Identifying the Poorest Identifying the Poorest Questions Test two to threeQuestions Test two to three

Qu

estio

n N

um

be

r

# C

orre

ct Pa

ss 1

# W

ron

g P

ass 1

We

igh

ting

Fa

ctor

# C

orre

ct Pa

ss 2

# W

ron

g P

ass 2

We

igh

ting

Fa

ctor

We

igh

ting

Diffe

ren

ceP

ass 2

-1

Q2 14 74 -348 79 9 250 598

Q6 28 60 -222 45 43 -109 331

Q8 22 66 -255 70 18 145 400

Q10 28 60 -215 48 40 -66 281

Q11 20 68 -281 57 31 20 301

Q18 28 60 -200 79 9 227 427

Q19 31 57 -202 74 14 166 368

Q21 27 61 -235 57 31 32 267

Q27 32 56 -210 66 22 106 316

Q28 27 61 -256 77 11 238 494

Q32 23 65 -254 82 6 265 519

Identifying Poorest Questions Identifying Poorest Questions from Three to Fourfrom Three to Four

Qu

estio

n N

um

be

r

# C

orre

ct Pa

ss 1

# W

ron

g P

ass 1

We

igh

ting

Fa

ctor

# C

orre

ct Pa

ss 2

# W

ron

g P

ass 2

We

igh

ting

Fa

ctor

We

igh

ting

Diffe

ren

ceP

ass 2

-1

Q6 32 56 -212 63 25 73 285

Q11 27 61 -219 75 13 217 436

Q16 19 69 -313 60 28 58 371

Q26 30 58 -225 81 7 253 480

Q40 30 58 -218 72 16 183 401

Passed through to Test FourPassed through to Test Four

13 of the poorest question13 of the poorest question

6 new questions based upon additional work 6 new questions based upon additional work between tests 3-4between tests 3-4

21 questions (cross section of questions 21 questions (cross section of questions and topics from previous tests)and topics from previous tests)

Improvement of Results Through Improvement of Results Through TestsTests

Questions Through Tests

-500

-400

-300

-200

-100

0

100

200

300

400

1 2 3 4 5 6 7 8

Question 5

Question 6

Question 10

Question 11

Question 14

Question 15

Question 16

Question 19

Question 20

Question 21

Question 26

Question 27

Question 40

Improvement Pass One of QuestionsImprovement Pass One of Questions

Pass One Question Improvements

-500

-400

-300

-200

-100

0

100

200

300

400

1 2 3 4

Question 5

Question 6

Question 10

Question 11

Question 14

Question 15

Question 16

Question 19

Question 20

Question 21

Question 26

Question 27

Question 40

Improvement for Question 5Improvement for Question 5

Stronger Students improved quickly through Stronger Students improved quickly through the teststhe tests

Became more confident in answering the Became more confident in answering the questionsquestions

Weaker Students improved less quickly than Weaker Students improved less quickly than stronger through the test progressstronger through the test progress

Is it just Confidence or Knowledge Is it just Confidence or Knowledge i.e. questions right or wrong?i.e. questions right or wrong?

Category Average # of Questions Correct Pass 1

Average # of Questions Correct Pass 2

Average Total # of Questions Correct

80> (1) 131.0 158.0 289.0

75-79 (2) 123.5 150.5 274.0

70-74 (3) 117.6 148.6 266.2

65-69 (4) 113.8 150.7 264.5

60-64 (5) 110.3 145.0 255.3

55-59 (6) 107.6 143.4 251.0

50-54 (7) 100.4 136.2 236.6

45-49 (8) 92.5 132.6 225.1

40-44 (9) 94.0 126.4 220.4

35-39 (10) 87.7 121.6 209.3

30-34 (11) 80.5 117.3 197.8

25-29 (12) 76.7 117.3 194.0

Some ConcernsSome ConcernsCategory Average # of Questions

Correct Pass 1Average # of Questions

Correct Pass 2Average Total # of

Questions Correct

12 81 127 208 (194)

3 102 136 238 (266)

7 78 117 195 (236)

8 112 141 253 (225)

10 97 137 234 (209)

Overall CommentsOverall Comments

Work in ProgressWork in Progress Results look very positiveResults look very positive Correctness Factor is dependant upon weighting Correctness Factor is dependant upon weighting

of scoresof scores Guessing not so much of a problem Guessing not so much of a problem Changed students mode of revising / studyChanged students mode of revising / study Fine granularity in the QUALITY OF A QUESTIONFine granularity in the QUALITY OF A QUESTION

Possibility of Computerized Adaptive TestingPossibility of Computerized Adaptive TestingHowever note the quality of the question change However note the quality of the question change

throughout the course of the cumulative teststhroughout the course of the cumulative tests Does gender come into it??Does gender come into it??

Contact InformationContact Information

Email: Email: pdavies@glam.ac.ukpdavies@glam.ac.uk Phone: 01443 482247Phone: 01443 482247

School of ComputingSchool of ComputingUniversity of GlamorganUniversity of GlamorganPontypriddPontypriddMid – GlamorganMid – GlamorganSouth WalesSouth Wales