Reutilization of multiple-choice questions

Reutilization of multiple-choice questions

PEDRO HERSKOVICDepartment of Pediatrics, East Campus, Luis Calvo Mackenna Hospital, School of Medicine,University of Chile

SUMM AR Y Multiple-choice questions (MCQs) (one-best-answer

type, with ® ve alternatives) used for assessment of ® fth-year

medical students in the Pediatr ics course are thoroughly discussed

and analysed with the students after each test. This study was

undertaken with the aim of establishing whether this kind of

exposure precludes the reutilization of questions in future years.

In all, 197 questions were used more than once from 1994 to

1998. Considering the ® rst and last time they were included in a

test, 50 (25%) increased their difficulty, 39 (20%) diminished

it, and 108 (55%) remained unchanged. The correlation coef-

® cient between the percentage of students that answered correctly

each question the ® rst and last time they were used was 0.51 (p

< 0.001). It is concluded that the kind of exposure students are

given by our course to MCQs, seems not to preclude their future

reutilization.

Introduction

Exposure to students of multiple-choice questions (MCQ),

by publication in a book (Bandaranayake et al., 1990), or by

reutilization with the same group of students (Rees, 1986;

Miller et al., 1993), leads from modest to marked improve-

ment in performance when the students are confronted for

a second time with the same questions.

Since 1994 the MCQs (one-best-answer type, with ® ve

alternatives) that are part of the assessment of ® fth-year

medical students in the Pediatrics course given on campus

at the University of Chile are discussed in depth with the

students, shortly after the tests.The purposes of this are: to

obtain feedback from students, to assure the fairness of the

assessments, and to help them with their learning process

(Herskovic & Herskovic, 1998). The students are not given

copies of the questions to keep.

During the ensuing 5 years we have reused many MCQs

in different years, so we wanted to establish whether the

kind of exposure our questions receive is damaging them as

assessment tools.

Methods

The Pediatrics MCQ pool was reviewed to detect those

questions that had been used more than once, in different

years, from 1994 to 1998. The percentage of students that

answered the item correctly, (pc value) was registered for

the ® rst and last time each particular question was used.

Results

In all, 197 questions were used more than once in different

years. Seventy-two were used twice, 67 three times, 48 four

times and 10 ® ve times.

Table 1 gives the pc values arranged in increments of 20

for the ® rst and last time every question was used. The pc

value of 108 questions (55%) remained unchanged, for 50

questions (25%) decreased, and for 39 items (20%)

increased.

The correlation coefficient between the pc values for

both times each question was considered was 0.51 (p <

0.001).

Discussion

The variations observed in the pc values of these 197 ques-

tions seem irrelevant. The correlation coefficient suggests

some correlation between the pc values the ® rst and last

time each MCQ was used. Some improvement from year to

year might have been expected because, besides using tests

for grading, we use the analysis of questions as a diagnostic

tool to improve the learning process in the forthcoming

years. Nevertheless, 50 items (25%) had a higher level of

difficulty when repeated, and 39 (20%) were easier, with

the remaining 108 (55%) showing no change.

We believe that if the exposure of questions in the post-

test discussions had made the items `public’ among students,

a consistent improvement in their performance would have

been found in this analysis. Whether or not the `damage’ to

questions is time dependent cannot be answered with these

data, because more than half of the items were used only

two or three times.

Correspondence: Dr Pedro Herskovic, 8644 Principe de Gales, La Reina, Santiago,

Chile, 6871619. Tel/fax: + 56 2 2739897; email: herko@cmet.net

Table 1. pc values for the ® rst and last time multiple-choice questions were used.

pc values (%) for the last use of questions

0 ± 20 21± 40 41± 60 61± 80 81± 100

pc values (%)

for the ® rst use

of questions

0± 20 1

21± 40 2 1 3 3 1

41± 60 6 14 6 7

61± 80 1 3 12 26 18

81± 100 1 2 3 20 67

Note: pc value = percentage of students that answered the item correctly.

Medical Teacher, Vol. 21, No. 4, 1999

ISSN 0142-159X print /ISSN 1466-187X online/99/040430± 02 ½ 1999 Taylor & Francis Ltd430

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y SU

NY

Sta

te U

nive

rsity

of

New

Yor

k at

Sto

ny B

rook

on

11/0

1/14

For

pers

onal

use

onl

y.

We conclude from post-test analysis that this kind of

exposure to students of MCQs seems not to preclude their

reutilization in future years.

Notes on contributor

PE D RO HER SK OVIC MD is Assistant Professor of Pediatrics at

the Department of Pediatrics, East Campus, Luis Calvo Mack-

enna Hospital, School of Medicine, University of Chile. His

interests are in nutrition and in medical education. From 1994 to

1997 he was in charge of the Pediatrics course for ® fth-year medical

students.

References

BANDARA NAYAK E, R.C., BUZZARD , A.J. & M ARSHALL, V.C. (1990)

Effects of publishing multiple-choice questions on their subsequent

item analysis, Australia and New Zealand Journal of Surgery, 60, pp.

937 941.

HERSKOVIC , P. & HERSKOVIC , J. (1998) Knowledge evaluation that

teaches and satis® es students, in: Proceedings of the AMEE Conference,

Prague 1998, p. 17 (Association for Medical Education in Europe).

M ILLER, B.J., EFFENEY, D.J. & GOUGH, I.R. (1993 ) Do medical

students remember multiple choice questions?, Australia and New

Zealand Journal of Surgery, 63, pp. 897± 900.

REES, P.J. (1986 ) Do medical students learn from multiple choice

examinations?, Medica l Education, 20, pp. 123 ± 125.

Reutilization of MCQs

431

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y SU

NY

Sta

te U

nive

rsity

of

New

Yor

k at

Sto

ny B

rook

on

11/0

1/14

For

pers

onal

use

onl

y.