EFFECTS OF QUESTION FORMATS ON STUDENT AND ITEM PERFORMANCE

Adam Pate, PharmDDavid Caldwell, PharmD

The fundamental problem in shock is:a. Decreased diastolic blood pressureb. Decreased systolic blood pressurec. Decreased tissue perfusiond. Increased serum lactic acid

Example

If you find a patient in shock, your first action should be:a. Begin CPRb. Call 911c. Circulation, airway, breathing assessmentd. Find an AED

Example

Haladyna, Downing, & Rodriguez

• Developed from an analysis of 27 textbooks and 27 research studies

• Purpose was to validate each guideline based on agreement in studied sources

Haladyna TM, Downing SM, Rodriguez SM. Applied Measurement in Education. 2002; 15(3):309-334.

Which of the following is the capital of Texas?a. Austin*b. Dallasc. El Pasod. Houston

Anatomy of a MC question

Guidelines: a closer look

Guideline For (%) Uncited (%) Against (%)

Use positives, no negatives 63 19 18

Write as many plausible distractors as you can 70 26 4

Use None of the Above carefully 44 7 48

Avoid All of the Above 70 7 22

All of the following drugs are metabolized by the P450 enzyme system except:a. Atazanavirb. Darunavirc. Raltegravird. Tipranavir

Use positives, no negatives

Stem negation = most appropriate whenused to measurerelevant objectives

Haladyna and Downing (1993)• Two-thirds of all items had only 1-2

effectively performing distractors• Only 1-8% of items had three

effective distractors• No measured effect on difficulty, but

more effective distractors = more discrimination

Write as many plausible distractors as possible

Haladyna TM, Downing SM. Education and Psychological Measurement. 1993; 53:999-1010.

Effect of numerous distractors unclear

Which of the following is the capital of Texas?A. DallasB. El PasoC. HoustonD. LubbockE. None of the above

Use “none of the above” carefully

3 of 4 studies failed to demonstrate differences in discrimination with “None of the above”

Which of the following is true regarding transdermal NRT?A. These products may be more appropriate in patients who anticipate

adherence problemsB. The 16-hr patch reduces the risk of insomnia and vivid dreaming

compared to the 24-hrC. Currently there is insufficient data to directly compare transdermal

NRT to oral NRTD. Transdermal NRT patches should never be cutE. All of the above

Avoid “all of the above”

70% of referenced sources recommend avoiding “all of the above”

50

60

70

80

90

100

Difficulty (p)

StandardFlawed

A retrospective study

0

0.2

0.4

0.6

0.8

1

Discrimination (rpb)

p=0.01 p=0.64

!

48%

52%

Guideline # Times broken % Total

Minimize reading 24 16.9

All of the above 24 16.9

Complex item format 19 13.38

Central idea in stem 17 11.97

Edit and proof 10 7.04

Choices homogenous 8 5.63

Plausible distractors 8 5.63

Trivial Content 6 4.23Length of choices equal 6 4.23

None of the above 6 4.23

Other 13 9.86

Ratio Standard:Flawed

Pate AN, Caldwell DJ. CPTL. 2014; 6(1):130-134.

A prospective study5 authors

Student randomizationstandard, n=55

flawed, n=54

Test administration

Analysis: one-sidedindependent t-tests

between the two scalesMean

discrimination, rpbMean itemdifficulty, p

40 exam items

25

35

45

55

65

75

Difficulty (p)

Standard Flawed

p=0.040

0.1

0.2

0.3

0.4

0.5

Discrimination (rpb)

p=0.44

Caldwell DJ, Pate AN. J Pharm Ed. 2013; 77(4):Article 71.

Investigator review

Test form 120 standard items

Test form 220 flawed items

Any questions?