Question Writing GuideABNM SAM Questions

J. Anthony Parker, MD PhDAssociate Executive Director, ABNM

Beth Israel Deaconess Medical CenterBoston, Massachusetts

Tony_Parker@BIDMC.Harvard.edu

Caveat Lector

PsychometricsOrigin for Questions Writing Rules

Psychometrics is the field of study concerned with the theory and technique of psychological measurement, which includes the measurement of knowledge, abilities, attitudes, personality traits, and educational measurement. The field is primarily concerned with the construction and validation of measurement instruments such as questionnaires, tests, and personality assessments.

– Wikipedia

Test Goal is to Measure AbilityLogistic regression where the probability of a correctanswer is a function of

Fn, the ability of person, nbi, the difficulty of question, i

Prob

abili

ty

Ability of person , n, Fn

Difficulty of question, i, bi

Ability 0.3

Wikipedia

Can Also Measure Question DifficultyLogistic regression where the probability of a correctanswer is a function of

Fn, the ability of person, nbi, the difficulty of question, i

Prob

abili

ty

Difficulty of question, i, bi

Ability of person , n, Fn

Difficulty 0.3

Wikipedia

Question DiscriminationThe slope determines how well a questiondiscriminates between candidate abilities

ai, ability discriminationlog-odds correct answer = ai • (Fn - bi)

Wikipedia

Prob

abili

ty

Difficulty of question, i, bi

Ability of person , n, Fn = 0.3

Difficulty 0.3

Slope, aiDiscrimination

ABNM Procedure

Assume questions are good and correctly keyedMeasure candidate tentative abilityUsing tentative abilities measure question difficulty

Discard questions that are too hard or too easyAlso measure question-ability discrimination

Correct wrong keysDiscard questions that not correlated with ability

Using improved test, measure final candidate abilityAdjust to standard independent of candidates and test

Writing Good Questions:Questions that Correlate with Ability

Psychometricians have developed rulesfor writing good questions

Rules are based on question performanceBad questions test abilities unrelated to

the test goal

Some Rules for Good Questions

No negatives (not, except, none)No absolutes (always, never, all, every, only, must)No overlapping rangesNo “all of the above” or “none of the above”No mutually exclusive answers (one is correct)Logical order (numerical, alphabetical, category)Single concept per question

No multiple true / false questions

Which of the following is truein Nuclear Medicine?

MIBG is used to image somatostatin receptorsA photon is completely absorbed during a

Compton interaction18F-FDG is transported into cells by GLUTLymphoma is never FDG-avid

Multiple true / false “which is true?”Multiple unrelated concepts“Never” is an absolute

With regard to the thyroid which is correct?

Pentetreotide is taken up in the parafollicular cellsFollicular cells are non-iodine-avidStromal cells take up iodidePentetreotide is not taken by the thyroid

Multiple true / false “which is correct?”First and last answers are mutually exclusive

One of them must be correct“Non” and “not” are negatives

In the thyroid, pentetreotide istaken up MOST avidly in:

parafollicular cells (C cells)follicular cellscolloidstroma

In the thyroid, pentetreotide istaken up MOST avidly in:

parafollicular cells (C cells)follicular cellscolloidstroma

Which of the following is not correct?

11C-acetate is metabolized in the the TCA cycle11C-choline can be used for prostate cancer18F-fluorodexoyglucose (FDG) is not

phosphorylated in the cell18F-fluorothymidine (FLT) is correlated with cellular

proliferation

Not in stem and in 3rd answerMultiple true / false “which is correct?”Multiple unrelated concepts

MOST closely correlatedwith proliferation?

11C-acetate11C-choline18F-fluorodexoyglucose (FDG)18F-fluorothymidine (FLT)

MOST closely correlatedwith proliferation?

Good distractors are often partially true,but not the best answer

11C-acetate11C-choline18F-fluorodexoyglucose (FDG)18F-fluorothymidine (FLT)

What is the BEST administered dose to use for a 18F-FDG PET pediatric scan?

≤ 1 MBq/kg (≤ 0.03 mCi/kg)1 - 4 MBq/kg (0.03 - 0.11 mCi/kg)4 - 16 MBq/kg (0.11 - 0.43 mCi/kg)10 - 64 MBq/kg (0.27 - 1.73 mCi/kg)

Overlapping ranges4-16 and 10-64 both include 10-161-4 and 4-16 both include 4

What is the BEST administered dose to use for a 18F-FDG PET pediatric scan?

1 MBq/kg (0.03 mCi/kg)4 MBq/kg (0.11 mCi/kg)16 MBq/kg (0.43 mCi/kg)64 MBq/kg (1.73 mCi/kg)

What is the BEST administered dose to use for a 18F-FDG PET pediatric scan?

1 MBq/kg (0.03 mCi/kg)4 MBq/kg (0.11 mCi/kg)16 MBq/kg (0.43 mCi/kg)64 MBq/kg (1.73 mCi/kg)

Gelfand: Pediatric radiopharmaceutical administeredDoses: 2010 north american consensus guidelines.J Nucl Med 2011;52:318

Parallel Answers / Avoid Teaching

Answers should be parallel in constructionand length

Good test takers know that “teachers like to teach”The correct answer is often the longest

Tendency to make correct answer exactly correctFor testing, avoid teaching

Urinary excretion of FDG is related to:

FDG concentration overwhelms GLUTrenal GLUT has low affinity for FDGboth of the proximal and distal convoluted tubule

sodium glucose transporters (SGLT1 & SGLT2respectively) have low affinity for FDG

there is little hexokinase in the kidney

Asymmetric answersTeaching

Urinary excretion of FDG is related to:

FDG concentration overwhelms GLUTrenal GLUT has low affinity for FDGSGLT1/2 have low affinity for FDGthere is little hexokinase in the kidney

Urinary excretion of FDG is related to:

FDG concentration overwhelms GLUTrenal GLUT has low affinity for FDGSGLT1/2 have low affinity for FDGthere is little hexokinase in the kidney

Glucose Reuptake in the Kidney

Stem (Question)

1. Ask a questionGiven …, what is the MOST likely diagnosis?

2. Give and incomplete statementGiven …, the BEST next step is:

3. Describe a situation followed #1 or #2A one-year-old girl had jaundice and a5 x 6 cm mass in the porta. Total bilirubinis 7 mg/dL. …

Stem (Question)

Put most of the content in stemAnswers short

Move common words in answers to stemRemove unnecessary words or sentences

In a 73-year-old man with metastaticmelanoma, what is the MOST likely causeof the FDG uptake shown in the figure?

prostate cancermelanomanormal bladdernormal urethra

In a 73-year-old man with metastaticmelanoma, what is the MOST likely causeof the FDG uptake shown in the figure?

prostate cancermelanomanormal bladdernormal urethra

Distractors (Incorrect Answers)

Should be plausibleLogical misconceptionsCommon misunderstandingsReasonable

not the BEST answernot the MOST likely answer

Testing vs. TeachingWright Map

candidates itemsmore able

less able

hard

easy

Testingdistinguish candidatesdifficult itemsjudgment requiredlow passing %

Teachinghelp learningeasier itemsemphasize key points

Question Taxonomy

Recall (fact)The energy of 99mTc is:

Interpretive (diagnosis)A patient with … had an FDG-PET/CT shownin the figure. The MOST likely diagnosis is:

Problem solving (management, judgment)A patient with … had an FDG-PET/CT shownin the figure. The BEST next step is:

Higher Taxonomy Preferred

Often involve more advanced reasoningMore complex medical taskStill single clinical concept

Distractors may be easier to writeSome topics require lower taxonomy questions

Stable iodine (SSKI or Lugol’s solution) should be administered prior to:

pentetreotide (Octreoscan®)iobenguane (MIBG)a contrast FDG-PET/CTibritumomab tiuxetan (Zevalin®)

Stable iodine (SSKI or Lugol’s solution) should be administered prior to:

pentetreotide (Octreoscan®)iobenguane (MIBG)a contrast FDG-PET/CTibritumomab tiuxetan (Zevalin®)

Recall questionpentetreotide confused with MIBG

ibritumomab tiuxetan confused with tositumomab

Based on the electrocardiogram, whatis the BEST diagnosis?

Difficult to come up with 4 plausible diagnoses

A 45 y/o diabetic man with weakness and shortness of breath for 3 days presents for exercise MPI. The pre-

exercise ECG is shown.

What should be done next?

Obtain right precordial leads.Use dipyridamole stress instead.Accompany the patient to the EW.Proceed as planned.

What should be done next?

Obtain right precordial leads.Use dipyridamole stress instead.Accompany the patient to the EW.Proceed as planned.

Writing Distractors

Difficulty writing reasonable distractorsChange taxonomy

Management distractors often easier to write

The BEST diagnosis is:

The BEST diagnosis is:

normal thymusmediastinal metastases??

Diagnostic question

The uptake in the chest is related to:

The uptake in the chest is related to:

the Na / I symportersomatostatin receptor, type 2brown fatmetastases

Diagnosis & biodistribution

A syringe of 90Y-ibritumomab tiuxetan (Zevalin ®) measuring 1025 MBq (27.7 mCi) needs to be discarded due to QC failure. After transfer to a vial, measurements are vial: 815 MBq (22.0 mCi), syringe: 15 MBq

(0.4 mCi). What should be done next?

Declare a medical eventSurvey the areaCall the radiation safety officer (RSO)Store for decay

A syringe of 90Y-ibritumomab tiuxetan (Zevalin ®) measuring 1025 MBq (27.7 mCi) needs to be discarded due to QC failure. After transfer to a vial, measurements are vial: 815 MBq (22.0 mCi), syringe: 15 MBq

(0.4 mCi). What should be done next?

Declare a medical eventSurvey the areaCall the radiation safety officer (RSO)Store for decay

Which Isotope is MOST Difficult toAccurately Measure in a Dose Calibrator?

18F90Y99mTc111In

Which Isotope is MOST Difficult toAccurately Measure in a Dose Calibrator?

18F90Y99mTc111In

What radiopharmaceutical was administered?

99mTc-pertechnetate67Ga-citrate111In-pentetreotide123I-MIBG

What radiopharmaceutical was administered?

99mTc-pertechnetate67Ga-citrate111In-pentetreotide123I-MIBG

What’s the Radiopharmaceutical?Stupid Questions, but I Use Them

Should knownormal and common variant biodistributionat organ, cell, and receptor level

To be able toidentify abnormal biodistributionidentify radiopharmaceutical QCunderstand unusual findings

After injection of 111In-DTPA in a patientwith normal renal function?

the half-time of excretion is 16 minutesthe half-time of excretion 8 minutesthe half-time of excretion 1.7 hoursthe half-time of excretion 36 minutes

Repeated words in answers

How long after injection of 111In-DTPA is 1/2 excreted with normal renal function?

16 minutes8 minutes1.7 hours36 minutes

Answers not in logical order

How long after injection of 111In-DTPA is 1/2 excreted with normal renal function?

8 minutes16 minutes36 minutes1.7 hours

How long after injection of 111In-DTPA is 1/2 excreted with normal renal function?

8 minutes16 minutes36 minutes1.7 hours

http://www.doseinfo-radar.com/

Renogram

Some Rules for Good Questions (1/2)

No negatives (not, except, none)No absolutes (always, never, all, every, only, must)No overlapping rangesNo “all of the above” or “none of the above”No mutually exclusive answers (one is correct)Logical order (numerical, alphabetical, category)Symmetric answers

Some Rules for Good Questions (2/2)

Put concepts and common words in stemDistractors should be reasonableTry to use higher taxonomy

Often easier to write distractorsSome fact questions are appropriate

Single concept per questionNo multiple true / false questionsMedically, not question, complexity

Reference Guidelines

3-page guideABNM_Well_Formatted_SAM_Question.docx

1-page list of does and don’tsABNM_QuestionWritingDoesAndDonts.docx

26-page guide (ABNM psychometric consultants)http://www.measurementresearch.com/

24-page guideABR_Item_Writers_Guide.pdf

Applause