MCQ’s 2: Technical Flaws to Avoid. MCQ Check-list This presentation will focus on: Why we write them Why we write them What type is best What type is

MCQ’s 2: MCQ’s 2: Technical Flaws to AvoidTechnical Flaws to Avoid

MCQ Check-listMCQ Check-list

This presentation will focus on:This presentation will focus on: Why we write themWhy we write them What type is bestWhat type is best What flaws are to be avoidedWhat flaws are to be avoided What content to testWhat content to test How to construct itHow to construct it

Writing MCQ’sWriting MCQ’s

At the end of this session, the At the end of this session, the participant will be able to participant will be able to

Describe the structure of an MCQDescribe the structure of an MCQ List the technical flaws common to List the technical flaws common to

MCQ’sMCQ’s

Why do we test?Why do we test?

Formative assessment Formative assessment Communicate to the students what material is Communicate to the students what material is

importantimportant Identify areas of deficiency in need of Identify areas of deficiency in need of

remediation or further learningremediation or further learning Summative assessmentSummative assessment

Determine final gradesDetermine final grades Make promotion decisionsMake promotion decisions

BothBoth Motivate students to studyMotivate students to study Identify areas where the course/curriculum is Identify areas where the course/curriculum is

weakweak

Anatomy of an MCQ ItemAnatomy of an MCQ Item

StemStem Lead-inLead-in OptionsOptions

One correct answerOne correct answer DistractorsDistractors

Types of MCQ’sTypes of MCQ’s

True/False familyTrue/False family Assess recall of isolated factsAssess recall of isolated facts

Stems must be clear and unambiguousStems must be clear and unambiguous Options must be absolutely true or false or Options must be absolutely true or false or

examinee must decide examinee must decide how truehow true options are options are

Generally not recommended – usually Generally not recommended – usually have subtle flawshave subtle flaws

Types of MCQ’s: True/FalseTypes of MCQ’s: True/False

Which of the following statements is true regarding Which of the following statements is true regarding contraception?contraception?

A)A) oral contraceptives should not be initiated until oral contraceptives should not be initiated until six weeks after therapeutic abortion to avoid the six weeks after therapeutic abortion to avoid the risk of hypercoagulabilityrisk of hypercoagulability

B)B) an intrauterine device can provide good an intrauterine device can provide good contraceptive protection 5 years after insertioncontraceptive protection 5 years after insertion

C)C) oral contraceptives should not be prescribed to oral contraceptives should not be prescribed to individuals under the age of 15 without parental individuals under the age of 15 without parental consentconsent

D)D) lesbian women do not require counseling about lesbian women do not require counseling about contraception or sexually transmitted diseasescontraception or sexually transmitted diseases

E)E) oral contraceptives should not be prescribed for oral contraceptives should not be prescribed for women overwomen over the age of 35the age of 35

Types of MCQ’sTypes of MCQ’s

One-Best-Answer family (A type)One-Best-Answer family (A type) Options may be diagrammed:Options may be diagrammed:

D C A E BD C A E B

Correct answer is the Correct answer is the most likelymost likely correct correct Better application of knowledge, Better application of knowledge,

integration, synthesis and judgmentintegration, synthesis and judgment

Least correct Most correct

Types of MCQ’s: Types of MCQ’s: One-Best-Answer familyOne-Best-Answer family

A 24 year old university student came to see you A 24 year old university student came to see you because she is missing several days of school each because she is missing several days of school each month because of severe dysmenorrhea month because of severe dysmenorrhea unresponsive to the OCP and NSAIDS. She also unresponsive to the OCP and NSAIDS. She also complains of deep dyspareunia and has a tender complains of deep dyspareunia and has a tender uterosacral nodule on pelvic examination. Which uterosacral nodule on pelvic examination. Which one of the following conditions is most likely one of the following conditions is most likely responsible for her symptoms?responsible for her symptoms?

A)A) Chronic pelvic inflammatory diseaseChronic pelvic inflammatory diseaseB)B) EndometriosisEndometriosisC)C) AdenomyosisAdenomyosisD)D) FibroidsFibroids

MCQ ‘Items’MCQ ‘Items’

Should test important information – Should test important information – either common or life-threateningeither common or life-threatening

Should be well-structuredShould be well-structured Flaws benefit test-wise examineesFlaws benefit test-wise examinees Irrelevant difficulty does not test Irrelevant difficulty does not test

examinees’ specific knowledgeexaminees’ specific knowledge

MCQ Technical FlawsMCQ Technical Flaws

Issues Related to Test-Wiseness Issues Related to Test-Wiseness Issues Related to Irrelevant DifficultyIssues Related to Irrelevant Difficulty


Issues Related to Test-WisenessIssues Related to Test-Wiseness provide clues to the right answer provide clues to the right answer examinee can eliminate options based examinee can eliminate options based

on flaws and not on knowledgeon flaws and not on knowledge

Issues Related to Irrelevant DifficultyIssues Related to Irrelevant Difficulty

Item Flaws: Test-wisenessItem Flaws: Test-wiseness

Grammatical cuesGrammatical cues Logical cuesLogical cues Absolute termsAbsolute terms Long correct answerLong correct answer Word repeatsWord repeats Convergence strategyConvergence strategy


Grammatical cuesGrammatical cues One or more distractors don’t follow One or more distractors don’t follow

grammatically from the stem grammatically from the stem


Grammatical cuesGrammatical cuesA 24 year old patient presents with a one year A 24 year old patient presents with a one year

history of dysmenorrhea, mid-cycle pain, history of dysmenorrhea, mid-cycle pain, dyspareunia, and painful bowel movements dyspareunia, and painful bowel movements with her periods. The most likely diagnosis with her periods. The most likely diagnosis is:is:

A)A) Fibroid uterusFibroid uterus

B)B) AdenomyosisAdenomyosis

C)C) EndometriosisEndometriosis

D)D) Psychosomatic conditionPsychosomatic condition

Item Flaws: Test-wiseness Item Flaws: Test-wiseness

Logical cuesLogical cues A subset of the options are A subset of the options are

collectively exhaustivecollectively exhaustive


Logical cuesLogical cues Arterial blood gases in a normal pregnant Arterial blood gases in a normal pregnant

woman, compared to her non-pregnant woman, compared to her non-pregnant state may show which of the following?state may show which of the following?

A)A) Decreased pODecreased pO22

B)B) Increased pCOIncreased pCO22

C)C) Decreased pCODecreased pCO22

D)D) Decreased pHDecreased pH

E)E) No difference in pCONo difference in pCO22


Logical cuesLogical cues One or more distractors don’t follow One or more distractors don’t follow

logicallylogically from the stem from the stem


Logical cuesLogical cuesAn important tool for assessing fetal well-being An important tool for assessing fetal well-being

is the Non Stress test. A well baby is the Non Stress test. A well baby demonstrates the following fetal heart rate demonstrates the following fetal heart rate pattern:pattern:

A)A) A baseline heart rate above 120 bpmA baseline heart rate above 120 bpmB)B) Evidence of fetal movementEvidence of fetal movementC)C) Baseline variabilityBaseline variabilityD)D) Two fetal heart rate accelerations in 20 Two fetal heart rate accelerations in 20

minutesminutesE)E) Fetal breathing movementsFetal breathing movements


Absolute termsAbsolute terms Always, neverAlways, never Not a problem if stem is long and Not a problem if stem is long and

lead-in asks a precise questionlead-in asks a precise question


Absolute termsAbsolute terms

In patients with a previous Cesarean section:In patients with a previous Cesarean section:A)A) Vaginal birth is NEVER an optionVaginal birth is NEVER an option

B)B) Repeat Cesarean section is less dangerous to Repeat Cesarean section is less dangerous to the motherthe mother

C)C) The uterine scar may rupture during labourThe uterine scar may rupture during labour

D)D) Care should ALWAYS be supervised by an Care should ALWAYS be supervised by an obstetricianobstetrician

E)E) No further pregnancies should EVER be No further pregnancies should EVER be consideredconsidered


Long correct answerLong correct answer Correct answer is longer, more Correct answer is longer, more

specific, or more complete than other specific, or more complete than other optionsoptions


Long correct answerLong correct answer Which of the following problems in Which of the following problems in

vaginal breech delivery is most likely to vaginal breech delivery is most likely to cause serious long-term complications?cause serious long-term complications?

A)A) Failure to descendFailure to descendB)B) Cord prolapseCord prolapseC)C) Nuchal armsNuchal armsD)D) Entrapment of the after coming head with Entrapment of the after coming head with

an incompletely dilated cervixan incompletely dilated cervixE)E) 1 minute Apgar score of 41 minute Apgar score of 4


Word repeatsWord repeats A word or phase is included in the A word or phase is included in the

stem and in the correct answerstem and in the correct answer


Word repeatsWord repeats A 24 year old with a previous history A 24 year old with a previous history

of chronic PID presents with primary of chronic PID presents with primary infertility. Select the procedure that infertility. Select the procedure that would be most appropriate to evaluate would be most appropriate to evaluate the status of her the status of her tubestubes and pelvic and pelvic anatomyanatomy

A)A) Post-coital testPost-coital testB)B) Endometrial biopsyEndometrial biopsyC)C) HysterosalpingogramHysterosalpingogramD)D) Laparoscopy with dye insufflation of the Laparoscopy with dye insufflation of the

fallopian tubesfallopian tubes


Convergence strategyConvergence strategy The correct answer includes the most The correct answer includes the most

elements in common with the other elements in common with the other optionsoptions


Convergence strategyConvergence strategyNormal bladder neck mobility is measured Normal bladder neck mobility is measured by:by:

A)A) A Q-tip angle measurement under 40 degreesA Q-tip angle measurement under 40 degrees

B)B) A Q-tip angle measurement under 30 degreesA Q-tip angle measurement under 30 degrees

C)C) A bladder neck ultrasound angle less than 30 A bladder neck ultrasound angle less than 30 degreesdegrees

D)D) A Q-tip angle measurement under 20 degreesA Q-tip angle measurement under 20 degrees


Issues Related to Test-Wiseness Issues Related to Test-Wiseness

Issues Related to Irrelevant DifficultyIssues Related to Irrelevant Difficulty do not assess knowledge of intended do not assess knowledge of intended

topictopic

Irrelevant DifficultyIrrelevant Difficulty

Options are long, complicated or doubleOptions are long, complicated or double Extraneous details in stemExtraneous details in stem Numeric data not stated consistentlyNumeric data not stated consistently Vague frequency termsVague frequency terms Non-parallel language in optionsNon-parallel language in options Stems are tricky or unnecessarily Stems are tricky or unnecessarily

complicatedcomplicated ‘‘None of the above’None of the above’


Options are long, complicated or Options are long, complicated or doubledouble

Extraneous details in stemExtraneous details in stem Shifts difficulty from content Shifts difficulty from content

knowledge to speed readingknowledge to speed reading

Irrelevant DifficultyIrrelevant DifficultyOptions are long, complicated or doubleOptions are long, complicated or double Mrs. Brown is a 70 year old woman who had a stroke Mrs. Brown is a 70 year old woman who had a stroke

6 months ago and is still impaired. Her daughter tells 6 months ago and is still impaired. Her daughter tells you that her mother’s recent urinary incontinence you that her mother’s recent urinary incontinence makes her care unmanageable. She is seeking makes her care unmanageable. She is seeking permanent placement. Mrs. Brown complains of permanent placement. Mrs. Brown complains of urinary frequency, nocturia, and urge incontinence. urinary frequency, nocturia, and urge incontinence. Which of the following statements is true?Which of the following statements is true?

A)A) Neurological conditions are involved in the majority of Neurological conditions are involved in the majority of women with symptoms like those presented by Mrs. Brownwomen with symptoms like those presented by Mrs. Brown

B)B) Diabetes has a greater impact on quality of life than Mrs. Diabetes has a greater impact on quality of life than Mrs. Brown’s condition.Brown’s condition.

C)C) The pharmacological blockage of muscarinic receptors in The pharmacological blockage of muscarinic receptors in the detrusor muscle controls symptoms compatible with the detrusor muscle controls symptoms compatible with Mrs. Brown’s conditionMrs. Brown’s condition

D)D) Overactive bladder is defined as urinary incontinence Overactive bladder is defined as urinary incontinence accompanied by or immediately preceded by urgencyaccompanied by or immediately preceded by urgency

E)E) Alpha stimulants are effective in treating this conditionAlpha stimulants are effective in treating this condition

Irrelevant DifficultyIrrelevant DifficultyExtraneous details in stemExtraneous details in stem You see in your office a 24 year old woman for her routine You see in your office a 24 year old woman for her routine

postpartum visit, at 6 weeks. Her baby is irritable and cries postpartum visit, at 6 weeks. Her baby is irritable and cries “all the time’ (including during her visit.) As she is changing “all the time’ (including during her visit.) As she is changing the diaper during her visit, you notice that the child has a very the diaper during her visit, you notice that the child has a very red bum but upon questioning, she tells you she did not notice red bum but upon questioning, she tells you she did not notice anything abnormal. Her partner is away in Afghanistan and anything abnormal. Her partner is away in Afghanistan and she does not have a lot of friends, as they just recently were she does not have a lot of friends, as they just recently were posted in Kingston. She states that she has a hard time posted in Kingston. She states that she has a hard time getting dressed in the morning, cries easily and feels like she getting dressed in the morning, cries easily and feels like she does not love her child. She denies any medical or psychiatric does not love her child. She denies any medical or psychiatric history. Her exam is essentially normal. She weighs 2 kg less history. Her exam is essentially normal. She weighs 2 kg less than at her first antenatal visit at 10 weeks. You most likely than at her first antenatal visit at 10 weeks. You most likely clinical diagnosis is:clinical diagnosis is:

A)A) Postpartum hypothyroidismPostpartum hypothyroidism

B)B) HyperthyroidismHyperthyroidism

C)C) Postpartum bluesPostpartum blues

D)D) Postpartum depressionPostpartum depression

E)E) Psychotic depressionPsychotic depression

Irrelevant Difficulty Irrelevant Difficulty

Numeric DataNumeric Data Not stated consistentlyNot stated consistently Mixed formatMixed format Illogical orderIllogical order


Numeric Data not stated consistentlyNumeric Data not stated consistently Following a second episode of salpingitis, Following a second episode of salpingitis,

what is the likelihood that a woman is what is the likelihood that a woman is infertile?infertile?

A)A) 20 – 30 % 20 – 30 %

B)B) Less than 20%Less than 20%

C)C) Greater than 50 %Greater than 50 %

D)D) 90%90%

E)E) 75%75%


Vague frequency termsVague frequency terms Often, usually, rarely, frequently Often, usually, rarely, frequently Not consistently definedNot consistently defined

*may be used in stem if otherwise *may be used in stem if otherwise carefully constructed unambiguous carefully constructed unambiguous questionquestion


Vague frequency termsVague frequency terms

Gonorrhea isGonorrhea is A) sometimes asymptomaticA) sometimes asymptomatic

B) treated with cefixime 450 mg orallyB) treated with cefixime 450 mg orally

C) caused by a gram positive intracellular C) caused by a gram positive intracellular diplococcusdiplococcus

D) diagnosed by a DNA testD) diagnosed by a DNA test

E) usually associated with vulvar ulcersE) usually associated with vulvar ulcers


Non-parallel language in optionsNon-parallel language in options Non-logical orderNon-logical order


Non-parallel language in optionsNon-parallel language in optionsIn a vaccine trial, 200 2-year-old boys were given a In a vaccine trial, 200 2-year-old boys were given a

vaccine against a certain disease and then vaccine against a certain disease and then monitored for five years for occurrence of the monitored for five years for occurrence of the disease. Of this group, 85% never contracted the disease. Of this group, 85% never contracted the disease. Which of the following statements disease. Which of the following statements concerning these results is correct?concerning these results is correct?

A)A) No conclusion can be drawn, since no follow-up No conclusion can be drawn, since no follow-up was made of non-vaccinated childrenwas made of non-vaccinated children

B)B) The number of cases (ie 30 cases over five The number of cases (ie 30 cases over five year) is too small for statistically meaningful year) is too small for statistically meaningful conclusionsconclusions

C)C) It is not possible to draw a conclusion because It is not possible to draw a conclusion because the trial involved only boysthe trial involved only boys

D)D) Vaccine efficacy (%) is calculated as 85-15/100Vaccine efficacy (%) is calculated as 85-15/100


““None of the above”None of the above” Problematic where judgment is Problematic where judgment is

involved or if options not absolutely involved or if options not absolutely true or falsetrue or false

Turns question into true/false Turns question into true/false questionquestion

Confuses very knowledgeable Confuses very knowledgeable studentsstudents


““None of the above”None of the above” The main reason that fertility is reduced The main reason that fertility is reduced

in a woman in her early 40’s is:in a woman in her early 40’s is:A)A) Irregular cycles make it hard to time Irregular cycles make it hard to time

intercourseintercourseB)B) Chromosomal damage adversely affects Chromosomal damage adversely affects

oocytes’ genetic complementoocytes’ genetic complementC)C) Male erectile difficulties increaseMale erectile difficulties increaseD)D) Hormone imbalance leads to poor endometrial Hormone imbalance leads to poor endometrial

developmentdevelopmentE)E) None of the aboveNone of the above

Irrelevant DifficultyIrrelevant DifficultyStems are tricky or unnecessarily complicatedStems are tricky or unnecessarily complicated Arrange the parents of the following children with Down’s Arrange the parents of the following children with Down’s

syndrome in order of the highest to lowest risk of recurrence. syndrome in order of the highest to lowest risk of recurrence. Assume that the maternal age in all cases is 22 years and Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are:karyotypes of the daughters are:

I: 46, XX, –14,+T (14q21q) patI: 46, XX, –14,+T (14q21q) patII: 46, XX, –14,+T (14q21q) de novoII: 46, XX, –14,+T (14q21q) de novoIII: 46, XX, –14,+T (14q21q) matIII: 46, XX, –14,+T (14q21q) matIV: 46, XX, –21,+T (14q21q) patIV: 46, XX, –21,+T (14q21q) patV: 46, XX, –21,+T (21q21q) (parents not typed)V: 46, XX, –21,+T (21q21q) (parents not typed)

A)A) III, IV, I, V, IIIII, IV, I, V, IIB)B) IV, III, V, 1, IIIV, III, V, 1, IIC)C) III, I, IV, V, IIIII, I, IV, V, IID)D) I, V, III, I, V, III, V, III, I, V, IIE)E) III, IV, I, II, VIII, IV, I, II, V

MCQ Check-listMCQ Check-list

Why we write themWhy we write them What type is bestWhat type is best What flaws are to be avoidedWhat flaws are to be avoided What content to testWhat content to test How to construct itHow to construct it

General Guidelines for Item General Guidelines for Item ConstructionConstruction Include as much of the item as possible in Include as much of the item as possible in

the stems: Long stems, short optionsthe stems: Long stems, short options Avoid negatively phrased items (Avoid negatively phrased items (exceptexcept or or

notnot in the lead-in). If you must, use only in the lead-in). If you must, use only with short optionswith short options

Avoid writing questions of the form:Avoid writing questions of the form: Which of the following statements is correct?Which of the following statements is correct? Each of the following statements is correct Each of the following statements is correct

EXCEPTEXCEPT Unfocussed Unfocussed Heterogeneous optionsHeterogeneous options

Basic Rules for MCQ’s (A type)*Basic Rules for MCQ’s (A type)*

Focus item on important conceptFocus item on important concept Item should assess application of Item should assess application of

knowledge, not recall of an isolated factknowledge, not recall of an isolated fact Stem must pose clear question – ‘Cover Stem must pose clear question – ‘Cover

the options’ testthe options’ test All distractors should be homogeneousAll distractors should be homogeneous Avoid technical item flawsAvoid technical item flaws

ReferenceReference

Constructing Written Test Questions Constructing Written Test Questions For the Basic and Clinical SciencesFor the Basic and Clinical Sciences Case & Swanson, NBME 1996Case & Swanson, NBME 1996

One last step…One last step…

Get a colleague to read over your Get a colleague to read over your questions questions Detects more than one right answerDetects more than one right answer Discovers the ‘what am I thinking?’ Discovers the ‘what am I thinking?’

questionquestion Identifies not enough information in the Identifies not enough information in the

stemstem ……

Documents

MCQ’s 2: Technical Flaws to Avoid. MCQ Check-list This presentation will focus on: Why we write them Why we write them What type is best What type is