Embed Size (px)
Citation preview
Outline Motivation. . . . .ItemSpecifications PacingandReporting
.
......
ImprovingStudentAssessmentinthePre-clinicalYears
TimRyan, LeelaSarathy, AdamLuce, AlecPeniche, BjornWatsjold
March13, 2012
Outline Motivation. . . . .ItemSpecifications PacingandReporting
...1 Motivation
...2 ItemSpecificationsGeneralDescriptionPromptAttributesResponseAttributesProhibitedItemTypesSampleItem
...3 PacingandReporting
Outline Motivation. . . . .ItemSpecifications PacingandReporting
Motivation
StudentsdesirehighUSMLE Step1scores.
Currentfailrateisofconcern.
Testwritersdeserveguidance.
.ItemFormat..
......
USMLE-styleexamstoimprovetestscores
Othertest-takingskillsnotimportant
Changecomesatlittlecost
Notochangeincoursecontent
USMLE-styleexamsmayhelpreducetestanxiety.
.ExamTiming..
......
Majorityofstudentsfinishexamsquicklyalready
Slowstudentsidentified before theytaketheUSMLE.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
Motivation
StudentsdesirehighUSMLE Step1scores.
Currentfailrateisofconcern.
Testwritersdeserveguidance.
.ItemFormat..
......
USMLE-styleexamstoimprovetestscores
Othertest-takingskillsnotimportant
Changecomesatlittlecost
Notochangeincoursecontent
USMLE-styleexamsmayhelpreducetestanxiety.
.ExamTiming..
......
Majorityofstudentsfinishexamsquicklyalready
Slowstudentsidentified before theytaketheUSMLE.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
Motivation
StudentsdesirehighUSMLE Step1scores.
Currentfailrateisofconcern.
Testwritersdeserveguidance.
.ItemFormat..
......
USMLE-styleexamstoimprovetestscores
Othertest-takingskillsnotimportant
Changecomesatlittlecost
Notochangeincoursecontent
USMLE-styleexamsmayhelpreducetestanxiety.
.ExamTiming..
......
Majorityofstudentsfinishexamsquicklyalready
Slowstudentsidentified before theytaketheUSMLE.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
Motivation
StudentsdesirehighUSMLE Step1scores.
Currentfailrateisofconcern.
Testwritersdeserveguidance.
.ItemFormat..
......
USMLE-styleexamstoimprovetestscores
Othertest-takingskillsnotimportant
Changecomesatlittlecost
Notochangeincoursecontent
USMLE-styleexamsmayhelpreducetestanxiety.
.ExamTiming..
......
Majorityofstudentsfinishexamsquicklyalready
Slowstudentsidentified before theytaketheUSMLE.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
Motivation
StudentsdesirehighUSMLE Step1scores.
Currentfailrateisofconcern.
Testwritersdeserveguidance.
.ItemFormat..
......
USMLE-styleexamstoimprovetestscores
Othertest-takingskillsnotimportant
Changecomesatlittlecost
Notochangeincoursecontent
USMLE-styleexamsmayhelpreducetestanxiety.
.ExamTiming..
......
Majorityofstudentsfinishexamsquicklyalready
Slowstudentsidentified before theytaketheUSMLE.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
Motivation
StudentsdesirehighUSMLE Step1scores.
Currentfailrateisofconcern.
Testwritersdeserveguidance.
.ItemFormat..
......
USMLE-styleexamstoimprovetestscores
Othertest-takingskillsnotimportant
Changecomesatlittlecost
Yestochangeinexamformat
USMLE-styleexamsmayhelpreducetestanxiety.
.ExamTiming..
......
Majorityofstudentsfinishexamsquicklyalready
Slowstudentsidentified before theytaketheUSMLE.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
Motivation
StudentsdesirehighUSMLE Step1scores.
Currentfailrateisofconcern.
Testwritersdeserveguidance.
.ItemFormat..
......
USMLE-styleexamstoimprovetestscores
Othertest-takingskillsnotimportant
Changecomesatlittlecost
Yestochangeinexamformat
USMLE-styleexamsmayhelpreducetestanxiety.
.ExamTiming..
......
Majorityofstudentsfinishexamsquicklyalready
Slowstudentsidentified before theytaketheUSMLE.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
Motivation
StudentsdesirehighUSMLE Step1scores.
Currentfailrateisofconcern.
Testwritersdeserveguidance.
.ItemFormat..
......
USMLE-styleexamstoimprovetestscores
Othertest-takingskillsnotimportant
Changecomesatlittlecost
Yestochangeinexamformat
USMLE-styleexamsmayhelpreducetestanxiety.
.ExamTiming..
......
Majorityofstudentsfinishexamsquicklyalready
Slowstudentsidentified before theytaketheUSMLE.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
Motivation
StudentsdesirehighUSMLE Step1scores.
Currentfailrateisofconcern.
Testwritersdeserveguidance.
.ItemFormat..
......
USMLE-styleexamstoimprovetestscores
Othertest-takingskillsnotimportant
Changecomesatlittlecost
Yestochangeinexamformat
USMLE-styleexamsmayhelpreducetestanxiety.
.ExamTiming..
......
Majorityofstudentsfinishexamsquicklyalready
Slowstudentsidentified before theytaketheUSMLE.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
GeneralDescription
.TestTasksonMajorExams:..
......
interpretgraphicandtabularmaterial,
identifygrossandmicroscopicpathologicandnormalspecimens,and
applybasicscienceknowledgetosolveclinicalproblems.
``Themajorityofquestionsrequiretheexamineetointerpretgraphicandtabularmaterial, toidentifygrossandmicroscopicpathologicandnormalspecimens, andtosolveproblemsthroughapplicationofbasicscienceprinciples.''1
1http://www.usmle.org/Examinations/step1/step1_content.html
Outline Motivation. . . . .ItemSpecifications PacingandReporting
GeneralDescription
.TestTasksonMajorExams:..
......
interpretgraphicandtabularmaterial,
identifygrossandmicroscopicpathologicandnormalspecimens,and
applybasicscienceknowledgetosolveclinicalproblems.
``Themajorityofquestionsrequiretheexamineetointerpretgraphicandtabularmaterial, toidentifygrossandmicroscopicpathologicandnormalspecimens, andtosolveproblemsthroughapplicationofbasicscienceprinciples.''1
1http://www.usmle.org/Examinations/step1/step1_content.html
Outline Motivation. . . . .ItemSpecifications PacingandReporting
GeneralDescription
.TestTasksonMajorExams:..
......
interpretgraphicandtabularmaterial,
identifygrossandmicroscopicpathologicandnormalspecimens,and
applybasicscienceknowledgetosolveclinicalproblems.
``Themajorityofquestionsrequiretheexamineetointerpretgraphicandtabularmaterial, toidentifygrossandmicroscopicpathologicandnormalspecimens, andtosolveproblemsthroughapplicationofbasicscienceprinciples.''1
1http://www.usmle.org/Examinations/step1/step1_content.html
Outline Motivation. . . . .ItemSpecifications PacingandReporting
GeneralDescription
.TestTasksonMajorExams:..
......
interpretgraphicandtabularmaterial,
identifygrossandmicroscopicpathologicandnormalspecimens,and
applybasicscienceknowledgetosolveclinicalproblems.
``Themajorityofquestionsrequiretheexamineetointerpretgraphicandtabularmaterial, toidentifygrossandmicroscopicpathologicandnormalspecimens, andtosolveproblemsthroughapplicationofbasicscienceprinciples.''1
1http://www.usmle.org/Examinations/step1/step1_content.html
Outline Motivation. . . . .ItemSpecifications PacingandReporting
PromptAttributes
Eachpromptwillbecomposedofaclinicalorexperimentalvignettefollowedby1-4questions..VignetteAttributes..
......
60-70words
Authentic
Graphics, tables, imagesorschematics
Oneormorepiecesofrelevantinformation
ExtraneousinformationOK,Falseinformationprohibited
.QuestionAttributes..
......
Identifyorrecallbasicinformation(Bloomlevel1)
Interpretinformationinthevignette(Bloomlevel2)
Applybasicinformationtothescenario(Bloomlevel3)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
PromptAttributes
Eachpromptwillbecomposedofaclinicalorexperimentalvignettefollowedby1-4questions..VignetteAttributes..
......
60-70words
Authentic
Graphics, tables, imagesorschematics
Oneormorepiecesofrelevantinformation
ExtraneousinformationOK,Falseinformationprohibited
.QuestionAttributes..
......
Identifyorrecallbasicinformation(Bloomlevel1)
Interpretinformationinthevignette(Bloomlevel2)
Applybasicinformationtothescenario(Bloomlevel3)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
PromptAttributes
Eachpromptwillbecomposedofaclinicalorexperimentalvignettefollowedby1-4questions..VignetteAttributes..
......
60-70words
Authentic
Graphics, tables, imagesorschematics
Oneormorepiecesofrelevantinformation
ExtraneousinformationOK,Falseinformationprohibited
.QuestionAttributes..
......
Identifyorrecallbasicinformation(Bloomlevel1)
Interpretinformationinthevignette(Bloomlevel2)
Applybasicinformationtothescenario(Bloomlevel3)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
PromptAttributes
Eachpromptwillbecomposedofaclinicalorexperimentalvignettefollowedby1-4questions..VignetteAttributes..
......
60-70words
Authentic
Graphics, tables, imagesorschematics
Oneormorepiecesofrelevantinformation
ExtraneousinformationOK,Falseinformationprohibited
.QuestionAttributes..
......
Identifyorrecallbasicinformation(Bloomlevel1)
Interpretinformationinthevignette(Bloomlevel2)
Applybasicinformationtothescenario(Bloomlevel3)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
PromptAttributes
Eachpromptwillbecomposedofaclinicalorexperimentalvignettefollowedby1-4questions..VignetteAttributes..
......
60-70words
Authentic
Graphics, tables, imagesorschematics
Oneormorepiecesofrelevantinformation
ExtraneousinformationOK,Falseinformationprohibited
.QuestionAttributes..
......
Identifyorrecallbasicinformation(Bloomlevel1)
Interpretinformationinthevignette(Bloomlevel2)
Applybasicinformationtothescenario(Bloomlevel3)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
PromptAttributes
Eachpromptwillbecomposedofaclinicalorexperimentalvignettefollowedby1-4questions..VignetteAttributes..
......
60-70words
Authentic
Graphics, tables, imagesorschematics
Oneormorepiecesofrelevantinformation
ExtraneousinformationOK,Falseinformationprohibited
.QuestionAttributes..
......
Identifyorrecallbasicinformation(Bloomlevel1)
Interpretinformationinthevignette(Bloomlevel2)
Applybasicinformationtothescenario(Bloomlevel3)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
PromptAttributes
Eachpromptwillbecomposedofaclinicalorexperimentalvignettefollowedby1-4questions..VignetteAttributes..
......
60-70words
Authentic
Graphics, tables, imagesorschematics
Oneormorepiecesofrelevantinformation
ExtraneousinformationOK,Falseinformationprohibited
.QuestionAttributes..
......
Identifyorrecallbasicinformation(Bloomlevel1)
Interpretinformationinthevignette(Bloomlevel2)
Applybasicinformationtothescenario(Bloomlevel3)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
PromptAttributes
Eachpromptwillbecomposedofaclinicalorexperimentalvignettefollowedby1-4questions..VignetteAttributes..
......
60-70words
Authentic
Graphics, tables, imagesorschematics
Oneormorepiecesofrelevantinformation
ExtraneousinformationOK,Falseinformationprohibited
.QuestionAttributes..
......
Identifyorrecallbasicinformation(Bloomlevel1)
Interpretinformationinthevignette(Bloomlevel2)
Applybasicinformationtothescenario(Bloomlevel3)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
PromptAttributes
Eachpromptwillbecomposedofaclinicalorexperimentalvignettefollowedby1-4questions..VignetteAttributes..
......
60-70words
Authentic
Graphics, tables, imagesorschematics
Oneormorepiecesofrelevantinformation
ExtraneousinformationOK,Falseinformationprohibited
.QuestionAttributes..
......
Identifyorrecallbasicinformation(Bloomlevel1)
Interpretinformationinthevignette(Bloomlevel2)
Applybasicinformationtothescenario(Bloomlevel3)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ResponseAttributes
.ResponseAttributes..
......
3-11responseoptionsa
Short(1-15words)
Lettered(A,B,C...)
Alphabeticalorlogicalorder
Distractorsmaybepartiallycorrect
Thereisonly onebest answer
a> 5 responseoptionspossiblewithExamSoft™CBT
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ResponseAttributes
.ResponseAttributes..
......
3-11responseoptionsa
Short(1-15words)
Lettered(A,B,C...)
Alphabeticalorlogicalorder
Distractorsmaybepartiallycorrect
Thereisonly onebest answer
a> 5 responseoptionspossiblewithExamSoft™CBT
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ResponseAttributes
.ResponseAttributes..
......
3-11responseoptionsa
Short(1-15words)
Lettered(A,B,C...)
Alphabeticalorlogicalorder
Distractorsmaybepartiallycorrect
Thereisonly onebest answer
a> 5 responseoptionspossiblewithExamSoft™CBT
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ResponseAttributes
.ResponseAttributes..
......
3-11responseoptionsa
Short(1-15words)
Lettered(A,B,C...)
Alphabeticalorlogicalorder
Distractorsmaybepartiallycorrect
Thereisonly onebest answer
a> 5 responseoptionspossiblewithExamSoft™CBT
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ResponseAttributes
.ResponseAttributes..
......
3-11responseoptionsa
Short(1-15words)
Lettered(A,B,C...)
Alphabeticalorlogicalorder
Distractorsmaybepartiallycorrect
Thereisonly onebest answer
a> 5 responseoptionspossiblewithExamSoft™CBT
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ResponseAttributes
.ResponseAttributes..
......
3-11responseoptionsa
Short(1-15words)
Lettered(A,B,C...)
Alphabeticalorlogicalorder
Distractorsmaybepartiallycorrect
Thereisonly onebest answer
a> 5 responseoptionspossiblewithExamSoft™CBT
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ProhibitedItemtypes
.Prohibiteditemtypesinclude, butarenotlimitedto, allofthefollowing..
......
Questionsaskingstudentstochoosethe worst answer.Allofthefollowingaretrue EXCEPTWhichofthefollowingis FALSE
Questionsaskingstudentstoselect multiplebest answers.AnswerchoicesB &C arecorrectAlloftheabove
Questionsthatstresstest-takingskillsnotrepresentedontheUSMLE (e.g. matching, ordering)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ProhibitedItemtypes
.Prohibiteditemtypesinclude, butarenotlimitedto, allofthefollowing..
......
Questionsaskingstudentstochoosethe worst answer.Allofthefollowingaretrue EXCEPTWhichofthefollowingis FALSE
Questionsaskingstudentstoselect multiplebest answers.AnswerchoicesB &C arecorrectAlloftheabove
Questionsthatstresstest-takingskillsnotrepresentedontheUSMLE (e.g. matching, ordering)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ProhibitedItemtypes
.Prohibiteditemtypesinclude, butarenotlimitedto, allofthefollowing..
......
Questionsaskingstudentstochoosethe worst answer.Allofthefollowingaretrue EXCEPTWhichofthefollowingis FALSE
Questionsaskingstudentstoselect multiplebest answers.AnswerchoicesB &C arecorrectAlloftheabove
Questionsthatstresstest-takingskillsnotrepresentedontheUSMLE (e.g. matching, ordering)
Outline Motivation. . . . .ItemSpecifications PacingandReporting
SampleItem
.Example..
......
A fifty-year-oldmanpresentstotheEmergencyDepartmentwithsudden, severe, left-sidedabdominalpain. Uponfurtherexamination,youconcludethatheissufferingfromanobstructionofhisabdominalvasculature, cuttingoffbloodsupplytopartofthecolon.1. Basedonyourknowledgeofbloodsupplyintheabdominalcavity,youdeterminethatanembolismismostlikelyinhis:
A. Celiacartery
B. InferiorMesentericArtery
C. InferiorVenaCava
D. Leftiliacartery
E. SuperiorMesentericArtery
Outline Motivation. . . . .ItemSpecifications PacingandReporting
SampleItem
.Example..
......
A fifty-year-oldmanpresentstotheEmergencyDepartmentwithsudden, severe, left-sidedabdominalpain. Uponfurtherexamination,youconcludethatheissufferingfromanobstructionofhisabdominalvasculature, cuttingoffbloodsupplytopartofthecolon.1. Basedonyourknowledgeofbloodsupplyintheabdominalcavity,youdeterminethatanembolismismostlikelyinhis:
A. Celiacartery
B. InferiorMesentericArtery
C. InferiorVenaCava
D. Leftiliacartery
E. SuperiorMesentericArtery
Outline Motivation. . . . .ItemSpecifications PacingandReporting
SampleItem
.Example..
......
A fifty-year-oldmanpresentstotheEmergencyDepartmentwithsudden, severe, left-sidedabdominalpain. Uponfurtherexamination,youconcludethatheissufferingfromanobstructionofhisabdominalvasculature, cuttingoffbloodsupplytopartofthecolon.1. Basedonyourknowledgeofbloodsupplyintheabdominalcavity,youdeterminethatanembolismismostlikelyinhis:
A. Celiacartery
B. InferiorMesentericArtery
C. InferiorVenaCava
D. Leftiliacartery
E. SuperiorMesentericArtery
Outline Motivation. . . . .ItemSpecifications PacingandReporting
SampleItem
.Example..
......
A fifty-year-oldmanpresentstotheEmergencyDepartmentwithsudden, severe, left-sidedabdominalpain. Uponfurtherexamination,youconcludethatheissufferingfromanobstructionofhisabdominalvasculature, cuttingoffbloodsupplytopartofthecolon.2. Acutemesentericischemiaismostoftencausedbyocclusionofthe:
A. Celiacartery
B. InferiorMesentericArtery
C. InferiorVenaCava
D. Leftiliacartery
E. SuperiorMesentericArtery
Outline Motivation. . . . .ItemSpecifications PacingandReporting
SampleItem
.Example..
......
A fifty-year-oldmanpresentstotheEmergencyDepartmentwithsudden, severe, left-sidedabdominalpain. Uponfurtherexamination,youconcludethatheissufferingfromanobstructionofhisabdominalvasculature, cuttingoffbloodsupplytopartofthecolon.2. Acutemesentericischemiaismostoftencausedbyocclusionofthe:
A. Celiacartery
B. InferiorMesentericArtery
C. InferiorVenaCava
D. Leftiliacartery
E. SuperiorMesentericArtery
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ExamPacingandReporting
TwoPacingModes.ExtendedTime..
......
90secondsperitem
40itemsperhour
BUSM I,semester2
.RegularTime..
......
72secondsperitem
50itemsperhour
BUSM II
.GradeReporting..
......
Allmajorexamsinallcourses
MedianandInterquartileRange
Mean, StandardDeviation,etc.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ExamPacingandReporting
TwoPacingModes.ExtendedTime..
......
90secondsperitem
40itemsperhour
BUSM I,semester2
.RegularTime..
......
72secondsperitem
50itemsperhour
BUSM II
.GradeReporting..
......
Allmajorexamsinallcourses
MedianandInterquartileRange
Mean, StandardDeviation,etc.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ExamPacingandReporting
TwoPacingModes.ExtendedTime..
......
90secondsperitem
40itemsperhour
BUSM I,semester2
.RegularTime..
......
72secondsperitem
50itemsperhour
BUSM II
.GradeReporting..
......
Allmajorexamsinallcourses
MedianandInterquartileRange
Mean, StandardDeviation,etc.
Outline Motivation. . . . .ItemSpecifications PacingandReporting
ExamPacingandReporting
TwoPacingModes.ExtendedTime..
......
90secondsperitem
40itemsperhour
BUSM I,semester2
.RegularTime..
......
72secondsperitem
50itemsperhour
BUSM II
.GradeReporting..
......
Allmajorexamsinallcourses
MedianandInterquartileRange
Mean, StandardDeviation,etc.
