Embed Size (px)
Citation preview
ABR Exams of the Future Examination Details Impact on Training
Duane Mezwa, MD, FACR ABR Trustee, GI Section
Professor and Chair, Radiology Oakland University William Beaumont SOM
Disclosure
The material presented today best represents the current thoughts and processes in developing the Core and Certifying exams of the ABR.
What We’ve Done Until Now Two comprehensive written examinations
Physics (first offered September of 2nd year) Diagnostic (first offered September of 3rd year) Candidates must pass both to qualify.
One comprehensive oral examination First offered in June of 4th year 11 categories examined
Written Examinations: Details
Physics Diagnostic, Nuclear, Radiobiology Recall and application of physics concepts No associated images
Diagnostic ~20 MCQs in each category Evaluated information recall No associated images Scored as a whole (not by category)
Written Examinations: More Details
Criterion-based exams Theoretically, failure rate could be 0-100%. Failure rates (first-time takers):
Physics: 10-15% Diagnostic: 8-10%
Both exams offered retake opportunities.
Oral Examinations: Details
25 minute oral examinations Centered on specific case images
Minimal fact recall Focus on observation, synthesis, management Communication skills and judgment paramount
Specific diagnosis less important than logical approach
Oral Examination: More Details
Criterion-based exam Complex grading system
Each candidate discussed in detail Potential for raising marginally failing scores Could fail 1-3 categories, but must re-test
Statistics for first-time takers: Pass: 85% Fail: 5-7% Condition: 8-10%
Criticisms of Examination Process: Written Exams
Physics examination is better suited to physics students, not radiologists. Rarely relevant to everyday practice Diagnostic exam:
No images Fact retrieval (trivia?) only Not scored categorically—candidates could pass
knowing nothing about (fill in the blank).
Criticisms of Examination Process: Oral Exams
Subjective No two tests the same For some candidates, pathologically stressful Exam reliability incalculable
Expensive For ABR: ~400 examiners X 5 days For candidates: single inconvenient location
Response by the ABR: Changes in Exam Structure
Qualifying (Core) exam: Comprehensive:
Covers all of diagnostic radiology Includes physics, patient safety, noninterpretive skills
Categorical: candidates must pass all sections. Candidates and training programs receive data. Opportunities for retakes: every six months
Knowledge level: basic to intermediate Appropriate for ending third-year resident Timed for June of third year (except 2013) About 40% fact recall, 60% higher-level evaluation
R1 R2 R3 R4 Fellowship/
employment
Internship
12 mos 12 mos 12 mos 12 mos 12 mos 12 mos 3 mos
Core Exam Certifying Exam
Core Exam in Diagnostic Radiology
Will be given September 30-October 4, 2013 Subsequent years will be offered in late June. Last full Oral exam in June 2013
Covers all of diagnostic radiology Comprehensive, categorical exam Candidates must pass all categories.
What Are These Categories?
Organ
systems
MSK, Thoracic, GI, Urinary, Neuro, Pediatrics, Cardiac,
Reproductive/Endocrine, Mammography, Vascular
Modalities CT, MRI, RF/Fluoro, Nucs, Interventional, Ultrasound
Fundamentals Physics, Safety
Exam Goals: Core Overarching goal: to protect the public by determining that
individual candidates have attained competence in basic diagnostic radiology
Specific goal: to create examinations that are relevant to current radiologic practice
Specific goal: to make a reliable pass/fail decision about candidate performance in each category For this high-stakes exam, reliability requires ~60 questions/category. Emphasis on unique importance of Physics content = 110 questions
Breast Cardiac GI MSK Neuro Peds Thorax
Repro /
Endo Urinary Vascular Q#
CT 60
IR 60
MR 60
NM/Molecular 60
Rad/Fluoro 60
US 60
Physics 110
Safety 60
Q# 60 60 60 60 60 60 60 60 60 60
Core Exam: Content Image-rich (unlike current written) ~40% fact recall (like current written) ~60% higher level (like current oral)
Differential diagnosis Management Mostly MCQs, may include new question types
Example: Extended Matching For each patient whose clinical and imaging information is shown, select the most
appropriate diagnosis from the list below. Each option may be used once, more than once, or not at all.
A. Focal nodular hyperplasia B. Liver cell adenoma
C. Cavernous hemangioma D. Inflammatory pseudotumor
E. Pyogenic abscess F. Fungal abscess
G. Nodular focal fat H. Biliary cystadenoma
I. Lymphoma J. Solitary metastasis
K. Intrahepatic cholangiocarcinoma L. Hepatocellular carcinoma
M. Fibrolamellar carcinoma
1: 35 year old woman who underwent sonographic evaluation for mild abdominal discomfort. She was referred for MR imaging to characterize a solitary liver mass seen on that ultrasound. Images are obtained 30 seconds (A), 70 seconds (B) and 1 hour (C) after administration of gadobenate intravenously.
A B C
A. Focal nodular hyperplasia B. Liver cell adenoma
C. Cavernous hemangioma D. Inflammatory pseudotumor
E. Pyogenic abscess F. Fungal abscess
G. Nodular focal fat H. Biliary cystadenoma
I. Lymphoma J. Solitary metastasis
K. Intrahepatic cholangiocarcinoma L. Hepatocellular carcinoma
M. Fibrolamellar carcinoma Key = A
2. 60 year old man with abdominal pain. Imaging performed elsewhere showed a liver mass, and he is referred for CT imaging to characterize it. Images are obtained before (A), 20 seconds after (B), 50 seconds after (C), and 5 minutes after (D) intravenous administration of iodinated contrast material.
A. Focal nodular hyperplasia B. Liver cell adenoma
C. Cavernous hemangioma D. Inflammatory pseudotumor
E. Pyogenic abscess F. Fungal abscess
G. Nodular focal fat H. Biliary cystadenoma
I. Lymphoma J. Solitary metastasis
K. Intrahepatic cholangiocarcinoma L. Hepatocellular carcinoma
M. Fibrolamellar carcinoma
Key = L
A coronal CT image obtained in a patient with ascites is shown.
Label the following structures: a) Left subphrenic space b) Lesser sac, inferior recess c) Lesser sac, superior recess d) Transverse mesocolon e) Gastrohepatic ligament f) Morison’s pouch g) Left paracolic gutter h) Root of intestinal mesentery
a b c d g e h f
Core Exam 2013+ Nuts and Bolts
~660 items Two half-day sessions during a single week Examination center(s)
One large Chicago center Smaller center in Tucson ABR commitment: distributed exam by 2018
Condition exams in June/October
5440 N. Cumberland Ave., 4½ miles and the second train stop
on the CTA Blue Line from O’Hare Airport. ~30,000 square feet
ABR FastPass Palm Vein Biometrics
Results: Test Scoring Each category (18) will be scored separately.
Score will be based on pre-test assessment of item difficulty (Angoff method).
Each test item will be scrutinized for its performance (p-value, r-value), and adjustments made for problem items.
Three possible results: Pass (all 18 categories passed)—qualify for Certifying Exam Fail (6 or more categories failed)—must retake whole exam Condition (1 to 5 categories failed)—must retake those categories failed
Core Examination: Impact on Training
Candidates must be exposed to all basic diagnostic radiology by end of third year.
“Core anxiety” and review sessions will occur early in third year.
Content of review sessions will change.
Core Exam: Review Sessions Physics content
Should focus on practical applications. Physics resource: rsna.org/education/physics.cfm
Diagnostic content 40% fact recall—even though image-rich, it’s reasonable to
study topics prevalent on previous written exam. 60% resembles oral boards—standard board review. Resource: Core study guide on theABR.org Resource to come: practice exam—90 minute simulation
Practice Exam • A taste of the full exam • Shortened version of all sections • A WIP • To be on line by end of 2012 • 90 minutes -- to simulate the real exam
RSNA/AAPM Physics Modules 46 complete, 1 in progress 30,081 enrolled, 10,467 completed Popular modules:
Basic Radiation Biology Basic Concepts in Radiography Radiation Effects Basic Principles of Nuclear Magnetic Resonance Atoms, Radiation and Radioactivity Radiation Risks
BEST Way to Prepare
Come to work and PARTICIPATE Fully
Piloting the Core Exam First core pilot: May 2011 Aims:
Nuts and bolts: software, image quality, time allotted, interface
Psychometric item data: p, r values; reliability
First Core Pilot Population: Residents taking Oral Exam Incentive: Passing score on electronic exam
could raise condition in that category. Caveat: Would not raise if 68 on Ora.l Would not raise if failed ≥ 4 categories.
Only six categories: MSK, Neuro, VIR, Peds, Breast, and Cardiac
Core Pilot 2011: Results 1,117 (81.5%) took exam. Neuro 26%, Breast 7%, rest ~18% p-value ranged from 0.66–0.81 (mean 0.72) Average passing score 69.2% Post-test survey: good questions, enough
time, appropriate difficulty, interface good
Core Pilot 2012 • Will again be given at this year’s May Oral exams.
• Two categories will be offered.
• One will be chosen by ABR, and other is candidate’s choice.
• Can be used to raise 2 sections in Orals.
Next Expanded Core Pilot • Full Exam to be given to PGY4 residents in ~ June 2013.
• Will be given at both Chicago and Tucson.
• Will try to accommodate all residents.
• Feedback will be given to help guide exam to be taken in September.
• Does NOT Count in place of real exam.
• For now a one-time only event in 2013
Core Exam: How Are We Doing?
13 item-writing committees (physicist embedded in each)
Scorable units submitted: 5580 Scorable units approved: 3284 All study guides completed and revised X 1. First test assembly meeting: January 2012
Exam Week Logistics
After the Core Candidates can decide (within constraints of
their chosen training program) on their future practice. General vs. subspecialized Which subspecialty(ies)
Begin the process of continuous learning.
Certifying Exam in Diagnostic Radiology
Will be first administered in October 2015 Contains five parts (but questions will appear in
random order) Noninterpretive skills Things every physician
should know
Essentials Things every diagnostic radiologist should know
Clinical Practice Areas (3 self-selected CPAs)
Things this specific radiologist should know
Clinical Practice Areas (CPA) Twelve categories
Organ system: MSK, Cardiac, Thoracic, GI, Urinary, Neuro, Peds, Breast
Technology: US, VIR, Nuclear Radiology General
Candidates can choose any combination of CPAs. Items will vary in both difficulty and scope.
Level 1: fundamental Level 2: advanced
Candidates selecting a CPA more than once will receive a higher proportion of level 2 items.
Certifying Exam: Content (CPA) Emulates clinical practice
Will include normals and variants. Will include important findings outside chosen area. Appropriateness, clinical vignettes, management decisions
Item types Familiar: MCQs, extended matching Unfamiliar*: Structured reporting, script concordance testing
*Examples posted on ABR Website >1 year before use
Certifying Exam: Goals To confirm candidate has acquired and
maintained necessary skills to practice independently
Dual role Final ABR Certification Exam First exam of practice-based learning (like MOC)
Certifying Exam: Nuts and Bolts ~300 items 5-hour exam (one half-day) Exam center(s): Chicago, Tucson Exams offered October/February
Certifying Examination: Scoring
Each of the five modules scored separately. Passing standard established prior to test
administration (Angoff method). All problem items analyzed, adjustments made. Two possible results:
Pass: all 5 modules passed—certified 10 years Fail: any module failed Retakes possible every six months
Certifying Exam: Impact on Training
After Core passed, some candidates may begin self-directed specialty training. Some part of 4th year available. Extent depends on program needs. Supplement, but will not replace Fellowship.
May impact first 3 months of practice.
Certifying Examination: Candidate Preparation
Good professional practice Medical judgment Communication skills Lifelong learning Critical thinking Self-assessment
Recognize the six competencies? Study Guides
Certifying Exam: Resources Noninterpretive skills and Essentials
Study guides to be posted on theABR.org, Clinical practice modules
Study guides at theABR.org CME, SAMs Practice-based learning Society-produced educational modules
Certifying Exam: How Are We Doing?
15 item-writing committees Submitted scorable units: 4439 Approved scorable units: 3323
