Does resident recruitment ranking predict subsequent
performance?Jonathan Fryer, Noreen Corcoran,
Brian George, Ed Wang, Deb DaRosaNorthwestern University Feinberg School of Medicine,
Chicago Illinois
Screening of Applications
Assessment of
Candidates
Ranking of Candidates
Resident Selection Process
1) How effective is the ranking process in selecting residents who will perform well?
2) How effective is the ranking process in predicting subsequent resident performances?
3) How effective are “adjustments” made to the preliminary rank list in improving the selection of residents who will perform well?
Questions?
General surgery categorical resident recruitment between 2002-2011 inclusive (n=46).◦ 4 categorical residents 2002-2004.◦ 5 categorical residents 2005-2011.
Residents who dropped out (n=1) or who were recruited after the PGY1 year (n=2) were excluded from the analyses.
Methods
We compared how successful candidates were ranked during recruitment with their subsequent performance in our program.
Methods
1) USMLE˄ Scores alone 2) Unadjusted Ranking Score (URS˄):
based on sum of 3 assessment scores ◦ Academic Profile (Coordinators)
Medical school rank, USMLE Step 1, Class Rank, Honors in Surgery
◦ Program Director Review (PDs) Research experience, extracurricular/community
involvement, LORs, Personal Statement, Dean’s Letter
◦ Faculty Interview score (Faculty) Averaged for 2 independent faculty interview scores.
Recruitment Ranking Parameters
˄ Higher score is better
3) Final Adjusted Ranking (FAR˅)◦ Modification of the preliminary rank list generated
by the URS ◦ Based on additional insights about specific
candidates provided by the resident selection committee and/or leadership
Endorsements from trusted colleagues Negative interactions with staff Concerns raised by residents, coordinators Other?
Recruitment Ranking Parameters
˅ Lower is better
Screening Academic Profile *
Assessment
URS *Ranking
FAR
Resident Selection ProcessNUFSM
*USMLE included
1) ABSITE˄ percentile alone 2) Resident Evaluation Grade (REG˄)
◦ Semiannual evaluation scores (Letter grade: A-F)◦ Group discussion and grade assignment based on:
Clinical Evaluations (360°): faculty, peers, med students, nurses, patients, etc.
Compliance: evaluations, case log, duty hour log, conference attendance, etc.
ABSITE, Mock Oral, PAME scores.
Resident Performance Measures
˄ higher is better
3) Independent Faculty Rating/Ranking (IFRR˅) ◦ Confidential survey with faculty independently
rating all residents using a 7-point Likert scale and ranking resident within their PGY1 recruitment cohorts.
◦ * not part of standard resident evaluation at NU
Resident Performance Measures
˅ lower is better
Full IRB approval was obtained. All resident ranking and performance data
was de-identified after collection and aggregated to protect resident confidentiality.
Methods
Semiannual Resident Evaluation Grades (A-F) were converted to numerical values (5-0, respectively) and averaged for analyses.
Data from Individual Faculty Rating/Ranking surveys were averaged for individual residents.
Methods
Statistical Analyses performed using SAS 9.2 software (Cary, NC).◦ Associations between ranking and performance
parameters were analyzed using Spearman’s correlation coefficient.
◦ Comparison of ranking parameters between poor and satisfactory performance used student t-test.
◦ Differences in performance based on ranking range were compared using F-test.
Methods
Poor Resident Performance Parameter
# (%) comments
Drop outs 1 of 46 (2.2%) excellent evaluations
ABS QE/CE fail 0 of 13 (0%)
REG < C * 1 of 46 (2.2%) Probation
IFFR > 4.0 6 of 46 (13.0%) >4.0 = below average
ABSITE < 35% * 12 of 46 (26.1%)
ResultsOverall resident performance
* Occurring at any time during residency training
Recruitment Ranking
USMLE˄ URS˄ FAR˅
Resident Performance
R* p R* p R* p
ABSITE (%tile)˄
0.61 <0.0001
0.06 0.6952 0.09 0.5891
PGY1 REG˄ 0.12 <0.4087 0.40 0.0058 0.17 0.2597
Overall Grade˄
0.16 <0.2783 0.34 0.0219 0.16 0.2783
Independent Faculty Rating˅
0.22 <0.1409 0.02 0.9020 -0.12 0.4245
Recruitment Ranking vs Performance
* Spearman correlation coefficient
˄ higher is better˅ lower is better
USMLE˄ URS˄ FAR˅Poor Resident Performance Criteria
Mean SD Mean SD Mean SD
*REG ˄< C (n=1) 239.00 - 111.00 - 27.00 -
≥ C (n=46) 238.42 15.27 110.44 6.00 18.48 11.83
p value 0.8008 0.6041 <0.0001
IFRR ˅ ≥ 4.0 (n=6) 232.50 17.44 111.33 5.20 13.14 9.12
< 4.0 (n=41) 239.33 14.76 110.33 6.08 19.52 12.00
p value 0.3073 0.7023 0.2199
*ABSITE ˄ % < 35 (n=12)
228.92 17.70 110.25 5.93 19.25 13.59
≥ 35 (n=35) 243.48 13.05 110.32 6.37 17.95 11.95
p value 0.0057 0.9752 0.7625
Predicting Poor Resident Performances
Predictors of Poor Resident Performances
* Occurring at any time during the residency
USMLE˄ URS˄ FAR˅Poor Resident Performance Criteria
Mean SD Mean SD Mean SD
*REG ˄< C (n=1) 239.00 - 111.00 - 27.00 -
≥ C (n=46) 238.42 15.27 110.44 6.00 18.48 11.83
p value 0.8008 0.6041 <0.0001
IFRR ˅ ≥ 4.0 (n=6) 232.50 17.44 111.33 5.20 13.14 9.12
< 4.0 (n=41) 239.33 14.76 110.33 6.08 19.52 12.00
p value 0.3073 0.7023 0.2199
*ABSITE ˄ % < 35 (n=12)
228.92 17.70 110.25 5.93 19.25 13.59
≥ 35 (n=35) 243.48 13.05 110.32 6.37 17.95 11.95
p value 0.0057 0.9752 0.7625
Predicting Poor Resident Performances
Predictors of Poor Resident Performances
* Occurring at any time during the residency
Performance criteria
ABSITE(% tile)
PGY1 REG Overall REG IFRR
Rank range( % tile)˅
Mean SD Mean SD Mean SD Mean
SD
<15 % (n=14) 61.69 28.82 4.57 0.55 4.50 0.44 3.10 0.78
15 -30% (n=17)
65.57 21.77 4.29 0.79 4.26 0.64 3.16 0.80
>30% (n=22) 66.43 23.73 4.00 0.76 4.07 0.56 3.22 0.68
P value (F-test) 0.8831 0.1106 0.1338 0.9158
Rank range vs. performance(URS)
Performance criteria
ABSITE˄ PGY1 REG˄
Overall REG˄
IFRR˅
Rank Range(% tile)˅
Mean SD Mean SD Mean SD Mean SD
<15% (n=19) 60.84 26.40 4.63 0.62 4.42 0.50 3.13 0.83
15-30% (n=9) 66.17 22.26 3.81 0.66 3.92 0.61 3.52 0.65
>30% (n=22) 70.00 28.81 4.36 0.71 4.54 0.38 2.70 0.43
P value (F-test) 0.6475 0.0025 0.0051 0.0156
Rank range vs. performance(FAR)
Performance criteria
ABSITE˄ PGY1 REG˄
Overall REG˄
IFRR˅
Rank Range(% tile)˅
Mean SD Mean SD Mean SD Mean SD
<15% (n=19) 60.84 26.40 4.63 0.62 4.42 0.50 3.13 0.83
15-30% (n=9) 66.17 22.26 3.81 0.66 3.92 0.61 3.52 0.65
>30% (n=22) 70.00 28.81 4.36 0.71 4.54 0.38 2.70 0.43
P value (F-test) 0.6475 0.0025 0.0051 0.0156
Rank range vs. performance(FAR)
Single center study URS confounded by USMLE score REG confounded by ABSITE No formal faculty orientation for IFRR survey
Study Limitations
USMLE scores were predictive of subsequent ABSITE performance only.
Unadjusted Ranking Scores (URS) were predictive of subsequent performance based on resident evaluation grades (REGs), while Final Adjusted Rankings (FAR) were not.
Summary
Our resident selection process has generally been successful in providing us with residents who perform well.
Our unadjusted ranking score appears to be a better predictor of subsequent resident performance than our final adjusted ranking… ….therefore caution should be exercised when considering adjustments to the preliminary rank list, as they may not engender selection of better performing residents.
Effectively defining a reliable rank list “cutoff”, beyond which performance will predictably decrease, may not be possible in our system.
Conclusions
◦ Drop outs: 1 (after PGY1 despite excellent performance)
◦ ABS exams first try pass rate (n=13): 100%
◦ Ever with REG <C 100% (i.e. probation): 1◦ IFR > 4.0 (i.e. below average): 6◦ Ever with ABSITE scores < 35%tile ever: 12
ResultsOverall resident performance
Range range vs. performance
USMLE URS FAR
Poor Resident Performance Criteria
Mean SD Mean SD Mean SD
REG < C (n=1)
239.00 - 111.00 - 27.96 -
≥ C (n=46)
238.42 15.27 110.44 6.00 18.48 11.83
p value 0.8008 0.6041 <0.0001
IFR ≥ 4.0 232.50 17.44 111.33 5.20 13.14 9.12
< 4.0 239.33 14.76 110.33 6.08 19.52 12.00
p value 0.3073 0.7023 0.2199
ABSITE % < 35
228.92 17.70 110.25 5.93 19.25 13.59
≥ 35
243.48 13.05 110.32 6.37 17.95 11.95
p value 0.0057
0.9752 0.7625
Predicting Poor Resident Performances
Predictors of Poor Resident Performance
Absolute ranking correlation with resident performance:◦ Absite◦ Semiannual evaluation grade◦ Faculty survey rating
Results Absolute Ranking
Absolute ranking (AR): Ranking among entire candidate group (n= 60-80).
Relative ranking (RR): Ranking among cohort of successful PGY1 applicants (n=4 or 5).
Recruitment Ranking
Within resident cohorts FAR did not correlate significantly with subsequent: ◦ ABSITE scores (r=0.22; p=0.1760) ◦ Semi-annual evaluation scores (r=0.20; p=0.1987)◦ Faculty survey cohort rankings (r=0.23;0.1175)
Conversely, USMLE scores exhibited a significantly positive correlation with subsequent:◦ ABSITE scores (r=0.46; p=0.0022), ◦ Semi-annual evaluation scores (r=0.41; p=0.0163)◦ Faculty cohort rankings (r=0.35; p=0.163)
Results (ABSTRACT)Relative Ranking
Resident recruitment involves a formal evaluation of candidates where a variety of objective and subjective criteria are used to rank candidates from best to worst.
Preliminary rank lists are often subsequently “adjusted” based on additional insights about the candidates.
Introduction
Screening of Applications
Assessment and Preliminary Ranking of Candidates
URS
Final Ranking of Candidates
FAR
Resident Selection ProcessNUFSM