Utilizing Toolsand Resources

to Assess Fellows

Art DeCross MD AGAFAssociate Professor

University of Rochester Medical Center

Disclosures

• Nothing.

• Absolutely nothing.

• Two kids in college, and I assure you, there are no finances left to disclose.

What are we going to talk about today?

• My needs in a tool

• Process assessment vs meaningful assessment.

• Specific tools

• Trainee at risk– Being helpful– Being careful

My needs• PD small to med program

• Talking to myself at faculty meetings

• Running the program – the Art show

• Time and energy, administrative efficiency– CEX? NO– Standardized patients? NO

Process assessment- going thru the motions -

• Chores

• Artificial

• Reciprocity assures fatal restraint

• Work arounds and short cuts limit value

(more) Meaningful Assessments

• Genuine opportunities for growth.

• Honest and accurate assessments assured by:– highly objective nature of the tool,– Statistical certainty due to high volume of events– Public display to multiple simultaneous evaluators– Permanent record of measured variable– No risk to the evaluator

Specific tools I use• Nearly Useless:– Self reflection (Fellow self assessment)– Current endoscopy metrics (yep, that’s what he said)– Attending global performance assessment– End of rotation assessments

• Useful:– Nurse and staff assessments– Patient assessments– Lecture, dictation assessments– QI Projects

• Most Useful:– The GTE exam

Nearly Useless Tools• Fellow self-reflection– How I use it: (automated survey to focus self-

directed learning)

– Why I still use it: (makes them read all the learning objectives in the curriculum)

– Shortcomings: the best fellows are hypercritical, and the ones who need self-reflection the most, aren’t capable of it.

– Room for improvement: figure out how to teach self-reflection better

Nearly Useless Tools• Current endoscopy metrics– How I use it: (self reporting, signed off by

attending supervisor)

– Why I use it (efforts to develop milestones in Patient Care)

– Shortcomings: focus on simple procedural metrics, universally achieved at a busy clinical program.

– Room for improvement: TBD

Nearly Useless Tools Attending GPA and End of rotation assessments:

How I use it: (automated quarterly e-value surveys)

Why I use it: (easy, satisfies means to assess up to 2 competencies, allows a mechanism for universal condemnation)

Shortcomings: useless assessments, compromised by halo effect and fear of reciprocal evaluation, completion rates poor without constant reminders and/or chief enforcement.

Room for improvement: scale could be improved to Needs improvement, Meets expectations, or Exceeds.

Useful tools

• Nurse and staff assessments of fellows:

– HOW: automated semi-annual survey. If problems, changed to quarterly.

– WHY: unsuspected evaluators, no reprisals, many nurses individually work with each fellow. These evaluation cover a lot of ground across the more difficult competencies.

Useful tools• Patient assessments:

– How: every fellow’s clinic patient gets survey to send back, entered de-identified, summated semi-annual

– Why: large numbers provide accurate snapshot of questions attendings really can’t answer (e.g. how well does this fellow explain a problem to a patient, is fellow respectful, etc.)

Useful tools• Lecture assessments:

– How: faculty evaluate by survey

– Why: evaluate performance as a teacher, assess effort to evaluate current data, treat colleagues with respect, etc. Allows fellows to demonstrate strengths or weaknesses across competencies in a manner not apparent from routine clinical work.

Useful tools• Dictation assessment:

– How: pull several dictations every few months per fellow.

– Why: easy, gives good insight into communication skills, language skills, thought organization, professionalism, comprehensive understanding of clinical medicine; based on elements included or not, not recall and less subjective than evaluating oral presentation.

Useful tools

• QI projects:

– How: required; standard measure, intervene, re-measure approach. Fellow designs project idea, carries out measurements, reports outcomes.

– Why: great tool for PBL competency, can be combined with clinical research and published. Fellows seem to enjoy these – more practical utility, sense of positive accomplishment?

Most Useful tool

• GTE exam:

– How we use GTE: annual, required, program pays. Our fellows required to reach 15% performance.

– Why use GTE: purely objective, identifies problem performers early enough to remediate.

Why I like the GTE• As of 2012, GTE includes:– 97% of allopathic GI programs, and – 82% of all GI fellows

• Although formative:– High internal consistency (Cronbach’s coefficient

alpha is 0.89)– Meaningful improvement F3>F2>F1 in global

performance and in each subject area, p<0.0001F1 n= 352 (30% of total) mean % correct: 51 +/- 8.7F2 n = 430 (37%) 57 +/- 9 F3 n = 378 (33%) 63 +/- 8.7

Why I like the GTE

– With near global participation, the exam reflects and compares a fellow’s Medical Knowledge performance within their peer group, the same cohort against which the fellow will take their board exam.

– The ABIM GI board pass rate for first time takers (regarded as a quality marker) ranges around 85%.

– More on GTE exam: Gastro 2012;142:201-204

Remediation- The trainee at risk -

• Decision: Is the deficit the responsibility of the training program?

• Zero tolerance conditions do exist.• Levels of infraction vary (involve DIO/Dean GME ?)• Program director must NOT/NEVER function as a

treating physician, mental health professional, interpersonal counselor. Refer these functions to employee health.

• Early assessment and reassessment is key.

Remediation• HOW: – Conversation/interaction should be constructive, not

judgmental, no emotional content– Always should occur with multiple faculty, (chief) – Remediation plans should be written, signed by fellow

and signed copy retained by program.– The written plan must clearly state what objective

was not met, identify target objective to be met, the time course to meet, the interim monitoring process, and the consequences of failure to remediate.

– Fellows progress monitored by committee, with written progress reports filed by program.

Remediation• Remediation failure:– Program must decide if method of failure at issue:• Lack of effort vs uncontrollable circumstances

• Program must decide whether failure to remediate identifies a trainee who should not continue in program – in that event, GME must be involved for academic probation, if they were not already involved.

Case examples:• Case #1: Fellow A is extremely deferential to

attending physicians, and generally a bright fellow, completes work, good team member, attends conferences, punctual. Gets glowing attending GPA assessments. However, on nursing evaluations 5/9 nurses report fellow unnecessarily rough with patients, unresponsive to their requests for analgesia, generally dismissive of the endoscopy nurse, and rate the fellow Unsatisfactory in several categories.

• Talked with fellow about negative nursing evaluations, asked for fellow’s perception of circumstances.

• Impressed upon fellow that while these evaluations are subjective, these reports were highly unusual and clearly an outlier.

• Behavior expected to improve.• Nursing was separately informed that fellow

was counseled on these issues, and instructed to monitor for changes, and not to judge fellow on prior performance.

• Nursing evaluations issued quarterly for this fellow.

• Nursing assessment of behavior improved dramatically, all the way into the superior category, and remained there the remainder of fellowship.

• Case #2: 1st year fellow scores in 3% on GTE exam. No excuses, no distractions, personal stresses, etc. Study plan written out, resources provided, progress monitored carefully by Associate PD with monthly completion of assignments. Fellow signed off on understanding that his next GTE exam % must be >15%, or academic probation is an option for the program.

End