Evaluating Surgical SkillsAnd Operating Room

Performance:Education/Remediation?

Certification/Credentialling?John J. Ferrara MD

Kanav Kahol PhDPhoenix Integrated Surgical Residency

Evaluating Surgical SkillsChallenges

How to maintain cardinal surgical “art and science” traditions when the sands that support educational paradigms are shifting? “Publish or perish” to “Produce (RVU’s) or perish” “Duty” hours “Public” opinion

Generational chasm “Linear” educational construct

Generation X: The BridgeBoomers (46-64 years) Defined by work ethic Independent Religious Financial success Career-driven Wanna be lead dogs Kumbaya Consumer-driven/TV

Millennials (18-29 years) Defined by technology Social agenda Secular Parenthood (non-traditional) Time-driven Lead only if asked Blunt Under-consumers/TV?

The good news: they respect (boomerang back to) their elders

Technical Skills EvaluationLinear Construct

Technology

Simulation Environment

“Real” Environment

Technical Skills EvaluationParallel Construct

LeveragedLeveraged

ScalableScalable AdaptableAdaptable

IntegrativeIntegrative

Goals Measure Technical Skills in a Simulated

Environment Create a system to measure skill set and to provide

immediate feedback to the user “Battleship down”

Measure Technical Skills in the Operating Room Develop and validate a system to analyze videos of

operations submitted to a panel for assessment

Objective Proficiency Measures Employ neurological and

kinesiological features to analyze task (surgical) proficiency

Construct task decomposition based feedback system Breaks complex motion

into simpler units that are: Easy to analyze Easy to comprehend Easy to modify by the user

Expert

Intermediate

Novice

Instrumentmovements

Rosen 2002

Hand Motion

Motorical ChunkingMeasure of Expertise

Expert Novice

Dynamic Virtual Reality Systems for Cognitive Training

Train residents for attention, working memory, intermodal transfer

Modify technique simulators to include a cognitive layer

Treat surgery as a combination of psychomotor and cognitive skill Original Task (Laparoscopic Training)

Modified to target working memory

Marble Mania

High (0.92) correlation with basic surgical gestures

Fine motor skills based game

Hand motions similar to laparoscopy

Marble Mania

CyberGlove Analysis

Non-Dominant Hand Dominant Hand Marble Mania

Ambidexterity

Technical Proficiency on ProMIS

5.9

3.9

6.5

2.5

6.3

1.0

2.9

Novices

Intermediates

ExpertsMasters

Skills Evaluation

Measuring Skills in the “Real” EnvironmentProposed Solution

Computer vision instrument automatically analyzes videos Develop means/ranges/standard deviations Set “minimal” performance grade Benchmarking?

Picks up events the naked eye misses Detailed movement analysis Cheap, “portable”, time-efficient

Web based access to rate videos for experts Web based training tools to train experts to rate videos

Video CaptureLaparoscopy

• Basic apparatus• Video capture system for laparoscopic system

and hand movements• Hand movements captured by external camera• Sites: ceiling/lighting system/tripod

• De-identified videos• Our system “syncs” these two streams for

presentation and analysis

Dual Capture System

Skills Evaluation

Expert Intermediate Novice

Tremor

Inst

rum

ent P

ath

Ineffi

cien

cy

Between Groupsp<0.05

Between GroupsP<0.05

Expert v NoviceP<0.05

Benchmarking?

Web-Based Training Upload/automatically

analyze videos on www.ratethesurgeons.com

Experts view videos off-site Can provide input/feedback

Novice raters• View expert ratings • Receive instruction to

become proficient raters• Reward system: pair teaching

Correlation of Subjective Measures with Various Objective Measures

1.0

0.4

ValidationR=0.93 p<0.05

Experts

Intermediates

Novices

Where We are Now

Validation of the technical analysis tool

Evaluation on simulators also being done with videos

Future Work

Enhance Database Develop Benchmarks Expand Skill Set Instrument Family Patient Care Applications

New Simulation Tasks

Motion History Images

Virtual World “Acute Care Surgery” Training

Challenges“The Uncanny Valley”

Masahiro Mori (1970)

Avatar

ChallengesThe Simulation Perfect Storm

Conventional computing is dead, and with it, the first generation (six figure) simulators

Computing life measured in months• Core processors• Naturalistic computing• Gaming consoles

How to maintain a database when evaluation instruments are constantly changing?

Conclusions

We (all) need help We have no magic bullet We need genomic variation

“The Two Word Definition of Dogma is Brain Dead”Zollinger (sometime during my residency)

ChallengesEngineers

Clinicians

Video Capture• Basic apparatus• Video capture system for

laparoscopic system and hand movements• Hand movements

captured by external camera

• Sites: ceiling/lighting system/tripod

• De-identified videos• Our system “syncs” these

two streams for presentation and analysis

Mobile simulator unit

We are becoming increasingly challenged with teaching new dogs old tricks

AND

We are not very good at teaching old dogs new tricks

Evaluation Poses a More Daunting Challenge

Analysis

• Basic movement tracking algorithms from computer vision, an established field with myriad algorithms to track movements and predict efficacy

• Proprietary state of the art tools analyze movements