Haptics in Medical Simulation
Allison M. Okamura
Associate ProfessorDepartment of Mechanical Engineering
Design and Biomechanics GroupsStanford University
Collaborative Haptics and Robotics in Medicine (CHARM) Labhttp://charm.stanford.edu
Haptics Tissue modeling forhaptic simulation
Augmented reality Current Projects
Allison Okamura ◆ Stanford University ◆ 2012
Haptics Tissue modeling forhaptic simulation
Augmented reality Current Projects
Allison Okamura ◆ Stanford University ◆ 2012
hearing
sight
smell
taste
touch
which sense is most valuable
to you?
which would you relinquish
last?
Allison Okamura ◆ Stanford University ◆ 2012
what would life be like without touch?
Ian Waterman
Allison Okamura ◆ Stanford University ◆ 2012
why do we have brains?
Daniel Wolpert
Allison Okamura ◆ Stanford University ◆ 2012
sightcentralized
broadpassive
cognitive
touchdistributednarrowactivephysical
Allison Okamura ◆ Stanford University ◆ 2012
hap·tic ('hap-tik)adj. Of or relating to the sense of touch.
[Greek haptikos, from haptesthai, to grasp, touch. (1890)]
TemperatureTexture
SlipVibration
Force
Location/configurationMotionForce
ComplianceCutaneous
Kinesthesia
The haptic senses work together with the motor control system to:- Coordinate movement- Enable perception
Johansson and Westling
Allison Okamura ◆ Stanford University ◆ 2012
haptics making inroadsEntertainment
Nintendo Stanford/JHU Immersion
Education
Human-Computer Interfaces
BMWSamsung Boeing
Allison Okamura ◆ Stanford University ◆ 2012
Haptics
Augmented reality Current Projects
Allison Okamura ◆ Stanford University ◆ 2012
Tissue modeling forhaptic simulation
Haptics for medical simulationMain Challenge:Users should be allowed to interact with the simulator in any manner, and the simulator should respond appropriately.
(Unanswered) Question:How realistic does the simulation need to be to have a significant impact?
This talk addresses the mechanical and haptic interaction components medical simulation.
Allison Okamura ◆ Stanford University ◆ 2012
is the simulator is any good?• Face validity: does the system present an environment
resembling that which is encountered during a medical procedure?
• Content validity: is a skill measured by the system measured the specific skill desired, and not a different one?
• Construct validity: can the system capture the differences between experts and novices?
• Concurrent validity: to what extent does testing performance with the simulator yield the same results as other measures?
• Predictive validity: does performance/training with the simulator transfer to improvements in clinical practice?
Allison Okamura ◆ Stanford University ◆ 2012
Reality-Based Modeling
UBC videos courtesy Simon DiMaio
Allison Okamura ◆ Stanford University ◆ 2012
Cutting with Scissors
Blank Skin
Liver Tendon Forc
e
Scissor angle
Allison Okamura ◆ Stanford University ◆ 2012
Needle Insertion
Time
Forc
e Ve
loci
ty
Allison Okamura ◆ Stanford University ◆ 2012
Needle InsertionAllison Okamura ◆ Stanford University ◆ 2012
Simulation
Accuracy
Needle InsertionAllison Okamura ◆ Stanford University ◆ 2012
Good models also lead to new technology: Needle Steering
Alterovitz, et al. Webster, Reed, et al.
Allison Okamura ◆ Stanford University ◆ 2012
Haptics Tissue modeling forhaptic simulation
Current Projects
Allison Okamura ◆ Stanford University ◆ 2012
Augmented reality
Surgeon
PatientImage source: www.physicianphotos.com
Open SurgeryAllison Okamura ◆ Stanford University ◆ 2012
Surgeon
Instrument/Camera
PatientImage source: www.womenssurgerygroup.com
Minimally Invasive SurgeryAllison Okamura ◆ Stanford University ◆ 2012
© 2008 Intuitive Surgical, Inc.
Surgeon
Master Console
Teleoperated Robot-Assisted Minimally Invasive Surgery
Information-EnhancedRMIS (with Haptics)
Patient-Side Robot
Patient
Instrument/Camera
Allison Okamura ◆ Stanford University ◆ 2012
da Vinci Surgical System(Intuitive Surgical)
surgeon’sconsole
patient-siderobot
Allison Okamura ◆ Stanford University ◆ 2012
Experimental Platform
research version of the da Vinci surgical system
Allison Okamura ◆ Stanford University ◆ 2012
Force feedback:haptics and graphics
Allison Okamura ◆ Stanford University ◆ 2012
Effects of force feedback1. Haptic + Graphical2. Haptic Only3. Graphical Only
4. No Force Feedback
erro
r (d
eg)
feedback
Error (deg)
haptic feedbackreduces error
Allison Okamura ◆ Stanford University ◆ 2012
Differentiating material propertiesAllison Okamura ◆ Stanford University ◆ 2012
porcine kidney
arti!cial lumps
!ducial markers
!ducial markers
!ducialmarkers
!ducialmarkers
!ducial markers
!ducial markers
!ducialmarkers
!ducialmarkers
porcine kidney
X-ray imageCamera image
Lump detection taskAllison Okamura ◆ Stanford University ◆ 2012
Lump detection Results1 2
3 4
Allison Okamura ◆ Stanford University ◆ 2012
What about manipulation?
no overlay dot overlay
In collaboration with D. D. Yuh of JHMI Cardiac Surgery
Graphical force feedback results in lower peak forces, lower variability of forces, and fewer broken
sutures for novice robot-assisted surgeons
Allison Okamura ◆ Stanford University ◆ 2012
Haptics Tissue modeling forhaptic simulation
Augmented reality
Allison Okamura ◆ Stanford University ◆ 2012
Current Projects
In medical simulation today, haptic feedback is typically restricted to that provided by
• artificial materials (e.g., mannequins)
or
• resolved forces (e.g., force feedback haptic interfaces in virtual reality simulators)
Laerdal’s SimMan
Simon DiMaio, UBC
Allison Okamura ◆ Stanford University ◆ 2012
Kinetic System for Medical Simulation (KineSys MedSim)
In collaboration with T. Judkins (IAI), J. E. Colgate (Tangible Haptics), D. Gaba (Stanford)
Allison Okamura ◆ Stanford University ◆ 2012
Kinesthetic haptic display
Images courtesy T. Judkins (IAI)
Allison Okamura ◆ Stanford University ◆ 2012
Electrostatic tactile display
Disney Research
Tangible Haptics
Allison Okamura ◆ Stanford University ◆ 2012
Haptic Jamming: A Deformable Geometry, Variable Stiffness Tactile Display
using Pneumatics and Particle Jamming
Jim Gwilliam and Andrew Stanley
• Simultaneously controllable surface geometry and mechanical properties
• Uses pneumatics and “particle jamming”: a cell made from a flexible membrane filled with granular material solidifies when air is vacuumed out
Allison Okamura ◆ Stanford University ◆ 2012
Haptic Jamming DisplayAllison Okamura ◆ Stanford University ◆ 2012
Force deflection
curves before vacuuming
the cells
0 5 10 15 200
5
10
15
20
25
30
35
Forc
e (
N)
Deflection (mm)
Vacuumed After Inflation
No Inflation Max Inflation
0 5 10 15 200
5
10
15
20
25
30
35Inflated, not Vacuumed
Deflection (mm)
1.0" Cell
2.0" Cell
1.5" Cell
3.0" Cell
Force-deflection
curves after vacuuming
the cells
Allison Okamura ◆ Stanford University ◆ 2012
Haptics Tissue modeling forhaptic simulation
Augmented reality Current projects
Allison Okamura ◆ Stanford University ◆ 2012
JHU APL
Allison Okamura ◆ Stanford University ◆ 2012
AcknowledgmentsPast PhD/Postdoc Lab MembersJake Abbott (Univ. Utah)Nasir Bhanpuri* (USC)Steven Charles*+ (BYU)Jessica Crouch+ (Old Dominion Univ.)David Grow (New Mexico Tech.)Paul Griffiths+ (Intuitive Surgical)Netta Gurari (IIT)Katherine Kuchenbecker+ (UPenn)Mohsen Mahvash+ (Boston Univ.)Panadda Marayong (Cal St Long Beach)Steven Marra+ (JHU)Sarthak Misra (U. Twente)Kyle Reed*+ (USF)Lawton Verner (Intuitive Surgical)Robert Webster* (Vanderbilt Univ.)Tomonori Yamamoto (EndoMaster) *co-advised, +postdoc
… And manytalentedundergraduateresearchers!
Current Lab MembersTroy Adebar (Stanford)Amy Blank* (JHU)Nick Colonnese (Stanford)Tricia Gibo* (JHU)Jim Gwilliam* (JHU)Ann Majewicz (Stanford)Kirk Nichols (Stanford)Ilana Nisky+ (Stanford)Zhan Fan Quek (Stanford)Michele Rotella (Stanford)Sam Schorr (Stanford)Andrew Stanley (Stanford)Tom Wedlick (JHU)
CollaboratorsAmy Bastian (KKI/JHU Neurosci.)Michael Choti (JHU Surgery)Noah Cowan (JHU Mech. Eng.)David Gaba (Stanford Medicine)Ken Goldberg (UCB IEOR)Gregory Hager (JHU Comp. Sci.)Steven Hsiao (JHU Neuroscience)Michael Hsieh (Stanford Urology)Peter Kazanzides (JHU Comp. Sci.)KT Ramesh (JHU Mech Eng.)Danny Song (JHU Radiology)Russell Taylor (JHU Comp. Sci.)Louis Whitcomb (JHU Mech. Eng.)David Yuh (JHU Surgery)Intuitive Surgical, Inc.
SupportNational Science FoundationNational Institutes of Health (NIBIB, NCI, NINDS) Whitaker FoundationJHU Brain Science InstituteDARPA
Allison Okamura ◆ Stanford University ◆ 2012