Overview of CIS systems - cs.jhu.eduhager/ftp_area/MICCAITutorial05/MICCAITutorial05-RHT.pdf · NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology

1

Copyright © CISST ERC, 2005 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology

Overview of CIS systems:Integrating Information to Action in 21’st Century Operating Rooms

Russell H. TaylorDept. of Computer Science/Mechanical Engineering/Radiology/SurgeryDirector, Center for Computer-Integrated Surgical Systems and TechnologyThe Johns Hopkins University3400 N. Charles Street; Baltimore, Md. [email protected]

NSF Engineering Research Center for Computer-Integrated Surgical Systems and Technology


Prediction

The impact of computer-integrated surgical systems and technology on medical care in the next 20 years will be as great as the impact of computer-integrated manufacturing systems and technology on industrial production over the past 20 years.

Engineering Research Center for Computer Integrated Surgical Systems and Technology


Imagers and other sensors

Robotic devices

Human-machine

interfacesInterface technology

Computers and networks

Image & sensor processing, anatomical modeling, surgical

planning and control, …

Patient data bases

Anatomic atlases and

surgical task models

patient surgeon

Images & other sensor data

Command & control

Decision support &

commands

2


Computer-assisted planning

Patient-specific Model

Update Model

Computer-Assisted

Execution

Update Plan

Computer-Assisted

Assessment

Preoperative Intraoperative

Atlas

Postoperative

Patient




Update Model

Computer-Assisted

Execution

Update Plan

Computer-Assisted

Assessment


Atlas

PostoperativePatient

Surgical “CAD”

Surgical “TQM”

Surgical “CAM”




Update Model

Computer-Assisted

Execution

Update Plan

Computer-Assisted

Assessment


Atlas

Postoperative

Patient

Surgical

Assistants

3


Computer-assisted

simulation & planning


Update Model

Simulate and replan in real

time

Computer-Assisted

Assessment


Atlas

Postoperative

Patient

The grand unified system*

* From Turf Valley Workshop (March 2003)


Drivers for adoption• Error-free surgery

– Designs & systems to make things safer– Integration of imaging, computer assistance, etc.

• Enablers for otherwise infeasible interventions– Superhuman capabilities (access, precision, …)– Delivery systems for novel image-guided therapies

• Enablers for research & development– Consistency, capability & data gathering

• Capture of information about cases• Use of robots as training tools

– Mentoring– Simulation integrated with robotics

* From Turf Valley Workshop (March 2003)


“Six Sigma Surgery”• Goal: Eliminate errors in interventional medicine.

– 44,000-98,000 hospital patients die each year as a result of errors (Institute of Medicine, Dec. 1999)

– Over 50% of “should never happen” events are surgery related (Minnesota Report on Adverse Health Events, Jan. 2005).

– What actually happens in the OR is poorly documented and variations in surgical technique are not well controlled in outcome studies

• Significant attention from Congress, DoD, NIH, …• Solution inherently requires systems approach

integrating process information management, human factors, technology.

4


Suggested Approaches to OR SafetyStephen Schimpff, MD*

• HUMAN FACTORS– LEADERSHIP– MANAGEMENT– TEAMWORK– COMMUNICATION– TRAINING– INFORMATION

TRANSFER– MANDATES

• TECHNOLOGY – INFORMATION SYSTEMS– IDENTIFICATION DEVICES– VIDEO– SIMULATORS– ROBOTICS

* Former CEO of U. Md. Hospital, presentation based on study on OR patient safety commissioned by TATRC


Suggested Approaches to OR SafetyStephen Schimpff, MD*

• HUMAN FACTORS– LEADERSHIP– MANAGEMENT– TEAMWORK– COMMUNICATION– TRAINING– INFORMATION

TRANSFER– MANDATES

• TECHNOLOGY – INFORMATION SYSTEMS– IDENTIFICATION DEVICES– VIDEO– SIMULATORS– ROBOTICS

* Former CEO of U. Md. Hospital, presentation based on study on OR patient safety commissioned by TATRC

Human factors and technology are synergistic elements of any comprehensive approach!


Research and technology barriers

Modeling & analysis• Segmentation• Registration• Atlases• Optimization• Visualization• Task characterization• etc.

Interface Technology• Sensing• Robotics• Human-machine

interfacesSystems• Safety & verifiability• Usability & maintainability• Performance and validation

5


• Multi-institution, multi-disciplinary center– Johns Hopkins University + Medical Institutions– MIT + Brigham & Women’s Hospital– CMU + UPMC– Others: Morgan State, Georgetown, Harvard, Penn, Columbia

• Funding– Year 7: Core NSF Grant = $3.9M; Total = ~$8.3M– Years 1-10: Core NSF Grant = $30M; Total = ~$55M

• University researchers, clinicians, industry• Research, Systems, Education, Outreach

NSF Engineering Research Center for Computer-Integrated Surgical

Systems and Technology


1998


2005

6


Modeling and

Analysis

Interface Technology

Systems Science

Surgical CAD/CAM

Surgical Assistants

Dual ways of thinking about CIS research


Modeling and

Analysis


Systems Science

Surgical CAD/CAM

Surgical Assistants

By Technology Family


• Building anatomic models• Data fusion• Reasoning about uncertainty• Managing complexity• Computational algorithms• Limited data

Modeling & Analysis

Core Challenge: Develop computationally effective methods for patient-specific modeling and analysis

Before

After

7


Modeling and Analysis Highlights

• Explosion of 3D imagingmodalities

– MRI, CT, Ultrasound, etc.• Patient-specific models

based on images• Statistical atlases and

deformable models are emerging as crucial enabling technologies within CIS

• Beginning to see application of real-time computer visionmethods

Davatzikos

Yao


Atlas-driven deformable registration

Image of deformable model (red)

Overlay image of deformable model (red) on target image (cyan)

Value of mutual information between model and target images, over registration progress

IterationO. Sadowsky, J. Yao, R. Taylor


Fast DRRs from deforming model

8


Atlas-augmented X-ray ReconstructionTaylor, Prince, Yao, Sadowsky, Ramamurthi

X-rays

Atlas

Deformable 2D-3D Instance of AtlasCT slice Ideal

ConebeamRecon

LimitedConebeamRecon

Atlas-mergeConebeamRecon Fill in missing

projections


Hybrid Reconstruction from Patient Images and Atlas

Sadowsky, Ramamurhti, Chintalapani, Ellingson, Prince,Taylor

Full Sweep (208º)Reconstruction

Partial Sweep (120º)Reconstruction

Hybrid (120º+88º from atlas)

Reconstruction


Real-time Video TechniquesHager/Thakor/Yuh/Lau (JHU)

Problem: Construct dynamically tracked models of deformable surfaces

Solution: Optimize a parametric surface from stereo imagery

Results: Real-time tracking of a beating heart with:1. Real-time performance2. Extremely high accuracy (< 1/10 pixel)3. Generalization to many imaging devices

and applications

Stereo tracking of in-vivo beatingheart using Intuitive Stereo Endoscope

9


Scale-Invariant Registration of Monocular Images to CT-Scans

for Sinus Surgery

Darius Burschka, Ming Li, Russell Taylor, Gregory D. Hager

Proc. MICCAI 2004St. Malo, France

• Accuracy Experiments– Plastic skull phantom– Surface reconstruction ~ 0.5 mm– Registration ~0.5 mm


Video tracking of surgical tools

Akinbiyi, Okamura & Yuh; Burschka & Hager


Modeling and

Analysis


Systems Science

Surgical CAD/CAM

Surgical Assistants


10



• Imaging and sensing• Robotics• Human-machine interfaces

Core Challenge: Fundamentally extend the sensory, motor, and human-adaptation abilities of computer-based systems in an unusually demanding and constrained environment


Interface Technology Highlights

• Emergence of robot manipulator designs specifically designed for surgical applications

– E.g., RCM, MRI compatible systems, “snake” robots, …

• Cooperative control for human skill augmentation

• Specialized systems with tissue or image sensing

S. Salcudean et al. (UBC)

R. Taylor, et al. (JHU)R. Howe, et al. (Harvard)


High Dexterity Robot for Confined Spaces

Patient

Master

Controller Patient

Master Slave

Throat

Ø4 mm Distal Dexterity Unit forSurgical tool manipulation

∅4 mm

N. Simaan, R. Taylor, P. Flint

11


Scalable robot for dexterous surgery in small spacesNabil Simaan, Russell Taylor, Paul Flint, MD

1 gripper2 moving platform3 parallel stage flexible links4 Gripper actuation link5 end disk6 spacer disk 7 central backbone tube8 base disk9 DDU holder

9

12346 5

78

parallel manipulation unit

Snake-like unit


Functional Prototypes: Large and Small Scale

Test-bed for Suturing Application & Low Level Controllers and Debugging

Large Scale Small Scale

N. Simaan, R. Taylor, P. Flint, A. Kapoor, P. Kazanzides


Sensorized Instruments

Receiver

Emitter

Small Intestine Oxygenation

0.8

0.85

0.9

0.95

1

1.05

1.1

0 20 40 60 80 100 120 140 160 180

Time (sec)

Pro

p. to

ox

cont

ent

Oxymetry Sensors Force Sensor

• Tool-to-tissue interaction force

• Tissue oxygenation, etc.

• Integrate with information infrastructure, process control, decision supports

12


Sensorized Instruments


Modeling and

Analysis


Systems Science

Surgical CAD/CAM

Surgical Assistants



Systems Science

• Modularity & Standards• Robustness & Safety• Customization• Validation • Information management

Core Challenge: Develop architectures, building blocks, and analysis techniques that facilitate rapid development and validation of versatile CIS systems & processes with predictable performance

Engineering Research Center for Computer Integrated Surgical Systems and Technology

13


Systems Highlights• Commercial systems

– CAD/CAM: Robodoc, Arthrobot, Neuromate, Accuray, Image-Guide, navigation systems…

– Assistants: daVinci, Zeus, AESOP, …

• Research systems being clinically evaluated

– E.g., JHU in-MRI prostate robot, US guided robots, …

• Recognized need for testbed architectures &shared components

• New interest in systems as enablers for biomedical research

Intuitive Surgical daVinci


Crucial need: common architecture & infrastructure

Ron Kikinis, Kiyo Chinzei, Clare Tempany - BWH

R. Taylor, G. Fichtinger, C. Burdette, M. Choti - JHU

Application Control & Visualization

CommunicationSoftware

Robot Control

Visualization Workstation

Intraoperative MRI System

Planning System

Robot control PC

High-B

andwidth N

etwork

MR Compatible Robot

Security monitoring host


Modeling and

Analysis


Systems Science

Surgical CAD/CAM

Surgical Assistants

By Systems Family

14


Manual Surgery Robotic Surgery

Robotic Joint Replacement Surgery


Robotic THR & TKR Systems in Research Phases (Partial List)

• Parallel link approaches– Aachen (GRIGOS)– KAIST – Technion

• “Conventional” arms– Northwestern– U. Washington– Rizzoli Institute

• Cooperative Control– Imperial College

(ACROBOT)– Grenoble (PaDyc)

Movie: Brian Davies

Movie: KAIST


X-ray based registration

R. Taylor

15


Iterative Cutting to Improve Accuracy

R. Taylor, R. Kumar, R. Goldberg & J. Yao


Statistical Atlas

“Virtual CT”

Deformable 2D-3D

Planning Updated Plans

Registered Model

Visualization & Guidance

X-raysX-rays

Rigid 2D-3D

R. Taylor & J. Yao


Registration-enhanced reconstruction

Sadowsky, Ramamurthi, Prince, Taylor; CAOS 2005

16


Ultrasound-based registration

CT

US

Sample contours

Surface model

Register

A. Jain, R. Taylor


US Image Probability map Segmented contour

• Construct 3D surface probability map from multiple tracked 2D US images

• Compute segmented contours from probability map

• Presented SPIE 2004


Image-Guided Needle-Based Interventions

Potentially significant impact on medical practice

• Minimally invasive (compared to open surgery)

• Faster recovery• Less morbidity• Fewer complications• Lower cost• Repeatable in many indications

• Sharply increasing number of procedures

Challenging but also doable•• Constrained process Constrained process ––

formally describableformally describable• Major challenges (in addition

to open/lap surgery):• no visibility• no access• no room to maneuver• no room to recover

17


Lavallee, Troccaz, et al. 1989

Robotic Needle Guidance

Kwoh, et al. 1988


Robotic Needle Guidance

Courtesy: Integrated Surgical Systems


Clinical example• Kidney biopsy• Robot registered to CT

from single image using markers on end-effector

Photos: D. Stoianovici, L. Kavoussi, A. Patriciu, S. Solomon (JHU Bayview)

Other contributors: R. Susil, G. Fichtinger, K. Masamune, R. Taylor (JHU WSE)

18


US-Guided Prostate Therapy (Fichtinger, et al.)

Jain, Mustufa, Fichtinger

Mechatronics

Registration

Okamura, Schneider

Reconstruction Segmentation

Jain, Zhou, ChirikjianPatriciu, Stoianovici

Shen, Davatzikos


CT Image Overlay

CT image shown over the subject

Optically stable without trackingFichtinger, Masamune, Taylor, …


Human-Grade System

Click for movieFichtinger, Masamune, Zinreich, Deguet, Iorchidata, Sauer, …

19


Example Application: Spinal Nerve Block

• 22 G needles inserted at nerve root in the lower back• No contrast, no therapeutic substance

• Needle tip landed close by nerve root• Needle shaft is in plane ☺

G. Fichtinger, J. Zinreich, F. Sauer, A. Deguet, et al.


Sonic Flashlight by George Stetten

• G. Stetten, V. Chib, "Magnified Real-Time Tomographic Reflection," MICCAI-2001, LNCS 2208, pp. 683-690, 2001

• G. Stetten, A. Cois, W. Chang, D. Shelton, R. Tamburo, J. Castellucci, O. von Ramm, “C-mode Virtual Image Display for a Matrix Array Ultrasound Sonic Flashlight,” MICCAI-2003, LNCS 2879, pp. 336-343.


Robotic Needle Placement in RodentsP. Kazanzides, G. Fichtinger, C. Ling

Research contract from Sloan Kettering

Credit: Jack Li, I. Iordachita Courtesy of MSKCC

Micro-PET scanner

20


Modeling and

Analysis


Systems Science

Surgical CAD/CAM

Surgical Assistants

By Systems Family


Navigation Systems

Photo: VTI Inc.


Cutting, Bookstein, Taylor, et al.

21


Robotic “Third Hand” Assistants

• Limb positioners• Retractors• Endoscope holders

– Aesop– IBM/JHU LARS– etc.

• Can incorporate sophisticated HMI, voice, vision, etc.

Credit: Yulun Wang


Telerobotic Surgical Augmentation

• Commercial systems include Zeus, daVinci

• Teleoperated MIS surgery– Late 1990s robotics

implementing late 1980s techniques

• Great potential to go beyond this– Information fusion– “Virtual fixtures”– Truly remote access

Photo: Intuitive Surgical


Telerobotic Surgical AugmentationSRI telesurgery system, circa 1992

IBM/JHU LARS, circa 1993-4

22


IBM/JHU LARS Interfaces


Cooperative force guiding of surgical robots

Photo: JHU

Photo: BGU


Cooperative force guiding of surgical robots

Davies, et al.

23


Robotic Assistants for Microsurgery


Handle force

Kv

Steady Hand Guiding for Microsurgery

R. Taylor & R. KumarFree hand motion Steady hand motion


Stable platform for interventions

R. Kumar, 2001

24


Next Generation Eye Robot

Photo: Iulian Iordachita


Biomanipulation with a steady hand robotRajesh Kumar(1), Ankur Kapoor(2), Russ Taylor(2)

• Cooperatively controlled robot for single-cell scale biomanipulation tasks

• Multiple control modes• Simple compliant guiding• “Augmented” compliant guiding• Shared/supervised autonomy

• Future work includes:• Next generation system• Visual “virtual fixtures”

(1) Foster-Miller; (2) CISST ERC, JHU


Evolution to human-machine partnership

• Situation assessment• Task strategy & decisions• Sensory-motor coordination

Augmentation System

• Sensor processing• Model interpretation• Display

atlases

• Manipulation enhancement

• Online references & decision support

• Cooperative control and “macros”

atlases

librariesR. Taylor

25


Evolution to human-machine partnership

• Situation assessment• Task strategy & decisions• Sensory-motor coordination

Augmentation System

• Sensor processing• Model interpretation• Display

atlases

• Manipulation enhancement

• Online references & decision support

• Cooperative control and “macros”

atlases

librariesR. Taylor


Steady Hand RobotHigh Level Constrained Control

Handle Force Kv

Joint Velocities

Reference Direction

Current Frame(s) Info.

Geometric Constraints on

Frame(s)

Optimization Framework


Combine constraints

• Single frame • Multiple framesTranslational part

Rotational part

Select one or more

Move along a

line

Rotate around a

line

Maintain a direction

Prevent plane penetrating

Stay on a point Customized virtual

fixtures

26


Example: Endoscopic Sinus Surgery

D. W. Kennedy, W. E. Bolger , and S. J. Zinreich

Diseases of the Sinuses: Diagnosis and Management. 2001.


Steady-hand sinus surgery with virtual fixtures derived from CT models

Ming Li, Russell Taylor

tip point pathbent tip portion

tool shaft portion


Virtual Fixture Online Implementation

Rob

ot in

terfa

ce

State

Path

Tool tip guidance virtual fixture

qΔdesPΔ

Constraint generation

( ) 2min

Subject to

tip desW J q P

G q g

⋅ ⋅Δ −Δ

⋅Δ ≥

Registered model

M. Li; R. Taylor; ICRA 2005

27


Cooperative control guiding

3D mouse guiding

View of path & toolM. Li; R. Taylor; ICRA 2005


Virtual Fixtures for SuturingAnkur Kapoor, Ming Li, Russ Taylor

EntryExit

Needle

Force Sensor

Phantom Tissue

__________________________________________________________________________________________________________________________________________________________________________________

Alois Knoll, TU Munich, Computer Science 105/28

Future research direction: selective automation

Principle of Skill Transfer(similarity to OCR)

• Multiple recordings of high-dimensional motionpattern data by going through many operations

• Identification of segments and definition of "motionand sensorimotor primitives" (abstraction step)

• Application to a given situation based on scene and description (sensor output) and task specificationthrough chaining of primitives

28


Computer-assisted

simulation & planning


Update Model

Simulate and replan in real

time

Computer-Assisted

Assessment


Atlas

Postoperative

Patient

The Computer-Integrated OR


• Improved quality– Reduce surgical errors– Lower morbidity– Greater consistency

• Enable new interventions– Transcend human limits– Minimize invasiveness– Process feedback

• Cost Effectiveness– Supply / workflow control– “Do it once”– Time

Drivers for Adoption


• Comprehensive real time computer models – Patient– Surgical task– Operating room

• Computer-driven technology– Robotics, visualization aids– Imaging equipment– Anesthesia, therapeutic devices,

logistics systems• Comprehensive information

capture and analysis– Error & “near miss” analysis– Outcomes & indications

• Integration with broader hospital information system

Key elements

29







Key elements







Key elements







The whole is greater than the sum of parts

30


Surgical Total Information Management

• Real time sensing & modeling of surgical situation– Patient-specific models– Surgical tools & tool-tissue interations– Surgeon movements– Etc.

• Application drivers– Assistance (robots & information support)– Outcomes assessment

• Technology drivers– Imaging, computer vision, image analysis– Sensors, sensor networks– Biomechanics, statistical modeling– Human-machine interfaces, haptics– Performance/task modeling


Multidisciplinary Integration is Crucial

Modeling & analysis• Segmentation• Registration• Atlases• Optimization• Visualization• Task characterization• etc.

Interface Technology• Sensing• Robotics• Human-machine

interfacesSystems• Safety & verifiability• Usability & maintainability• Performance and validation


How can we get there?

Strong and committed teams

– Surgeons– Engineers– Industry

Focus on systems that address important needs

Rapid iteration with measurable goals

Have fun!

31


The real bottom line: patient care

• Increase consistency and quality of surgical treatments

• Provide new capabilities that transcend human limitations in surgery

• Promote better outcomesand more cost-effectiveprocesses in surgical practice

Documents

Overview of CIS systems - cs.jhu.eduhager/ftp_area/MICCAITutorial05/MICCAITutorial05-RHT.pdf · NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology