100
Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology Research Issues in Robot Research Issues in Robot - - Assisted Assisted Needle Interventions Needle Interventions Gabor Fichtinger, PhD Director of Engineering, Associate Research Professor of Computer Science, Mechanical Engineering, and Radiology Center for Computer Center for Computer - - Integrated Surgical Systems and Integrated Surgical Systems and Technology, Technology, Johns Hopkins University Johns Hopkins University [email protected] [email protected]

Research Issues in Robot-Assisted Needle Interventions

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Research Issues in RobotResearch Issues in Robot--Assisted Assisted Needle InterventionsNeedle Interventions

Gabor Fichtinger, PhDDirector of Engineering,

Associate Research Professor of Computer Science, Mechanical Engineering, and Radiology

Center for ComputerCenter for Computer--Integrated Surgical Systems and Integrated Surgical Systems and Technology,Technology,

Johns Hopkins UniversityJohns Hopkins University

[email protected]@jhu.edu

Page 2: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

• Multi-institution, multi-disciplinary center– Johns Hopkins University + Medical Institutions– MIT + Brigham & Women’s Hospital (SPL)– Carnegie Mellon + Univ. of Western Pennsylvania– Others: Harvard, Penn, Georgetown, Morgan State

• Director/PI: Russell H. Taylor• Core funding from NSF• Supplemental funding from NIH via NCI, NIBIB• Mission: Couple Information to Surgical Action

Engineering Research Center for Engineering Research Center for ComputerComputer--Integrated Surgical Integrated Surgical

Systems and TechnologySystems and Technology

Page 3: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

ENGINEERS: Russ Taylor, Allison Okamura, Greg Chirikjian, Louis Whitcomb, Dan Stoianovici, Alex Patriciu, Dumitru Mazilu,

Greg Hager, Chad Schneider, Axel Krieger, Tabish Mustufa, Keenan Wyrobek, Sangyoon Lee, Yu Zhou, Randy Goldberg, Pat

Jensen, Peter Berkelman, Punet Gupta, Greg Fischer, Robert Webster, Ken Masamune, Gernot Kronreif,

Ron Kikinis, Steve Pieper, Nicole Aucoin, Clif Burdette, Attila Tanacs, Ergin Atalar, Rob Susil, Emese Balogh, Ameet Jain, Anand Viswanathan, Anton Deguet, Herve Mathieu, Rajesh

Kumar, Emad Boctor, Peter Kazanzides, Emad Boctor, Kevin Cleary, Sheng Xu, Ameet Jain, Simon DiMaio, …

CLINICIANS: Cynthia Menard, Jonathan Coleman, Vincent Lerie, Ted DeWeese, Lee Myers, Scott Borzillary, Howard Francis,

John Niparko, Dan Rothbaum, Gene DeJuan, Louis Kavoussi, Michael Choti, Clare Tempany, Ferenc Jolesz, …

and surely several others to whom I apologize…

Long (and incomplete) list of contributorsLong (and incomplete) list of contributors……

Page 4: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Financial support for GFFinancial support for GFNSF EEC-9731478NIH 1R01EB002963-01NIH 1R41CA106152-01A1NIH 5R44CA088139-04NIH 1R43CA099374-01NIH 1R41CA103468-01NIH 1R41RR019159-01NIH 2R42EB000374-02AdMeTechCMS Burdette Medical SystemsSiemens Corporate Research… and more of my collaborators

Page 5: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

OverviewOverview

• CIS (a.k.a. CAS or IGS)• Needle Based Surgery• Microsurgical Assistants• Point and Click Surgery

• CT-guided systems• US-guided systems• MRI-guided systems

• Lessons Learned

Page 6: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Benefits of CISBenefits of CIS“Better” Technology

• Greater accuracy & precision• Greater consistency

• “equalizer” among physicians• “equalizer” among communities

• Eventually faster, less expensive

“Better” Medicine & Life• Higher curative rate• Less morbidity• Fewer complications• Better outcome

Page 7: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

ImpactImpact• Procedures performed “better”• Availability increased

Qua

lity

of p

roce

dure

s Availability of procedures

Page 8: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

The Prime DirectiveThe Prime Directive

UsefulnessBut how?

• Read the clinician• Simplicity• Work incrementally

Page 9: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Computer Integrated SurgeryComputer Integrated Surgery

Page 10: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Fundamental research barriersFundamental research barriers

Modeling & Analysis• Segmentation• Anatomical Atlases• Registration• Reconstruction

Physical Interfaces • Sensing/Imaging• Robotics• Haptics

Systems Integration• System Design• Performance Analysis• Architecture• Clinical Applications

Page 11: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Sensible technical scope: needlesSensible technical scope: needles

Page 12: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Why needles?Why needles?Potentially significant impact on medical

practice• Minimally invasive (compared to open surgery)

• Faster recovery• Less morbidity• Fewer complications• Lower cost• Repeatable in many indications

• Sharply increasing number of procedures

Challenging but also doable•• Constrained process Constrained process –– formally describableformally describable• Major challenges (in addition to open/lap surgery):

• no visibility• no access• no room to maneuver• no room to recover

Page 13: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Sensible clinical scopeSensible clinical scope

Spine/Bone70% of population affected in lifetime400,000 metastatic

cancer /year

LiverMetastasis from colorectal cancer

130,000 new /year60,000 death /yearHepatitis worldwide

Prostate200,000 cancers/year

1M biopsies /year10M BPH currently

25% of men affected in lifetime

Why these?• Significant health problems• Right mix of challenge and doability• Clinical buy-in• Experience of investigators• Funding opportunities

United States numbers

Eye~100k/y retinal

occlusions, >100k/y age-related

macular degeneration

(AMD)

EarHearing loss of 30-35% of 65-

75 yo

40-50% over 75 yo

MACRO SCALE MICRO SCALE

ENT BrainKidney

Page 14: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Why these organ systems?Why these organ systems?

Humanitarian reasons• Significant health problems in US

and worldwide

Opportunistic reasons• Right mix of challenge and doability• Clinical buy-in• Funding opportunities• Experience of investigators• Publishing opportunities

Page 15: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Engineering GoalsEngineering Goals

1. “One-Stop Shopping”• Plan, Do, and Verify – in one sitting

2. “Plug & Play Surgery”• Rapidly configure for target organ,

imaging modality, and therapy• Predictable performance

3. Critical Mass of Components

Page 16: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

The War of RosesThe War of Roses

Serial Serial v.sv.s. Decoupled. Decoupled

Page 17: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Serial RobotsSerial RobotsWhere all joints move in synchWhere all joints move in synch

Pros:• Can move virtually anywhere• Lots of different motions • Smooth motion• Can work like human limbs

Cons:• Hard to constrain• Safety concerns• Complex control• Ugly math• Aggregating errors

Page 18: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Serial Robots ExamplesSerial Robots ExamplesWhere all joints move in synchWhere all joints move in synch

Most industry robots and derivatives

(SCARA, PUMA, Kawasaki…

Custom-developed robots (Neuromate

& see morning presentations…)

Page 19: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Decoupled RobotsDecoupled RobotsWhere joints can move selectivelyWhere joints can move selectively

Pros:• MAY BE INHERENTLY SAFE• Separates steps of surgery• Easy to constrain• Simpler control• Simpler Math• Curbed error aggregation

Cons:• Limited types of motions • Limited trajectory• Ragged motion

Page 20: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Main Benefits• Feels natural• Modular• Safe

Hopkins approach : decoupled RCM robotsHopkins approach : decoupled RCM robots

3D translation“move needle

to entry”

2D Rotation“orientneedle”

0D/1D/2D Insertion“drive needle”

1 2 3

Rotation about stationary fulcrum point

Page 21: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Chain drive & ball warm RCM robotsChain drive & ball warm RCM robots

Credit: Stonovici, Taylor, Whitcomb al.

• No friction• No backlash• 2x360o range• 1.6 kg• 20 cm

Page 22: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

EconoEcono--RCM a.k.a. “El Cheapo”RCM a.k.a. “El Cheapo”

• Inexpensive• Resizable• Mass-Producible• Possibly disposable

Credit: Lee, Webster, Kapur, Simaan, Taylor

Page 23: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

LARS: RCM robot w/ parallel linkagesLARS: RCM robot w/ parallel linkages

Credit: Taylor, Anderson, et al.

Page 24: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Photo: courtesy of Dr. Brian Davies

Davies: Goniometric arcs (for TURP)Davies: Goniometric arcs (for TURP)

Credit: Brian Davies et al.

Page 25: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

OverviewOverview

• Needles Everywhere• Microsurgical Assistants• Point and Click Surgery

• CT-guided systems• US-guided systems• MRI-guided systems

• Lessons Learned

Page 26: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Microsurgical assistant systemsMicrosurgical assistant systemsBackbone: SteadyHand robot

Credit: R. Taylor, D. Stoianovici, L. Whitcomb, A. Barnes

Page 27: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Freehand versus SteadyHandFreehand versus SteadyHand

Movies: CISST ERC, MADLAB, JHU

Credit: Kumar, Gupta, et al.

Steady Hand Motion:~ 0.02 mm drift~ 0.01 mm tremor

Free Hand Motion:~ 0.1 mm drift~ 0.05 mm tremor

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0 2 4 6 8 10

Time (Sec)

Tool

tip

posi

tion

(mm

)

Without RobotWith Robot

Page 28: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Forces in microsurgeryForces in microsurgeryInstrument tip forces in retina surgery are mostly below +/- 7.5 mN

human sensing threshold

-30

-20

-10

0

10

20

30

40

0 5 10 15 20 25 30

Time (sec)

Credit: Gupta, Berkelman, Kumar et al.

1-DOF 3-DOF

Force Scaling with SteadyHand robot

Page 29: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Application: retinal vein Application: retinal vein cannulationcannulationInsertion of a needle into the lumen (interior) of a retinal vein or artery in order to introduce theraputic drugs. May affect 200,000 patients per year in the US alone.

Credit: Kumar, Jensen, Hager, et al.

Cannulation of a 100 mm blood vessel w/ SteadyHand

2 mm

Page 30: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Application: mapping the Application: mapping the scalascala mediamedia• New high resolution US probe

(collaboration with K. Shung, Biomedical Ultrasonics Laboratory at USC)

• 30 µm resolution• Integrated US probe and needle

Credit: Rothbaum/Roy/Mustufa/Niparko/Francis/Whitcomb

Page 31: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

SteadyHand robot assisted injections of nuclear material into mouse eggs and embryos

– “Freehand” guiding– Visual virtual fixture assistance– Semi-autonomous

Credit: R. Kumar, A. Kapoor, R. Taylor

Application: cell manipulationApplication: cell manipulation

Industry partner: Foster-Miller, Inc

Page 32: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

““Point & Click” needle surgeryPoint & Click” needle surgery

Digital images

Planning & control computer

Patient

Imager

Robot

Coordinates

Physician

CT,MRI, US, XRT,

PET,SPECT

Page 33: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

OverviewOverview

• Needles Everywhere• Microsurgical Assistants• Point and Click Surgery

• CT-guided systems• US-guided systems• MRI-guided systems

• Lessons Learned

Page 34: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

The problem in CT guidanceThe problem in CT guidance

• Hand-eye coordination • Image is shown off the surgical field• Split attention (b/w display and patient)

Page 35: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

CTCT--guided prostate biopsyguided prostate biopsy

Credit: Stoianovici, Patriciu, Fichtinger, et al.

Page 36: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

An embodiment: 3An embodiment: 3--DOF robotDOF robotNeedle

1-DOF needle injector

Joysticks and safety switches

Amplifier box

Table side robot mount

2-DOF RCM

7-DOF passive

arm

Locking arm

Credit: Stoianovici, Masamune

Page 37: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Another one: 5Another one: 5--DOF robotDOF robot

Credit: Stoianovici, Patriciu

Page 38: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

P1

P2

P3P4

P5

P6

P7

SingleSingle--Slice registration of robot to CTSlice registration of robot to CT

L

P1

cd23= | P2 → P3 |

P3

| P1 → P2 |= d12

P2

Image plane

c = L* d23/(d12+d23)

Credit: Susil, Taylor, Masamune et al.

Closed form: fast and computationally

robust

Page 39: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

IntraIntra--op treatment planningop treatment planning

Credit: A. Tanacs

Page 40: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Similar for many organs…Similar for many organs…

Credit: A. Tanacs

Page 41: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Robot registered to CT from a single image using stereotactic frame on the end-effector

Credit: D. Stoianovici, L. Kavoussi, A. Patriciu, S. Solomon (JHU Bayview)

CTCT--guided kidney biopsyguided kidney biopsy

Page 42: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

DICOM images

CT table

CT computer &DICOM server

Path planning& robot control Robot

Patient

CT gantry

Vitrea

Anatomic rendering

view from inferior view from superior

IntraIntra--cranial hemorrhage removalcranial hemorrhage removal

Blind spotsCollision3DOF insufficient

Credit: Ellis, Fichtinger, et al.

Page 43: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Robot registered to CT using the scanner’s alignment laser

Credit: D. Stoianovici, L. Kavoussi, A. Patriciu, S. Solomon, JHU Bayviewand G. Fichtinger, ERC

CTCT--guided to lung biopsyguided to lung biopsy

Page 44: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Motion tracking in spine biopsyMotion tracking in spine biopsy

TrackerInvasive

Credit: Xu, Cleary, Fichtinger et al.

Page 45: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

CTFCTF--Guided Lung BiopsyGuided Lung Biopsy

Frame Grabber

3D CT Volume

CTCT Fluoroscopy

Register real-time CTF to CTThen compensate with robot

Credit: Xu, Cleary, Fichtinger et al.

Page 46: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

CTFCTF--Guided Lung BiopsyGuided Lung BiopsyPreliminary Results

Planned needle path Actual needle path

On synthetic human image data, respiratory excursion: 0-20 mm:

• Average tracking error is 0.6 mm

• The standard deviation of the error is 0.4mm.

Tracking

Overall Needle Placement Visually correct, not yet quantified

Page 47: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

OverviewOverview

• Needles Everywhere• Microsurgical Assistants• Point and Click Surgery

• CT-guided systems• US-guided systems• MRI-guided systems

• Lessons Learned

Page 48: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

The challenge of US guidanceThe challenge of US guidance

• Operator dependent• Invasive (deforms tissue)• No image outside the body

Page 49: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Salcudean et al.: 5Salcudean et al.: 5--DOF DOF teletele--echographechograph

Credit: Tim Salcludean et al.

Page 50: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

DegoulangeDegoulange tete al.: al.: HippocrateHippocrate teletele--echographechograph

Page 51: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

TroccazTroccaz et et alal.: 6.: 6--DOF DOF teletele--echographechograph• Master-Slave system• Remote exam by an expert• With force feedback

(Phantom)• US probe moved by a light

6DOF robot • Several communication

media (RNIS, LAN, VTHD, etc.)

• Clinical validation (Brest/Grenoble – aorticabdominal aneurysms)

Page 52: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

The Hopkins TRCM RobotThe Hopkins TRCM Robot

Credit: Randy Goldberg

L0LT

L1

L2L3

USM

Page 53: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Credit: Randy Goldberg

TRCM in action

Page 54: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Robotic 3D US w/ LARS robotRobotic 3D US w/ LARS robot

Both direct control and force compliant mode

Pixel/voxel ratioRobot: 1.5 - 1.6Freehand: 0.3 - 0.8 Credit: Boctor, Fischer, et al.

Page 55: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Hopkins dual arm testbed

Credit: Boctor, Fischer, Taylor, Fichtinger, et al.

Testbed for calibration, control, and interventions

Page 56: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Dual arm alternative configurations

Credit: Boctor, Fischer, Taylor, Fichtinger, et al.

Passive / Passive•Freehand 3D Ultrasound•Passive arm for needle

Passive / Active•Freehand 3D Ultrasound•Robot Needle Placement

Active / Passive•Robotic 3D Ultrasound•Passive arm for needle

Active / Active•Robotic 3D Ultrasound

•Robot Needle Placement

Page 57: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

“Quasi-decoupled kinematics”

Virtual RCM needle placement• Needle calibrated to tracker• Pose from tracker

Credit: Boctor, Webster, Okamura, Fichtinger

Un-calibrated kinematics ☺

• 3-DOF Cartesian manipulator• 2-DOF rotation module• Unencoded passive arm• EM tracker (FOB)

Page 58: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

CISUS CISUS –– SlicerSlicer--based planning and based planning and control systemcontrol system

Page 59: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

US probe calibration US probe calibration ––a plethora a plethora of problemsof problems

⎟⎟⎟⎟⎟

⎜⎜⎜⎜⎜

=

⎟⎟⎟⎟⎟

⎜⎜⎜⎜⎜

=

=

1000

,

10 xy

x

x

xPR

RT

TC

x

CvSuS

P

PTTTC

(Courtesy of R. Prager)

DO NOT UNDER ESTIMATE IT

The formulation of transformations in the cross-wire

Page 60: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Dual arm system: preliminary results

Average Error: ~3mm (from 10 trials)Error Propagation:

US Calibration: 1.2 mmFloating point truncation: 1-2 pixelsNeedle tip calibration error: ~0.5 mmEM sensor uncertainty: 2.54mm

Credit: Boctor, Fischer, Fichtinger, et al.

Page 61: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

US Machine

RCM-PAKY 5-DOF Needle Driver Robot

C-arm

3D Slicer SurgicalWorkstation

BiopsyNeedle

Active needle / passive US Active needle / passive US –– live piglive pig

Page 62: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

US Elastography in segmentation and US Elastography in segmentation and monitoringmonitoring

US probe

LARS RobotProbe holder

RFA device

Liver

Force sensor

Credit: Boctor, Fichtinger, Taylor, Choti et al.

B-Mode image PathologyStrain image

RFA device

Gel-Liverphantom

Compression plate

Page 63: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

US probe

Needle-guidingTemplate

TRUS-guided prostate brachytherapyUltrasound

Prostate Needle (in)

Needle (out)

NEEDLE

TRUS

PROSTATE

Courtesy of Paul Meskell, Beth Israel, Boston

Page 64: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Plan versus result

PRE-OP PLAN

POST-OP RESULT

☺☺

Courtesy of Burdette Medical Systems, Inc.

Page 65: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Why do implants go wrong?Why do implants go wrong?(1) Needle Deflection(1) Needle Deflection

Prescribed path

Actual needle path

Prostate

Prostate

(2) Motion and Deformation (2) Motion and Deformation Caused by Needle and ProbeCaused by Needle and Probe

Deformed Prostate

Seeds in relaxed prostate

Prostate

Prostate

TRUS probe IN

TRUS probe OUT

(3) Intra(3) Intra-- and Postand Post--Operative Operative EdemaEdema

Page 66: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Interplant® Interplant® JHUJHUInterplant® FDA-

approved treatment planning & monitoring

computer system

Physician

Ultrasound

Robot

Patient

Robot control

Our take: robotic assistance

Industry partner: CMS Burdette Medical

Page 67: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

7-DOF passive arm

2-DOF rotation motion stage

1-DOF Needle insertion stage

Ultrasound stepper

Mounting bridge

Ultrasound probe

3-DOF Cartesian motion

66--DOF decoupled DOF decoupled AccubotAccubot™™

Credit: Stoianovici, Whitcomb, Burdette, Fichtinger

Ultrasoundprobe

RCM Robot

Needle

Page 68: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Needles and seeds tracked in TRUSNeedles and seeds tracked in TRUS

Needle captured in live transverse image Seed captured in live sagittal image

Credit: Stoianovici, Whitcomb, Burdette, Fichtinger

Page 69: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Phantom results• Reached all relevant locations in prostate• Demonstrated arbitrary entry and angle• Aiming error:

• ~ 2.0 mm• worst case 2.5 mm• 80% in 2 mm margin

• Depth error:• ~ 2.5 mm• worst case 5.0 mm• 70% in 2.5 mm margin

• Large size, cumbersome calibration• Difficult to capture implants in TRUS

Credit: Stoianovici, Whitcomb, Burdette, Fichtinger

Page 70: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

44--DOF decoupled brachytherapy robotDOF decoupled brachytherapy robot

• Integrated with TRUS• Manual but encoded needle

insertionCredit: Kronreif, Burdette, Iordachita, Kazanzides, Fichtinger

Page 71: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Fusion of TRUS and XFusion of TRUS and X--ray fluoroscopyray fluoroscopy

Credit: Chirikjian, Burdette, Fichtinger

Prostate

TRUS imaging Optimize plan Insert needle

to see soft tissue

to see implants

Industry partner: CMS Burdette Medical

X-ray and TRUS are not concurrent &mutually exclusive

Page 72: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Protective rectal sheath w/ 6Protective rectal sheath w/ 6--DOF XDOF X--ray fiducialsray fiducials

Credit: Wyrobek, Mustufa, Jain, Burdette, Fichtinger

Air channel cover

Protective sheath

Quick release mount

Sheath mount to TRUS stepper

Depressurizing channels

X-ray fiducials

Page 73: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Register each image to the fiducialRegister each image to the fiducialSource1

Image1

Source2

Image2

0.090.140.120.320.030.05STD

0.150.210.210.550.040.07Mean

ZYXZYX

Rotation (deg)Translation (mm)

Credit: Jain, Zhou, Chirikjian, Burdette, Fichtinger

Page 74: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

IntraIntra--rectal robot around TRUS probe rectal robot around TRUS probe –– decoupled 2+2+2 DOFdecoupled 2+2+2 DOF

Needles bent inside Needles bent inside curved guidescurved guides

Prostate stabilizer

2-DOF for needle insertion

(currently manual)

Prostate

2-DOF for half sheath

2-DOF for TRUS probe

Credit: Okamura, Schneider, FichtingerIndustry partner: Acoustic MedSystem

Page 75: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

First phantom experimentsFirst phantom experiments

• Mean 2.5 mm (p=0.30)

• Reduced tissue deformation

Credit: Okamura, Schneider, Fichtinger

Page 76: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

OverviewOverview

• Needles Everywhere• Microsurgical Assistants• Point and Click Surgery

• CT-guided systems• US-guided systems• MRI-guided systems

• Lessons Learned

Page 77: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

The MRI challengeThe MRI challenge

•No space•B field

60x200 cm

Page 78: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

US CT MRI

Why to bother?Why to bother?

•• 1M prostate biopsy, 200,000 new cancer 1M prostate biopsy, 200,000 new cancer •• TRUS imaging misses 20% cancerTRUS imaging misses 20% cancer•• Freehand biopsy may miss <1 cc nodesFreehand biopsy may miss <1 cc nodes

Page 79: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Masamune et Masamune et alal.: MRI compatible .: MRI compatible manipulatormanipulator

Piezo-electric deviceUltrasound motor

PlasticsBearings

CeramicsBall bearings

SUS304、 fluoroplasticFeed screw

PlasticsScrew, Nuts

MaterialPartsMaterials

Page 80: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

TroccazTroccaz et et alal.: .: PatientPatient--mountedmounted 66--DOF robotDOF robot

• No trajectory limitation• Embedded localization• Pneumatic actuation• Accuracy <1.5mm

Page 81: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Fischer et al.: in-MRI breast Biopsy

Credit: Fischer et al, Karlsruhe

Page 82: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Brigham open MRI robot testbedIRB-approved for Robot-Assisted Prostate

Biopsy/Brachytherapy in Open-MRI

Credit: DiMaio, Chinzei, Hata, Kikinis, Jolesz, Tempany

Page 83: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

10mm

20G needles

Beads

20G needle

Brigham open MRI robot testbedIRB-approved for Robot-Assisted Prostate

Biopsy/Brachytherapy in Open-MRI

Credit: DiMaio, Chinzei, Hata, Kikinis, Jolesz, Tempany

Page 84: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Common system architecture

Credit: DiMaio, Kikinis

Page 85: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

A B

C D

Common system interface

Page 86: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

MRI Scanner Computer

Patient

MRI Scanner

Robot

Physician

Images

Physician’s Interface

Targeting Parameters

Real-time Tracking System

Position

Prostate interventions in highProstate interventions in high--field MRIfield MRI

Credit: Susil, Krieger, Menard, Coleman, Whitcomb, Atalar, Fichtinger

Page 87: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Rectum wall

Needle

Kinematic concept: decoupled Kinematic concept: decoupled 33--DOFDOF

3

Prostate

12

3 Degrees Of Freedom1. Translate2. Rotate3. Insert needle

End-effector

Page 88: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

The endThe end--effectoreffector

Movable needle guide

NeedleIntrarectal imaging coil

Stationary rectal sheath Positioning

stage

Tracking coil

Page 89: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Integrated robot in canine studiesIntegrated robot in canine studies

Page 90: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Needle voidAfter insertion

TargetRectalsheath

Prostategland

Prostategland

Rectalsheath

Target Urethra Urethra

BEFORE INSERTION AFTER INSERTION

Page 91: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Liquid injections in dogLiquid injections in dog

FSPGRTE = 1.2 FOV = 24x24cmTR = 6 ST = 10mm FA = 90o 256x128BW = 62.5KHz

0.80 sec/image

Prostate

Needle void

Real-time image in the plane of the needle

FSPGRTE = 2.8 FOV = 16x16cmTR = 85 ST = 3mm FA = 60o 256x256BW = 31.25KHz NEX=4Scan time = 3:00

Coverage of a 5 mm transverse slice at a time

Page 92: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Seed placement in dogSeed placement in dog

Seed #1

1cm

Target Needle Seed

error ~ 2 mmno motion

Seed #2

1cm

error ~ 2 mmno motion

Seed #3

1cm

error ~ 2 mmno motion

In-plane pattern achieved

Page 93: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

HumanHuman--grade devicegrade device

17mm OD

Page 94: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Clinical trials in progressClinical trials in progress

~20 patients (seed placement & biopsy)

Page 95: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Biopsy result 1Biopsy result 1

Page 96: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Biopsy result 2Biopsy result 2

Page 97: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Some lessons learnedSome lessons learned

•• Tissue motion and deformation remain the greatest Tissue motion and deformation remain the greatest unsolved problemsunsolved problems

• “Brute force” solutions can help only to some extent• Image-based and sensory tracking provides only part of

the solution (and we are not even there)• Predictive models will play a role• Needle steering will play a role• Deploy needle from close range

shorter needle – less deflectionless room for actuation and maneuvering

• Intracavity needle deployment is most promising•• Future: deformable statistical atlases with tissue Future: deformable statistical atlases with tissue

properties linked to anatomical structures and properties linked to anatomical structures and medical conditionsmedical conditions

☺☺

Page 98: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Statistical Atlas for Optimal Prostate BiopsyStatistical Atlas for Optimal Prostate Biopsy

ATLAS WARPED TO PATIENT

INTRA-OP MRI RECONSTRUCTION

Apex

Base

Posterior

Anterior

Left Right6

7

4

3

12

5

ATLAS w/ cancer found

Samples: 200+ prostates w/ pathology at CDPR

INTRA-OP MRI

Data: Brigham

Template

98% sensitivity from7 needles

Credit: Davatzikos, Shen, Tempany, Fichtinger

Page 99: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Status:– Developed a kinematic model to

describe needle bending in soft tissue due to the asymmetry of a bevel tip

– Developed an apparatus to insert and steer very flexible needles

Plans:– Validate deterministic and stochastic

models for needle steering– Develop computer simulations that

include needle bending for training and planning

Active needle steeringActive needle steering

CLICK FOR MOVIECredit: A. Okamura, N. Cowan, G. Chirikjian, T. DeWeese, K. Murphy

Page 100: Research Issues in Robot-Assisted Needle Interventions

Copyright © CISST ERC, 1999-2003 Engineering Research Center for Computer Integrated Surgical Systems and Technology

Passive needle steeringPassive needle steering

Credit: Fichtinger, et. al

Straight

Screw

NeedleSteering conduit

Tilted

Screw

NeedleSteering conduit

NeedleStraight outer cannula

pre-tensed inner cannula

STRAIGHTNEEDLE

STEEREDNEEDLE

Telescopic cannula patented by Salcudean, Rohling, et al.

Steerable nozzle idea: patent by Fichtinger et. al