Stereoscopic Video Overlay with Deformable Registration

Balazs VagvolgyiProf. Gregory Hager

CISST ERC

Dr. David Yuh, M.D.Department of Surgery

Johns Hopkins University

The CASA Project

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Today’s Surgical Assistant: A Simple Information Channel

The CASA Project

Stereo surface tracking

Stereo tool tracking

Virtual fixtures with

da Vinci Robot

Task graph execution system

HMM-based Intent Recognition

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Information Fusion with

da Vinci Display

Ultrasound

Capabilities of a Context-Aware Surgical Assistant (CASA)

Tissue Classification

PreoperativeImagery

The CASA Project

Stereo surface tracking

Stereo tool tracking

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Information Fusion with

da Vinci Display

Developing a Context-Aware Surgical Assistant (CASA)

PreoperativeImagery

Information Overlay

• Problem setting:– Given pre-operative scan

data from a suitable imagingmodality

– Video sequence from a stereo endoscope

• Add value– Overlay underlying anatomy on the stereo video

stream (x-ray vision)

– Include annotations or other information tied to imagery

Key Problem: Nonrigid registration of organ surface to data

Inputs: What Do We Know?

1. Pre-operative 3D model- most probably volumetric- only a portion of it will be visible on the endoscope- anatomy will be deformed during the surgical procedure

2. Camera system properties can be measured- optical & stereo calibration- local brightness/contrast/color response

3. Stereo image stream- 3D surface can be reconstructed- texture information

4. A guesstimate of model–endoscope 3D relationship- We can guess where to start searching [i.e. patient position]

Outputs: What Do We Generate?

1. Position of 3D model registered to stereo image

2. Model deformed to the current shape of anatomy

3. Rendering a synthetic 3D view on the stereo stream

4. Everything done real-time

Original Image Stereo Data Deformed Mesh

2D 3D

All this in a flow chart

Stereo imagepre-processing

Building andoptimizing

disparity map

DeformableRegistration to

3D surface

3D texturetracking

Recognizingdeformations

optical parameters

stereo video stream

Imageoverlay

disparity

3D data

image data

parameters3D model

Classical Stereo Vision: The Problem

• Blocks of each image are compared using SAD

• Optimization for each block independently on entire depth range

+ Very fast implementation (GPU)

¬ Lousy results

Small Vision Systemfrom Videre Design

(w/o structured light):

• Input images downsized to several scale levels (½, ¼, …)• Each scale processed with the same algorithm

– Propagate coarse search results to the finer scale

+ Quality of disparity map is better + Even faster than single scale computation¬ Requires

structured light

Solution #1: Lighting and Multi-Scale

SVL implementation(using structured light):

• Solve a (spatially) global optimization with regularization

– O(D) = min SAD(D) + Smooth(D)

• GLOBAL optimum found in polynomial time

Solution #2: Dynamic Programming

1. Defining the recursive cost function

2. Memoization

3. Finding lowest cost path, which is the disparity map (DM in red)

SmoothnessError

Solution #2: Dynamic Programming

Dynamic Programming on Images

• Minor issue: previous approach applies to scanline

• Approximate DP applied to entire image

- 3D disparity space (D):

- Cost function (C):

- Memoization (P):

Dynamic Programming: Results

Dynamic Programming: In Vivo Results

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Stereo recordings from the da Vinci robot Focal length of ~ 700 pixels ~5mm baseline Distance to surface of 55mm to 154mm.

Raw Disparity Map Textured 3D Model

Surface to 3D Model Registration

• Inputs:– point cloud from the stereo surface modeler– point cloud generated from a model or volume image

• Outputs:- transformation to register the 3D model to the 3D surface

QuickTime™ and a decompressor

are needed to see this picture.

Results: Rigid Registration

Complete system (stereoplus registration) operatesat 5 frames/second

QuickTime™ and a decompressor

are needed to see this picture.

Current algorithm usesIPC with modificationsto account for occlusionsdue to viewpoint (z-buffer)

From Rigid to Deformable

• Calculate residual errors in z direction

• Define a spring-mass system

• Perform local gradient descent

Deformable Registration Results

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Final registration error of < 1mm exceptfor the area where the tool enters the image

Coming in CASA

The Language of Surgery

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Tool Tracking

Tissue Surface Classification

Thank you!

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.