Real-time Structured Light Depth Extraction

Kurtis Keller & Jeremy AckermanKurtis Keller & Jeremy Ackerman

Department of Computer ScienceDepartment of Computer Science

University of North CarolinaUniversity of North Carolina

Chapel Hill, North Carolina, USAChapel Hill, North Carolina, USA

EI2000

Divisions of talkDivisions of talk

Application and reason for researchApplication and reason for research Comparisons of various, relevant depth Comparisons of various, relevant depth

extraction systems extraction systems (structured light, etc..)(structured light, etc..)

First design with large digital projectorFirst design with large digital projector Second design with real time custom Second design with real time custom

projectorprojector Improvements...Improvements...

High speed structured light depth High speed structured light depth extraction for Augmented reality extraction for Augmented reality

surgical proceduressurgical procedures

Application: Application: MedicalMedical

Virtual / Augmented Reality view Virtual / Augmented Reality view from single incision.from single incision.

This information can be rendered This information can be rendered with textures to surgeon’s with textures to surgeon’s viewpoint.viewpoint.

Must be real-time, small, able to Must be real-time, small, able to image shiny/curved surfaces.image shiny/curved surfaces.

Comparison of various, Comparison of various, relevant Depth Extraction relevant Depth Extraction

SystemsSystems

Stereo laparoscopesStereo laparoscopes Traditional viewing of imagesTraditional viewing of images Automated depth extraction poorAutomated depth extraction poor

Structured light depth ExtractionStructured light depth Extraction Too slow with standard projector and cameraToo slow with standard projector and camera Can be used on curved surfacesCan be used on curved surfaces

Laser scan Laser scan Too slow for real time useToo slow for real time use MiniaturizationMiniaturization

Surface Texture and Z Surface Texture and Z depth scanning methods depth scanning methods for real-time medical usefor real-time medical use

Speed Automaticfeatureextraction

LinearSurfaces

CurvedSurfaces

Combineddepth / texturegrabbing

Line Scan Slow Yes Yes Yes Separate imageror scan required

OpticalStereo

Real time Somewhat Yes No Combined

StructuredLight

Almostreal time

Yes Yes Yes Combined

Structured lightStructured light (Binary encoded)(Binary encoded)

Start with coarse projected Start with coarse projected texture, then increase texture, then increase number of bars for finer number of bars for finer resolution.resolution.

11

2233

First SystemFirst System

SGI O2

DLPProjector

Camera

Structuredlight images

Videoimages

Range imageComposite video image

First Design:First Design: Structured Light with large Structured Light with large

digital projectordigital projector

Laparos c ope(projec tion)

Laparos c ope(c am era)

Cam era

D M D P rojec tor

M atc hed O ptic s

Body Cavity

T o VideoG rabber

First SystemFirst System Not Real timeNot Real time Utilized off the shelf (modified) Utilized off the shelf (modified)

componentscomponents TI micromirror projectorTI micromirror projector Custom lens to flex scope to laparoscopeCustom lens to flex scope to laparoscope

3D scans in lab and animal cadaver 3D scans in lab and animal cadaver teststests

BulkyBulky

First SystemFirst System((in operation)in operation)

Close to Real time (60Hz Close to Real time (60Hz frame imaging, 5Hz frame imaging, 5Hz updates)updates)

Custom componentsCustom components Projector LensProjector Lens Custom lens to flex scope to Custom lens to flex scope to

laparoscopelaparoscope

Off the shelf componetsOff the shelf componets TI Micromirror projectorTI Micromirror projector LaparoscopeLaparoscope Pulnics progressive scan cameraPulnics progressive scan camera

Second SystemSecond System (real time)(real time)

Second SystemSecond System

Almost Real time (~10 Hz)Almost Real time (~10 Hz) Utilized custom componentsUtilized custom components

Displaytech ferro-reflective LCD engine in custom projectorDisplaytech ferro-reflective LCD engine in custom projector Custom lens to flex scope to laparoscopeCustom lens to flex scope to laparoscope

Fits in with standard surgical environment Fits in with standard surgical environment Custom projector snap onto standard medical laparoscopeCustom projector snap onto standard medical laparoscope Light source 2 meters away from patient by fiber lightLight source 2 meters away from patient by fiber light Projector about the size of an orange, fitting to surgeons hand Projector about the size of an orange, fitting to surgeons hand

and light weightand light weight All exposed surfaces surgical steel and autoclaveable.All exposed surfaces surgical steel and autoclaveable.

High-speed ProjectorHigh-speed Projector

Miniature size Miniature size (50mm Dia x (50mm Dia x 75mm long)75mm long)

Custom Custom componentscomponents Displaytech Displaytech

640x480 180Hz. 640x480 180Hz. engineengine

Custom lens to Custom lens to laparoscopelaparoscope

High-speed Miniature High-speed Miniature ProjectorProjector

Attached to Attached to LaparoscopeLaparoscope

Exploded viewExploded view

Real timeReal time High-speed High-speed

componentscomponents Custom, 180Hz projectorCustom, 180Hz projector 180+ Hz 512x512 pixel camera180+ Hz 512x512 pixel camera High speed graphics processorHigh speed graphics processor

Hardware ImprovementsHardware Improvements

Color additive scanning by Color additive scanning by synchronized color wheelsynchronized color wheel

Single tube Stereo LaparoscopeSingle tube Stereo LaparoscopeReduction of Specular rejection Reduction of Specular rejection

problemsproblems Increased light level bit depthIncreased light level bit depth

Image processing/rejectionImage processing/rejection

Color WheelColor Wheel

FUTURE RESEARCH inFUTURE RESEARCH inHIGH SPEED STRUCTURED LIGHTHIGH SPEED STRUCTURED LIGHT

Obtain Ferro reflective (or other) displays Obtain Ferro reflective (or other) displays which are not hampered by crystal migrationwhich are not hampered by crystal migration

If possible, purchase graphics processor and If possible, purchase graphics processor and high speed camera from same vendorhigh speed camera from same vendor

Full projection imaging is being performed, not Full projection imaging is being performed, not micro internals that we emphasized. micro internals that we emphasized.

Virtual TeleconferencingVirtual Teleconferencing InspectionInspection SurveillanceSurveillance