Projection Defocus Analysis for Scene Capture and Image

Display

Li Zhang Shree NayarColumbia University

IIS-00-85864

SIGGRAPH ConferenceJuly 2006, Boston, USA

Modeling Complex Scenes is Hard

Courtesy of Maia C @ flickr.com

LimitationGoal

The Limits of 3D Photography

Scene

Depth Map

Structured Light & Stereo

Depth from Camera Defocus

Photometric Stereo

Missing Data near

Occlusions

Inaccurate at Discontinuitie

s

Only for Single

Surface

Image-Complete

Accurate

for Complex Scenes

Projection Defocus Analysis

Projector Defocus Model

Robust Depth Recovery

Image Refocusing Video Composition

Enhanced Image DisplayIncreasing Depth of Field

Depixelation

Camera Defocus Cue

Courtesy of Koesbong @ flickr.com

Limitation of Depth from Camera Defocus

Focal Plane Surface Camera

Camera Defocus:

Depth from Projector Defocus

Focal Plane

Lamp

Projector

Depth from Projector Defocus

Focal Plane

Lamp

Surface Projector

Depth from Projector Defocus

Focal Plane

Lamp

Surface Projector

Projector Defocus:

Camera Defocus vs. Projector Defocus

Projector Defocus:

Camera Defocus:

Temporal Defocus MethodFocal Plane

LampSh

ift

Surface ProjectorCamera

Projector Defocus Model:

Experimental Setup

Projector

Half-mirror

Camera

Projector

Camera

Half-mirror

Experimental Setup

Projector

Half-mirror

Camera

Books and Gate: Sharp Discontinuities

Scene Depth Map

3D Model

Books and Gate: Sharp Discontinuities

Scene Depth Map

Refocus on Gate

+

Books and Gate: Sharp Discontinuities

Scene Depth Map

Refocus on Back

+

Application: Refocus Synthesis

Wrestlers: Curved and Specular Surfaces

Scene Depth Map

3D Model

Fence and Leaves: Complex Occlusions

Scene Depth Map

3D Model

Application: Refocus Synthesis

Playing Cards: Object Insertion

Scene Depth Map

Green Screen Capture Depth-based Composition

Projection Defocus Analysis

Projector Defocus Model

Robust Depth Recovery

Image Refocusing Video Composition

Enhanced Image DisplayIncreasing Depth of Field

Depixelation

Focused Projection at Multiple Depths

Focused Projection at Multiple Depths

A Multi-Projector Solution [Bimber & Emmerling 2006]

Focused Projection at Multiple Depths

Our Single Projector Solution Defocus Compensation

30 inches

30 inches

3 meters

?

?

Defocused ProjectionOriginal ImageCompensation Image

Defocus CompensationDefocus Kernel

Compensation Image

?

Defocused Projection

Defocus CompensationDefocus Kernel

Simple Sharpening Does NOT work.

Result

?

Defocus CompensationOriginal ImageDefocus KernelCompensation Image

CompensatedProjection

Front Plane Middle Plane Back Plane

Original Image

Compensation Results

Normal Projection

Defocus-Based Depixelation

Screen-Door Effect

Depixelation Results

DepixelatedProjection

Original Image

Normal Projection

Contributions

• Reducing Pixelation for Digital Projectors

• Increasing DoF without Changing Optics

• Estimating Depth from Projector Defocus

Limitations and Future Work

Real-time Compensation

Real-time Depth Recovery

Translucency Acquisition

Matte Estimation

The End