BiDi ScreenDepth and Lighting Aware Interactive Display

Matthew HirschMIT Media Lab

Douglas LanmanBrown University

Ramesh RaskarMIT Media Lab

Henry HoltzmanMIT Media Lab

BiDi Screen

Inspiration

Light Sensitive Displays Depth Cameras

Multitouch

Beyond Multi-touch: Hover Interaction

• Seamless transition of multitouch to gesture

• Thin package, LCD

Beyond Multi-touch: Mobile

Laptops

Mobile

Results: Analysis

BiDi Screen

• Capture 3-D to enable depth and lighting aware interaction

• Collocated image capture and display

• Walk-up interaction

• Mobility and portability – thin device

Design Overview

Dis

play

with

em

bedd

ed o

ptic

al s

enso

rs

Sharp Microelectronics Optical Multi-touch Prototype

Design Overview

Dis

play

with

em

bedd

ed o

ptic

al s

enso

rs

LCD , displaying mask

Opt

ical

sen

sor a

rray

~2.5 cm~50 cm

Design Vision

Object Collocated Captureand Display

Bare Sensor

Spa

tial L

ight

Mod

ulat

or

Design Overview

Mask

Array of Virtual Cameras

Sensor

Manipulating an object with 3D gesture

Alternatives to capture depth from an LCD

• Adapted Touch

• Capacitive

• Resistive

• Optical

• Camera arrays

• Behind screen

• To sides

• Depth Sensors/Cameras

Adapting Traditional Touch?

• Resistive touch screen

• Confined to screen

• Capacitive

• Not lighting dependant

• Calibration problems for gesture

• No existing technology trend for gesture

?

Camera Arrays

• Cameras behind screen

• Interfere with backlight

• Expensive, Large

• Han , SecondLight, ThinSight

• No tech curve

• Cameras to side

• Transition region

• Second modality

?

Depth Cameras and Sensors

• Depth Cameras

• Behind screen w/ projector (MSR TouchlLight)

• No longer a thin device

• No light sensitivity

• Touch difficult

Theory: Depth from light-field capture

• LCD used for Spatial Heterodyne Light Field Capture and Display

• Pinholes or tiled broadband masks

• Separate Sensor and Mask

• Fourier Refocusing

• Depth from focus

• Blob tracking (gesture)

Theory: Lightfield

θx

y θ

x

.

Sensor integrates these rays

(Sensor)

Theory: Lightfield Frequency Domain

fθ

fx

θ

x

Lightfield FT

Fourier Slice Theorm

FT

Theory: LF Skew in Free-Space Propogation

θ

x

x

y

mask

θ

mask

Theory: Convolution with Delta Functions

Theory: Transforms

fθ

fx

θ

x

Lightfield FT

fmaskmaskFT

Tiled Broadband Mask

Theory: Spatial Heterodyning

x

y

mask

Multiplication in prime domain is convolution in frequency domain

Theory: Spatial Heterodyning

fθ

fxfmask

fθ

fx

Theory: Spatial Heterodyning

fθ

fx

Reconstructed LightfieldSpectrum

Band Limited

Desired PrototypeActual Prototype

Pipeline

Max ContrastOperator

Pipeline

Theory: Depth and Spatial Resolution

Theory: Depth and Spatial Resolution

Results: Analysis

Manipulating an object with 3D gesture

Navigating a 3D world with 3D gesture

Lighting a vritual scene with a real light

Limitations

• Requires separation between display and sensor (adds thickness)

• 2.5 cm for 50 cm range in our prototype

• 750mm for 8 cm range in iPhone like device

• Sensitive to room lighting or requires its own light source

• This is true of many optical systems

• Time multiplexing of display/capture

• Requires fast capture and screen refresh to stay below flicker fusion rate

• 240 Hz LCDs coming to market

Conclusions

• Future Work

• Dynamic Masks

• Change frequency characteristics to match scene

• Video capture / Video chat (higher resolution)

• SIGGRAPH Asia 2009 Paper

• BiDi Screen: Depth and Lighting Aware Interaction and Display

Conclusions

• Enable multitouch and gesture interaction on a thin display screen

• Walk-up interaction does not require gloves / fiducials

• Capture depth using array of vitual cameras

• Thin portable devices possible with area sensor