Raskar Keynote at Stereoscopic Display Jan 2011

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Camera CultureMIT Media Lab

Computational Displays in

4D, 6D and 8D

Slow Glass: Time Shift

http://baens-universe.com/articles/otherdays

Light of Other Days by Bob Shaw

http://www.fantasticfiction.co.uk/s/bob-shaw/other-days-other-eyes.htm

Shift Glass

Shift GlassSpace Shifting

Angle ShiftingTime Shifting

Illumination Shifting

4D 4Dt4D 4D

Capture

Analyze

Display

Shift Glass

Capture

Analyze

Display

5D: Looking around corners4D: Plenoptic Camera3D: Flutter Shutter Camera

6D: View and Lighting Aware4D: Rank Deficient4D: Netra for Optometry

4D, 6D, 8D: Augmented Light Field

Shift Glass

Can you look around a corner ?

Without any device in the line of sight

Femto-Photography: Higher Dimensional LFFemtoFlash

UltraFast Detector

Computational OpticsSerious Sync

Kirmani, Hutchinson, Davis, RaskarICCV’2009, Marr Prize Honorable Mention

Rescue and Planning

Robot, Car Path Planning

Endoscopy

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Team

Moungi G. Bawendi, Professor, Dept of Chemistry, MITJames Davis, UC Santa CruzAndreas Velten, Postdoctoral Associate, MIT Media LabAhmed Kirmani, RA, MIT Media LabTyler Hutchison, RA, MIT Media LabRohit Pandharkar, RA, MIT Media LabAndrew Matthew Bardagjy, RA, MIT Media LabEverett Lawson, MIT Media Lab

Ramesh Raskar, MIT Media Lab

Capture

Analyze

Display

5D: Looking around corners4D: Plenoptic Camera3D: Flutter Shutter Camera

6D: View and Lighting Aware4D: Rank Deficient4D: Netra for Optometry

4D, 6D, 8D: Augmented Light Field

Slow Display

Light Reactive Monostable Materials

16 Megapixel, 2 Watt

Day/Night visible

g

SlowDisplay.orgSaakes, Chiu, Hutchison, .., Inami, Raskar, Siggraph 2010 Etech

Demo

6D Photo Frames

One Pixel of a 6D Display = 4D Display

1 2

11

2D 2D 2D

Single Pixel of

6D FrameMartin Fuchs, Ramesh Raskar,Hans-Peter Seidel, Hendrik P. A. Lensch

Respond to Viewpoint + Ambient Light

6D DisplayLight sensitive 4D display

One Pixel of a 6D Display = 4D Display Raskar, Saakes, Fuchs, Siedel, Lensch, 2008

Beyond Multi-touch: Thin LCD for touch+hover

Laptops

Mobile

BiDi Screen: Multi-touch + Hover 3D interface

Overview: Sensing Depth from Array of Virtual Cameras in

LCD

Bits

Phot

ons

CV / Machine Learning

Optics

Sensors

Computational Displays

Signal Processing

Light TransportDisplay

s

HCI

View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale

Two Layer Displays

barrier

sensor/display

lenslet

sensor/display

PB = dim displaysLenslets = fixed spatial and angular resolution

Dynamic Masks = Brighter, High spatial resolution

Parallax barrier

LCD display

Limitations of 3D Display

Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010

Front

Back

][][],[ kgifkiL

`

i

k

gfL

light box

Light Field Analysis of Barriers

g[k]k

f[i]i

L[i,k]

L[i,k]

f[i]

g[k]

L[i,k]

light box

`

FGL ~

G

Content-Adaptive Parallax Barriers

k

i F L~

Implementation

Components• 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]

f[i]

g[k]

L[i,k]

light box

`

FGL ~

F

G

L~

Content-Adaptive Parallax Barriers

k

i

0,for ,21 min arg 2

GF, GFFGL

F

G

`L~ =

Content-Adaptive Parallax Barriers

Rank-Constrained Displays and LF Adaptation

All dual layer display = rank-1 constraint

Light field display is a matrix approximation problem

Exploit content-adaptive parallax barriers

0,for ,21 min arg 2

GF, GFFGL

W

F

G

L̀~ =

Content-Adaptive Parallax Barriers

Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)

Optimization: Iteration 1

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

Optimization: Iteration 10

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Optimization: Iteration 20

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

Optimization: Iteration 30

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

Optimization: Iteration 40

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

Optimization: Iteration 50

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

Optimization: Iteration 60

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

Optimization: Iteration 70

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

Optimization: Iteration 80

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

Optimization: Iteration 90

rear mask: f1[i,j] front mask: g1[k,l]

reconstruction (central view)

))](([)]([

]))([(])[(

FGWFLWFGG

GFGWGLWFF

t

t

t

t

Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

Content-Adaptive Front Mask (1 of 9)

Content-Adaptive Rear Mask (1 of 9)

Emitted 4D Light Field

Conclusion

• Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks, emitted light field is rank-1

• Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks, emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels

• Demonstrated light field display is a matrix approximation problem• Introduced content-adaptive parallax barriers

‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

0,for ,21 min arg 2

GF, GFFGL

W

F

G

L̀~ =

Content-Adaptive Parallax Barriers

Lightfield vs Hologram Displays

Is hologram just another ray-based light field?Can a hologram create any intensity distribution in 3D?Why hologram creates a ‘wavefront’ but PB does not?Why hologram creates automatic accommodation cues?What is the effective resolution of HG vs PB?

Parallax Barrier: Np=103 pix.

Hologram: NH=105 pix.

θp=10 pix

w

θH =1000 pixϕP∝w/d ϕH∝λ/tH

Fourier Patch

Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009

Augmented Lightfield for Wave Optics Effects

Wigner Distribution Function

Light Field

LF < WDF

Lacks phase propertiesIgnores diffraction, interferrence

Radiance = Positive

LF

Augmented Light Field

WDF

ALF ~ WDF

Supports coherent/incoherent

Radiance = Positive/Negative

Virtual light sources

Free-space propagation

Light field transformer

Virtual light projectorPossibly negative radiance

62

L(x,θ) W(x,u) Wm= sincd = delta

q Wm

pq

p d(θ)

pqq d(θ)*

p Wm*

*

Rays: No Bending 1 Fresnel HG Patch

θ u

*

Zooming into the Light Field

s1

m2 m2s1*

s1

Rank-1 Rank-1

Algebraic Rank Constraint

x

u

-Transform<t(x+xʹ/2)t*(x-xʹ/2)>

Interferencexʹ

x

(a) Two Slits, Coherent

t(x+xʹ/2)t*(x-xʹ/2)W(x,u)

2x

1x

Rank-1

t(x1)t*(x2)

Transform-1

u R45, D

L2L1

L3

ϕ1

ϕ1

ϕ1

ϕ1

L1(x,θ)L2(x,θ)

L3(x,θ)

d

z1

hH

r

z2

L1(x,θ) L2(x,θ) L3(x,θ)

s1m2

(a)

A B C

Vary IlluminationDirection:-5 ̊ , 0 ̊, 5 ̊

A B C A …

-5 ̊

5 ̊

0 ̊

No Slits24m

m

36mm

tH=25μm

w=125μmzH=10cm(c)

M2

M1

M3

ϕ1ϕ1

L1(x,θ)

L2(x,θ)

L3(x,θ)

dz1

r

z2

s1

m2

s1m2 s

1

m2

s1* s1 s

1

s1*Rank-1

Rank-1

Rank-3

Is hologram just another ray-based light field?Can a hologram create any intensity distribution in 3D?Why hologram creates a ‘wavefront’ but PB does not?Why hologram creates automatic accommodation cues?What is the effective resolution of HG vs PB?

MIT media lab camera culture EyeNetra.com

NETRA: Interactive Display for Estimating Refractive Errors and Focal Range

Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar

70

MIT media lab camera culture EyeNetra.com

Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar

71

NETRA: Near Eye Tool for Refractive Assessment

6.5 Billion people

4.5B withMobile phone

2Brefractive errors

0.6B uncorrected

refractive errors

NETRA at LVP Eye Institute

Retino scope w/

Lenses

Auto-refracto-

meter

Chart with

Lenses

In-Focus: Focometer Optiopia

Solo-health: EyeSite

NETRA

Technology Shining Light plus lenses

Fundus Camera

Moving lenses + target

Moving lenses + target

Reading chart on monitor

Cellphone + eyepiece

Cost to buy $2,000* ~$10,000 ~$100 ~$495 ~$200 -- $30

Cost per test ~$36 ~$36 ~$5 -- -- -- ~$1

Data capture No Comp. No No No Comp. Phone

Mobility <500g >10Kg 2kg 1kg <5kg >10Kg <100g

Speed Fast Fast Medium Medium -- Fast Fast

Scalability No No No Yes Probably No Yes

Accuracy 0.15 0.15 0.5 0.75 -- -- <0.5

Self evaluation No No Yes Yes Yes Yes Yes

Electricity Req No Yes No No -- Yes No

Astigmatism Yes Yes Yes/No No -- Yes Yes

Network No Yes No No No Yes Yes

Training High High High Medium Medium Low Low

* Phoropter-based: $5,000.00

Needs expert, Moving parts, Shining lasers

MIT media lab camera culture EyeNetra.com

Shack-Hartmann Wavefront Sensor

Expensive; Bulky, Requires trained professionals

Wavefront aberrometer

74

MIT media lab camera culture EyeNetra.com

Shack-Hartmann Wavefront Sensor

Laser

Sensor

75

Microlens Array

Planar Wavefront

Shack & Platt 1971Liang et al 1994

David Williams et al, Rochester

Spot Diagram

MIT media lab camera culture EyeNetra.com

Laser

Sensor

76

Displacement = Local Slope

of the Wavefront

Spot Diagram

Shack-Hartmann Wavefront Sensor

Shack-Hartmann ~ Lightfields

Levoy et al 2009 Zhang and Levoy 2009: Observable Light Field

Oh, Raskar, Barbastathis 2009: Augmented Light Field

MIT media lab camera culture EyeNetra.com

NETRA = Inverse of Shack-Hartmann

77

Spot Diagram on LCD

Cell Phone Display

Eye Piece

MIT media lab camera culture EyeNetra.com

NETRA = Inverse of Shack-Hartmann

78

Spot Diagram on LCD

Cell Phone Display

Eye Piece

MIT media lab camera culture EyeNetra.com

79

Spot Diagram on LCD

Inverse of Shack-HartmannUser interactively creates the Spot Diagram

Displace 25 points

MIT media lab camera culture EyeNetra.com

80

Spot Diagram on LCD

Inverse of Shack-HartmannUser interactively creates the Spot Diagram

Displace 25 points but 3 parameters

MIT media lab camera culture EyeNetra.com

Limitations• Children• Ability to align lines

– Retina, Animals

• Single Eye test– Other eye for convergence-forced accommodation

• Resolution is a function of the display DPI– Samsung Behold II – 160 DPI – 0.35D– Google Nexus One – 250 DPI – 0.2D– Apple iPhone 4G – 326 DPI – 0.14D

81

Capture

Analyze

Display

5D: Looking around corners

6D: View and Lighting Aware4D: Rank Deficient, multilayer4D: Netra for Optometry

4D, 6D, 8D: Augmented Light Field

MIT Media Lab Ramesh Raskar http://raskar.info

Shift Glass

F

G

L̀~ =WDF

Light Light FieldField

Augmented LF