Click here to load reader
Upload
cameraculture-mit-media-lab
View
2.257
Download
1
Embed Size (px)
Citation preview
Raskar, Camera Culture, MIT Media Lab
Camera Culture
Ramesh Raskar
Camera CultureMIT Media Lab
Computational Displays in
4D, 6D and 8D
Slow Glass: Time Shift
http://baens-universe.com/articles/otherdays
Light of Other Days by Bob Shaw
http://www.fantasticfiction.co.uk/s/bob-shaw/other-days-other-eyes.htm
Shift Glass
Shift GlassSpace Shifting
Angle ShiftingTime Shifting
Illumination Shifting
4D 4Dt4D 4D
Capture
Analyze
Display
Shift Glass
Capture
Analyze
Display
5D: Looking around corners4D: Plenoptic Camera3D: Flutter Shutter Camera
6D: View and Lighting Aware4D: Rank Deficient4D: Netra for Optometry
4D, 6D, 8D: Augmented Light Field
Shift Glass
Can you look around a corner ?
Without any device in the line of sight
Femto-Photography: Higher Dimensional LFFemtoFlash
UltraFast Detector
Computational OpticsSerious Sync
Kirmani, Hutchinson, Davis, RaskarICCV’2009, Marr Prize Honorable Mention
Rescue and Planning
Robot, Car Path Planning
Endoscopy
Raskar, Camera Culture, MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G. Bawendi, Professor, Dept of Chemistry, MITJames Davis, UC Santa CruzAndreas Velten, Postdoctoral Associate, MIT Media LabAhmed Kirmani, RA, MIT Media LabTyler Hutchison, RA, MIT Media LabRohit Pandharkar, RA, MIT Media LabAndrew Matthew Bardagjy, RA, MIT Media LabEverett Lawson, MIT Media Lab
Ramesh Raskar, MIT Media Lab
Capture
Analyze
Display
5D: Looking around corners4D: Plenoptic Camera3D: Flutter Shutter Camera
6D: View and Lighting Aware4D: Rank Deficient4D: Netra for Optometry
4D, 6D, 8D: Augmented Light Field
Slow Display
Light Reactive Monostable Materials
16 Megapixel, 2 Watt
Day/Night visible
g
SlowDisplay.orgSaakes, Chiu, Hutchison, .., Inami, Raskar, Siggraph 2010 Etech
Demo
6D Photo Frames
One Pixel of a 6D Display = 4D Display
1 2
11
2D 2D 2D
Single Pixel of
6D FrameMartin Fuchs, Ramesh Raskar,Hans-Peter Seidel, Hendrik P. A. Lensch
Respond to Viewpoint + Ambient Light
6D DisplayLight sensitive 4D display
One Pixel of a 6D Display = 4D Display Raskar, Saakes, Fuchs, Siedel, Lensch, 2008
Beyond Multi-touch: Thin LCD for touch+hover
Laptops
Mobile
BiDi Screen: Multi-touch + Hover 3D interface
Overview: Sensing Depth from Array of Virtual Cameras in
LCD
Bits
Phot
ons
CV / Machine Learning
Optics
Sensors
Computational Displays
Signal Processing
Light TransportDisplay
s
HCI
View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale
Two Layer Displays
barrier
sensor/display
lenslet
sensor/display
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter, High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010
Front
Back
][][],[ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[i,k]
L[i,k]
f[i]
g[k]
L[i,k]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Components• 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]
f[i]
g[k]
L[i,k]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0,for ,21 min arg 2
GF, GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0,for ,21 min arg 2
GF, GFFGL
W
F
G
L̀~ =
Content-Adaptive Parallax Barriers
Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
Optimization: Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
Optimization: Iteration 10
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization: Iteration 20
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
Optimization: Iteration 30
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
Optimization: Iteration 40
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
Optimization: Iteration 50
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
Optimization: Iteration 60
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
Optimization: Iteration 70
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
Optimization: Iteration 80
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
Optimization: Iteration 90
rear mask: f1[i,j] front mask: g1[k,l]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
• Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks, emitted light field is rank-1
• Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks, emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
• Demonstrated light field display is a matrix approximation problem• Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0,for ,21 min arg 2
GF, GFFGL
W
F
G
L̀~ =
Content-Adaptive Parallax Barriers
Lightfield vs Hologram Displays
Is hologram just another ray-based light field?Can a hologram create any intensity distribution in 3D?Why hologram creates a ‘wavefront’ but PB does not?Why hologram creates automatic accommodation cues?What is the effective resolution of HG vs PB?
Parallax Barrier: Np=103 pix.
Hologram: NH=105 pix.
θp=10 pix
w
θH =1000 pixϕP∝w/d ϕH∝λ/tH
Fourier Patch
Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF < WDF
Lacks phase propertiesIgnores diffraction, interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherent/incoherent
Radiance = Positive/Negative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
62
L(x,θ) W(x,u) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)*
p Wm*
*
Rays: No Bending 1 Fresnel HG Patch
θ u
*
Zooming into the Light Field
s1
m2 m2s1*
s1
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform<t(x+xʹ/2)t*(x-xʹ/2)>
Interferencexʹ
x
(a) Two Slits, Coherent
t(x+xʹ/2)t*(x-xʹ/2)W(x,u)
2x
1x
Rank-1
t(x1)t*(x2)
Transform-1
u R45, D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(x,θ)L2(x,θ)
L3(x,θ)
d
z1
hH
r
z2
L1(x,θ) L2(x,θ) L3(x,θ)
s1m2
(a)
A B C
Vary IlluminationDirection:-5 ̊ , 0 ̊, 5 ̊
A B C A …
-5 ̊
5 ̊
0 ̊
No Slits24m
m
36mm
tH=25μm
w=125μmzH=10cm(c)
M2
M1
M3
ϕ1ϕ1
L1(x,θ)
L2(x,θ)
L3(x,θ)
dz1
r
z2
s1
m2
s1m2 s
1
m2
s1* s1 s
1
s1*Rank-1
Rank-1
Rank-3
Is hologram just another ray-based light field?Can a hologram create any intensity distribution in 3D?Why hologram creates a ‘wavefront’ but PB does not?Why hologram creates automatic accommodation cues?What is the effective resolution of HG vs PB?
MIT media lab camera culture EyeNetra.com
NETRA: Interactive Display for Estimating Refractive Errors and Focal Range
Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar
70
MIT media lab camera culture EyeNetra.com
Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar
71
NETRA: Near Eye Tool for Refractive Assessment
6.5 Billion people
4.5B withMobile phone
2Brefractive errors
0.6B uncorrected
refractive errors
NETRA at LVP Eye Institute
Retino scope w/
Lenses
Auto-refracto-
meter
Chart with
Lenses
In-Focus: Focometer Optiopia
Solo-health: EyeSite
NETRA
Technology Shining Light plus lenses
Fundus Camera
Moving lenses + target
Moving lenses + target
Reading chart on monitor
Cellphone + eyepiece
Cost to buy $2,000* ~$10,000 ~$100 ~$495 ~$200 -- $30
Cost per test ~$36 ~$36 ~$5 -- -- -- ~$1
Data capture No Comp. No No No Comp. Phone
Mobility <500g >10Kg 2kg 1kg <5kg >10Kg <100g
Speed Fast Fast Medium Medium -- Fast Fast
Scalability No No No Yes Probably No Yes
Accuracy 0.15 0.15 0.5 0.75 -- -- <0.5
Self evaluation No No Yes Yes Yes Yes Yes
Electricity Req No Yes No No -- Yes No
Astigmatism Yes Yes Yes/No No -- Yes Yes
Network No Yes No No No Yes Yes
Training High High High Medium Medium Low Low
* Phoropter-based: $5,000.00
Needs expert, Moving parts, Shining lasers
MIT media lab camera culture EyeNetra.com
Shack-Hartmann Wavefront Sensor
Expensive; Bulky, Requires trained professionals
Wavefront aberrometer
74
MIT media lab camera culture EyeNetra.com
Shack-Hartmann Wavefront Sensor
Laser
Sensor
75
Microlens Array
Planar Wavefront
Shack & Platt 1971Liang et al 1994
David Williams et al, Rochester
Spot Diagram
MIT media lab camera culture EyeNetra.com
Laser
Sensor
76
Displacement = Local Slope
of the Wavefront
Spot Diagram
Shack-Hartmann Wavefront Sensor
Shack-Hartmann ~ Lightfields
Levoy et al 2009 Zhang and Levoy 2009: Observable Light Field
Oh, Raskar, Barbastathis 2009: Augmented Light Field
MIT media lab camera culture EyeNetra.com
NETRA = Inverse of Shack-Hartmann
77
Spot Diagram on LCD
Cell Phone Display
Eye Piece
MIT media lab camera culture EyeNetra.com
NETRA = Inverse of Shack-Hartmann
78
Spot Diagram on LCD
Cell Phone Display
Eye Piece
MIT media lab camera culture EyeNetra.com
79
Spot Diagram on LCD
Inverse of Shack-HartmannUser interactively creates the Spot Diagram
Displace 25 points
MIT media lab camera culture EyeNetra.com
80
Spot Diagram on LCD
Inverse of Shack-HartmannUser interactively creates the Spot Diagram
Displace 25 points but 3 parameters
MIT media lab camera culture EyeNetra.com
Limitations• Children• Ability to align lines
– Retina, Animals
• Single Eye test– Other eye for convergence-forced accommodation
• Resolution is a function of the display DPI– Samsung Behold II – 160 DPI – 0.35D– Google Nexus One – 250 DPI – 0.2D– Apple iPhone 4G – 326 DPI – 0.14D
81
Capture
Analyze
Display
5D: Looking around corners
6D: View and Lighting Aware4D: Rank Deficient, multilayer4D: Netra for Optometry
4D, 6D, 8D: Augmented Light Field
MIT Media Lab Ramesh Raskar http://raskar.info
Shift Glass
F
G
L̀~ =WDF
Light Light FieldField
Augmented LF