Multi-Aperture Photography
Paul Green – MIT CSAILWenyang Sun – MERLWojciech Matusik – MERLFrédo Durand – MIT CSAIL
Motivation
http://photographertips.net
Portrait
Landscape
Small Aperture
Large Aperture
Depth of Field Control
Shallow Depth of Field
Large Depth of Field
plane of focus
Depth and Defocus Blur
sensor lens
defocus blur depends on distance from plane of focus
subject
rays from point in focus converge to single pixel
circle of confusio
n
Defocus Blur & Aperture
lens plane of focus
defocus blur depends on aperture size
aperture
http://photographertips.net
sensor
subject
circle of confusio
n
Goals
Aperture size is a critical parameter for photographers
■ post-exposure depth of field control
■ extrapolate shallow depth of field beyond physical aperture
Outline
Multi-Aperture Camera■ New camera design■ Capture multiple aperture
settings simultaneously
Applications■ Depth of field control■ Depth of field extrapolation■ (Limited) refocusing
Related WorkComputational Cameras
■ Plenoptic Cameras■ Adelson and Wang ‘92■ Ng et al ‘05■ Georgiev et al ‘06
■ Split-Aperture Camera■ Aggarwal and Ahuja ‘04
■ Optical Splitting Trees■ McGuire et al ‘07
■ Coded Aperture■ Levin et al ’07■ Veeraraghavan et al ’07
■ Wavefront Coding■ Dowski and Cathey ‘95
Depth from Defocus■ Pentland ‘87
Georgiev et al‘06
Aggarwal and Ahuja ‘04McGuire et al ‘07
Adelson and Wang ‘92
Levin et al ’07 Veeraraghavan et al ’07
Plenoptic Cameras
Capture 4D LightField■ 2D Spatial (x,y)■ 2D Angular (u,v
Aperture)
Trade resolution for flexibility after capture■ Refocusing■ Depth of field control■ Improved Noise
Characteristics
Lens Aperture
u
v
Sensor (x,y)
Lenslet Array
Subject
Lens (u,v)
1D vs 2D Aperture Sampling
u
v
Aperture
2D Grid Sampling http://photographertips.net
4 Samples4 Samples
u
v
Aperture
2D Grid Sampling
1D vs. 2D Aperture Sampling
Aperture
1D “Ring” Sampling
45 Samples45 Samples
http://photographertips.net
Optical Splitting Trees
General framework for sampling imaging parameters■ Beamsplitters ■ Multiple cameras
Large ApertureCamera
Small ApertureCamera
McGuire et al ‘07
Beamsplitter
Incoming light
Goals
■ post-exposure depth of field control
■ extrapolate shallow depth of field
■ (limited) refocusing
■ 1d sampling■ no beamsplitters■ single sensor■ removable
Outline
Multi-Aperture Camera■ New camera design■ Capture multiple aperture
settings simultaneously
Applications■ Depth of field control■ Depth of field extrapolation■ Refocusing
Optical Design Principles
Aperture
3D sampling■ 2D spatial■ 1D aperture
size■ 1 image for
each “ring”
Sensor
http://photographertips.net
Goal: Split aperture into 4 separate optical paths■ concentric tilted mirrors■ at aperture plane
Aperture Splitting
Tilted Mirrors
Aperture Splitting
Incoming light
Sensor
MirrorsFocusing lenses
Tilted Mirrors
Aperture Splitting
Photographic Lens
Aperture Plane
Relay system
Aperture splitting optics
New Aperture Plane
X
Ideally at aperture plane
, but not physically possible!Solution: Relay Optics to create virtual
aperture plane
Optical Prototype
Mirror Close-up
main lens
relay optics
mirrors
tilted mirrors
lenses
SLR Camera
Sample Data
Raw data from our camera
Ideally would be ringsGaps are from
occlusion
Point Spread Function Occlusion
combinedinner ring 1 ring 2 outer
Outline
Multi-Aperture Camera■ New camera design■ Capture multiple aperture
settings simultaneously
Applications■ Depth of field control■ Depth of field extrapolation■ Refocusing
DOF Navigation
0I 2I
1I 3I
Approximate defocus blur as convolution
DOF Extrapolation?
0I 1I 2I 3I
?EI
)(0 nn KII
)( nK - Circular aperture blurring kernel
Depends on depth and aperture size
What is at each pixel in ?
EEI
Blu
r siz
e
Aperture Diameter
Largest physical aperture
DOF Extrapolation Roadmap
capture
estimate blur fit
model
extrapolate blur
IIEE
I1I2
I0
I3
Blu
r siz
e
Aperture Diameter D
I1 I2
IE
I0
σ
I3
Largest physical aperture
Defocus Gradient
Defocus blur
o
sos
fd
fddfdG
)(
o
sos
fd
fddfd )(DG
odsd
σ D
G is slope of this line
Defocus Gradient Map
Defocus Gradient
focal length
aperture diameter
sensor distance
object distance
Blur proportional to aperture diameter
Optimization
solve for discrete defocus gradient values G at each pixel
Data term
Graph Cuts with spatial regularization term
i
Ni iGKIIGD )()( 1
0
Defocus Gradient Map
Smallest Aperture Image
Depth of Field Extrapolation
Synthetic Refocusing
Modify gradient labels and re-synthesize image
gradient map
“refocused” map
extrapolated f/1.8
“refocused” synthetic f/1.8
Synthetic Refocusing Video
Depth Guided DeconvolutionDeconvolve (deblur) with kernel given
by defocus gradient map
Before After depth-guided deconvolution
Defocus gradient mapSmallest aperture image
Discussion■ Occlusion
■ Could help depth discrimination (coded aperture)
■ Difficult alignment process■ Mostly because prototype
■ Refocusing limited by Depth of Field■ helped by depth-guided deconvolution
■ Texture required for accurate defocus gradient map■ Not critical for depth of field and refocus
Summary■Multi-aperture camera■1D sampling of aperture■Removable
■Post-Exposure depth of field control
■Depth of field extrapolation■ Limited refocusing■Depth-guided deconvolution
ThanksPeople
■ John Barnwell■ Jonathan
Westhues■ SeBaek Oh■ Daniel Vlasic■ Eugene Hsu■ Tom Mertens■ Britton Bradley■ Jane Malcolm■ MIT Graphics
Group
Funding■ NSF CAREER award
0447561■ Ford Foundation
predoctoral Fellowship■ Microsoft Research New
Faculty Fellowship■ Sloan Fellowship
Recommended
