Coded Photography
Lenslet-based Light Field camera
[Adelson and Wang, 1992, Ng et al. 2005 ]
Light Field Inside a Camera
Stanford Plenoptic Camera [Ng et al 2005]
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses
Digital Refocusing
[Ng et al 2005]
Can we achieve this with a Mask alone?
Mask based Light Field CameraMask Sensor
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]
Mask based Light Field CameraMask Sensor
1/f0
Mask Tile
Cosine Mask Used
Captured 2D Photo
Encoding due to Mask
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]
2D FFT
Traditional Camera Photo
Heterodyne Camera Photo
Magnitude of 2D FFT
2D FFT
Magnitude of 2D FFT
In Focus Photo
LED
2D Photo
Out of Focus Photo: Open Aperture
Out of Focus Photo: Coded Aperture
Captured Blurred Photo
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]
Refocused on Person
Increase DoF +large aperture
Out of Focus Photo: Coded Aperture
Engineering the PSF when you cannot capture Lightfield
Digital Refocusing
Captured Blurred Photo
Digital Refocusing
Refocused Image on Person
Mask? SensorMask
SensorMask? SensorMask? Sensor
Digital Refocusing
Mask? SensorMask
SensorMask? Sensor
MaskSensor
Mask? Sensor
Heterodyne Light Field Camera
Digital Refocusing
2D Photo
4D Light Field
Computing 4D Light Field2D Sensor Photo, 1800*1800 2D Fourier Transform
2D FFT
Rearrange 2D tiles into 4D planes200*200*9*94D IFFT
4D Light Field
9*9=81 spectral copies
200*200*9*9
MaskSensor
MaskSensor
Digital Refocusing
Heterodyne Light Field Camera
Mask? Sensor
Mask = more information?
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan], Siggraph 2007
MERL Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Veeraraghavan, Raskar, Agrawal, Mohan & TumblinDifferences with Plenoptic Camera
• Micro-lens array
• Samples individual rays
• Needs alignment precision
• Some pixels wasted
• Narrowband Cosine Mask
• Samples coded comb of rays
• More flexible
• No wastage
- Half brightness, diffraction
Mask
Sensor
Microlens array
Sensor
Plenoptic Camera Heterodyne Camera
Novel Sensors
• Color– Foveon
• Dynamic Range– HDR Camera, Log sensing– Gradient sensing
• Identity– Demodulation
• 3D– ZCam, Canesta
• Motion– Line scan Camera– Flutter Shutter
Foveon: All Colors at a Single Pixel
High Dynamic Range
Fuji's SuperCCD S3 Pro
Sensor with high and low sensitivity sensors per pixel location
Gradient Camera
• Sense Pixel Intensity Difference with unknown locally adaptive gain
• Reconstruct image from 2D gradient field
Ramesh Raskar, MERLWork with Jack Tumblin, Northwestern U,
Amit Agrawal, U of Maryland
High Dynamic Range Images
Scene Intensity camera saturation map
Gradient camera saturation map
Intensity camera fail to capture rangeGradients saturate at very few isolated pixels
Natural Scene Properties
x1
105
x1
105
Intensity Gradient
Intensity Histogram Gradient Histogram
1 105 -105 105
Motion _ _
Line Scan Camera: PhotoFinish 2000 Hz
Figure 2 results
Photo with motion blur
Rectified Image to make motion lines parallel to scan lines.
Image Deblurred by solving a linear system.
Approx Cut-out
Fluttered Shutter Camera[Raskar, Agrawal, Tumblin] Siggraph2006
Coded Exposure Coded Aperture
Temporal 1-D broadband code
Spatial 2-D broadband code
Novel Sensors
• Color– Foveon
• Dynamic Range– HDR Camera, Log sensing– Gradient sensing
• Identity– Demodulation
• 3D– ZCam, Canesta
• Motion– Line scan Camera– Flutter Shutter
Perspective? Or Not?
Agrawala et al, Long Scene Panoramas, Siggraph 2006
Rademacher et al, MCOP, Siggraph 1998
Multiperspective Camera?
[ Jingyi Yu’ 2004 ]
Future ..• ‘Cloth-cam’: ‘Wallpaper-cam’
– Fusion of 4D light emission and 4D capture in the surface of a cloth…
• Human Augmentation– Cameras to replace human eyes, for blind or limited vision– Camera on the ‘back’
• More Sensors– GPS, Compass, Temperature, fingerprint recognition, face recognition– When, Where, What, How .. Why?
• Photo Sharing and Community:– Photo Clip and Scene Completion– City Scanning, Live
Light Sensitive Fabric
Bayindir, Fink 2004
Computational Photography
Novel Illumination
Novel Cameras
Scene: 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
R ≈ 0.0
G ≈ 0.2
B ≈ 0.8
Fixed Color Gamut
B
G
R
“Best” primaries compromise:
Wide Gamut vs. High Power
B
G
R
Wider color gamut
λ400nm 700nm550nm
Adaptive Color Primaries
Agile Spectrum Imaging
With Ankit Mohan, Jack Tumblin [Eurographics 2008]
C
B
A
A’
B’
C’
Pinhole
Lens L1
Prism orDiffraction Grating
Lens L2Sensor
Rainbow Plane
C’’
B’’
A’’
Scene
Rainbow Plane inside Camera
Computational Photography1. Epsilon Photography
– Low-level vision: Pixels– Multi-photos by perturbing camera parameters– HDR, panorama, …– ‘Ultimate camera’
2. Coded Photography– Mid-Level Cues:
• Regions, Edges, Motion, Direct/global– Single/few snapshot
• Reversible encoding of data– Additional sensors/optics/illum– ‘Scene analysis’
3. Essence Photography– High-level understanding
• Not mimic human eye• Beyond single view/illum
– ‘New artform’