When Does Computational Imaging Improve...

When Does Computational

Imaging Improve Performance?

Oliver Cossairt

Assistant Professor

Northwestern University

Collaborators: Mohit Gupta, Changyin Zhou,

Daniel Miau, Shree Nayar (Columbia University)

How to capture the best quality photograph?

Iphone 4GPhotograph

Canon DLSRPhotograph

http://campl.us/posts/iPhone-Camera-Comparison

http://thehdrimage.com/tag/metering/

Digital vs. Film Photography

• Dynamic range fixed at time of exposure

1ms Exposure Time 4ms Exposure Time

8ms Exposure Time 16ms Exposure Time

Digital Camera

• Dynamic range can be extended computationally

High Dynamic Range Image

Film Camera

http://graphics.stanford.edu/courses/cs178-13/lectures/astrophotography-02may13-150dpi.pdf

Digital Photography: Noise

• Noise level fixed at capture time

• Limited by film grain size

Milky Way, 6 min exposure (Jesse Levinson)

Digital Camera• Noise can be

averaged away

• SNR unlimited in principle

Milky Way, 10x 6 min exposures averaged

Film Camera

• Take multiple pictures and computationally combine

Computational Imaging: Increased Functionality

Others• Multiview Stereo• Depth from Focus/Defocus• Tomography• Structured Light• Deconvolution microscopy• etc.

HDR Imaging Panoramic Stitching

Digital Holography

[Greenbum et al. ’12]

Image-Based Lighting

[Debevec et al. ’00]

Light Field Capture

[Wilburn et al. ’04]

[Caraso, Opt. Eng ‘06]

Focus Blur

• Blur is fixed at time of capture

M100 Galaxy captured by Hubble Telescope

Digital Camera

• Images can be deblurredvia deconvolution

M100 Galaxy after blind deconvolution

Film Camera

Digital Camera

• Long exposure produces blurry image

50 millisecond exposure

Motion Blur

Film Camera

• Short exposure avoids motion blur

• Image is noisy

1 millisecond exposure (noisy)

• Blur can be removed via deconvolution

(blurry)(deblurred)

Coded Blur and Multiplexing

CameraExposure

Time50 millisec

[Raskar et al. ’06]

Coded Blur and Multiplexing

CameraExposure

Time50 millisec

[Raskar et al. ’06]

Blur is shifted and summed copies

Coded Blur and Multiplexing

CameraExposure

Time50 millisec

[Raskar et al. ’06]

Which copies to keep?

Coded image capture for increased performance

Computational Imaging: Increased Performance

Reflectance

[Schechner ‘03][Ratner ‘07][Ratner ‘08]

Defocus Blur

[Hausler ’72][Nagahara ’08]

[Levin et al. ’07][Zhou, Nayar ’08]

[Dowski, Cathey ‘96]

Motion Blur

[Raskar ’06] [Levin ‘08][Cho ’10]

Multi/Hyper-Spectral

[Sloane ’79] [Hanley ’99]

[Baer ‘99][Wetzstein et al., ’12]

Light Field Capture

[Lanman ’08] [Veeraraghavan ’07]

[Liang ‘08]

Coded Aperture

[Mertz ’65] [Gottesman ’89]

Coded Imaging Performance

CameraExposure

Time50 millisec

Coded Exposure

Time50 millisec

Short Exposure

Vs.

Deblurred Image

When does computational imaging improve performance?

CameraExposure

Measuring Computational Imaging

Performance

• No diffraction

• Fully determined

Image Formation Model

Assumption:

A) Linear model of incoherent image formation

Scene CodedImage

ComputationalCamera

Image

Optical Coding Equation

CodedImage

CodingMatrix Noise

• Signal dependent / independent noise

• Ignore Dark current, fixed pattern

Affine Noise Model

Assumption:

B) Affine noise model (photon noise is Gaussian)

Noise Variance at kth Pixel:

photon noise

aperture, lighting, pixel size

read noise

electronics,ADC’s, quantization

• Photon noise modeled as Gaussian

(ok for more than 10 photons)

• Photon noise spatially averaged

Noise PDF:

Signal-level and photon noise depend on illumination

Lighting Conditions

Assumption:

C) Naturally occurring light conditions for photography

Ex) q=.5, R = .5, F/8, t = 6ms, p=6um

Isrc (lux)

Quarter moon Full moon Twilight Indoor lighting Cloudy day Sunny day

10-2 1 10 102 103 104

8×10-3 0.8 8.1 81.4 814.3 8143J (e-)

Signal level

Illumination

Illumination(lux)

Aperture ExposureTime (s)

PixelSize (m)

QuantumEfficiency

ReflectivityAverageSignal (e-)

[Cossairt et al. TIP ‘12]

For Gaussian noise, Mean-Squared-Error (MSE) can be computed

analytically

Measuring Performance

Observation:

1) Multiplexing performance depends on coding matrix

Ex) Coded Motion Deblurring

Long Exposure Coded Exposure

Impulse imaging (identity sampling)

Multiplexing vs. Impulse Imaging

Observation:

2) Multiplexing increases signal-dependent noise

Noise variance

Coded imaging (multiplexed sampling)

Noise variance

Increasedthroughput

Observation:

3) Performance depends on multiplexing and signal prior

Multiplexing vs. Impulse Imaging

SNR Gain over impulse imaging:

Decreases with C

Noise Dependent

Increases with C

Coding Dependent

Coded Aperture Astronomy

Decreasing contrast

Increasing scene points

[Mertz ’65]

Fresnel zone plate

[Sloane ’79]

Hadamard Multiplexing:

No SNR gain for large signal

Assume we have a PDF for images, e.g.

Image Prior Models

Assumption:

D) Signal prior models naturally occurring images

MSE difficult to express analytically when

Compute the Maximum A Posteriori (MAP) estimate

Data term Prior term

Other priors

•Total Variation (TV)

•Wavelet/sparsity prior

•Learned priors (K-SVD)

Power Spectra Prior

Image Priors and Noise

Observation:

4) Signal priors help more at low light levels

PSNR = 5.5 dB

Twilight

(10 lux)

PSNR = 16.4 dB

Denoise

Daylight

(10 lux)5

PSNR = 35 dB PSNR = 35.9 dB

Denoise

How to capture the best quality photograph?

Observations:

1) Multiplexing performance depends on coding matrix

2) Multiplexing helps most in low light

3) Performance depends on both multiplexing and signal prior

4) Signal priors help most in low light

Assumptions

A) Incoherent imaging

B) Affine noise model

C) Natural lighting conditions

D) Natural image prior

Example:

Motion Deblurring

Motion Deblurring vs. Impulse Imaging

CameraExposure

Time50 millisec

Computational Imaging (Coded Exposure)

Time50 millisec

Impulse Imaging (Short Exposure)

Vs.

What is the best possible coding performance we can get?

Optical efficiency (C)= total ‘on’ time

[Ratner ‘07]

[Ratner and Schechner ‘07]

Optical Efficiency (C)

SN

R G

ain

(G

)

Average signal level

Read noise

5 10 15 20 25 30

1

2

3

4

S-Matrix

Multiplexing Performance Bound

When Does Motion Deblurring Improve Performance?

Motion InvariantLevin et al.

Flutter ShutterRaskar et al.

Upper Bound on SNR Gain:

Performance depends only on lighting conditions!

[Cossairt et al. TIP ‘12]

Read

Noise

Average

signal level

q=.5, R = .5, F/2.1, p = 1um,

Maximum object speed (pixels/sec)

Twilight

(10 lux)

Flutter Shutter Simulation

Impulse (4ms) Flutter Shutter (180ms) Deblurred

Cloudy Day

(10 lux)3

PSNR = 12.4 dB PSNR = 10.1 dB

q=.5, R = .5, F/2.1, pixel size = 1um, read noise

PSNR = -7.2 dB PSNR = -3.0 dB

Example:

Extended DOF Imaging

http://en.wikipedia.org/wiki/Focus_stacking

Depth of Field

Microscope Tachinid Fly

Small DOF

http://en.wikipedia.org/wiki/Focus_stacking

Depth of Field

Microscope Tachinid Fly

Large DOF

F 1.4F 2.8F 5.6F 8.0

LensImage

Depth of Field and Noise

Small apertures have large depth of field and low SNR

LensSensor

Focal Sweep

400 600 900 1500 20001200 1700 (depth)

Point Spread Function (PSF)

[Hausler ‘72, Nagahara et al. ‘08]

Focal Sweep

400 600 900 1500 2000 (depth)1200 1700

+ + + + + + =

t = 1 t = 2 t = 3 t = 4 t = 5 t = 6 t = 7(400) (600) (900) (1500) (2000)(1200) (1700)

Integrated PSF

[Hausler ‘72, Nagahara et al. ‘08]

LensSensor

Quasi Depth Invariant PSF

Extended depth of field with a single deconvolution

1200mm

750mm

400mm2000mm

-25 0 25

0.016

0.012

0.008

0.004

0

1

60

1200mm

-25 0 25

0.016

0.012

0.008

0.004

0

750mm

400mm 2000mm

×

Traditional Camera PSFFocal Sweep PSF

Focal Sweep: Captured

75 m

50 m

Extended Depth of Field Telescope

Meade LX200 8’’ Telescope

Traditional Image

75 m

50 m

75 m

50 m

Focal Sweep: Processed2000mm FL

LensImage

Focal Sweep Without Moving Parts

Focal Sweep

LensImage

Diffusion Coding

Radial Diffuser

(No Moving Parts)

[Cossairt et al. Siggraph ‘10]

500 x 3 micron

Diffusion Coding: Evaluation

RM

S D

eblu

rrin

gErr

or

Depth (mm)

Cubic Phase PlateFocal Sweep

noise

Diffusion Coding

400 600 900 1200 1500 1700 2000

.07

.06

.05

.04

.03

.02

.01

[Dowski and Cathey ‘95]

[Hausler ‘72, Nagahara et al. ‘08]

[Cossairt et al. Siggraph ‘10]

Diffusion coding gives best performance without moving parts

Diffusion Coding F/1.8 (Deblurred)

Traditional F/1.8 Diffusion Coding F/1.8 (Captured)

Diffusion Coding vs. Traditional Camera

Traditional F/18 (Normalized)

Face Detection

Diffusion Coding Camera (F/2.0)Traditional Camera (F/2.0)

AnnularAperture

Mirror 2 Mirror 1 SensorDiffuser

80” Focal Length

8” dia

Diffusion Coded Telescope: Optical Design

Telephoto Focal Sweep with Deformable Optics

Canon 800mm EFL Lens

Deformable Lens

Sensor

[Miau et al. ICCP ‘13]

Telephoto Video Quality Comparison

Conventional EDOF (Deformable Lens)

Noise Variance:Impulse Camera

lenssensor

A

Focal Sweep Performance

Mean-Squared Error:

Noise Variance:

light increase

Focal Sweep

lens

diffuser

sensor

C*A

Mean-Squared Error:

[Cossairt et al. TIP ‘12]

When Does Defocus Deblurring Improve Performance?

Performance depends only on lighting conditions!

[Cossairt et al. TIP ‘12]

Read

Noise

Average

signal level

Focal sweep multiplexing gain can be expressed analytically

q=.5, R = .5, t = 20ms, p = 5um,

Maximum defocus at F/1 (pixels)

Focal Sweep

(F/2.0)

Traditional

(F/2.0)

Twilight

(10 lux)

Pixel size = 5um

Read noise

Focal Sweep Simulation

Daylight

(10 lux)5

PSNR = 35 db PSNR = 38.5 db

Traditional

(F/20.0)

PSNR = 5.5 db PSNR = 18.5 db

Focal Sweep

(F/2.0)

Focal Sweep Simulation (with Prior)

Daylight

(10 lux)5

PSNR = 35.9 dB / 35 dB PSNR = 39.6 dB / 38.5 dB

BM3D Algorithm: [Dabov et al. ‘06]

Twilight

(10 lux)

Traditional

(F/20.0)

PSNR = 16.4 dB / 5.5 dB PSNR = 22.8 dB / 18.5 dB

Traditional

(F/2.0)

Pixel size = 5um

Read noise

Simulated Focal Sweep Performance

[Cossairt et al., TIP ‘12]

Focal Sweep performance bound is weak at low light levels

Focal Sweep Performance

ImpulseImaging

Conclusions

• Results for Motion Deblurring, EDOF also applicable to many

other computational cameras

• Computational imaging performance should always be

measured relative to impulse imaging

• Computational imaging performance depends jointly on

multiplexing, noise, and signal priors

• Important question: “How much performance improvement

from multiplexing above and beyond use of signal priors?”

Visual Quality Metrics

[Cossairt et al., TIP ‘12]

SSIM Metric UQI Metric SSIM MetricVIF Metric

Performance bound roughly holds for all metrics

Computational Gigapixel Camera

BallLens

LensArray

BallLens

SensorArray

Computations

GigapixelImage

Also See: MOSAIC Program, Duke, UCSD, Distant Focus

Ball LensSensor

Pan/Tilt Motor

Computational Camera Design Prototype Camera

Resolution vs. Lens Scale

Pixels

Poin

t Spre

ad F

unction

0-50 50

PSF size increases linearly

Resolution vs. Lens Scale

Pixels

Poin

t Spre

ad F

unction

0-50 50

PSF size increases linearly

Scale (M)

RM

S D

eblu

rrin

gErr

or

1 12 25 37 50 62 75 87 100

0.02

0.04

0.06

0.08

0.10

0.12

0.14

Deblurring Error is sub-linear

[Cossairt et al. JOSA ‘11]