Computational Imaging

7/27/2019 Computational Imaging

1/51

MIT Media Lab

Camera Culture

Ramesh Raskar

Jack Tumblin

Computational Photography:

Advanced Topics

Paul Debevec


2/51

Speaker: Jack Tumblin

Associate Professor of Computer Science at Northwestern Univ.

His LookLab group pursues research on new methods to capture

and manipulate the appearance of objects and surroundings, in the

hope that hybrid optical/computer methods may give us new ways

to see, explore, and interact with objects and people anywhere in

the world. During his doctoral studies at Georgia Tech and post-docat Cornell, he investigated tone-mapping methods to depict high-

contrast scenes. His MS in Electrical Engineering (December 1990)

and BSEE (1978), also from Georgia Tech, bracketed his work as

co-founder of IVEX Corp., (>45 people as of 1990) where his flight

simulator design work was granted 5 US Patents. He was an

Associate Editor of ACM Transactions on Graphics (2000-2006), a

member of the SIGGRAPH Papers Committee (2003, 2004), and in2001 was a Guest Editor of IEEE Computer Graphics and

Applications.

http://www.cs.northwestern.edu/~jet


3/51

Speaker: Paul Debevec

Research Associate Professor ,

University of Southern California and the

Associate director of Graphics Research,

USC's Institute for Creative Technologies.

Debevec's Ph.D. thesis (UC Berkeley, 1996) presented Faade,

an image-based modeling and rendering system for creating

photoreal architectural models from photographs. Pioneer in

high dynamic range photography, he demonstrated new image-

based lighting techniques in his films Rendering with Natural

Light (1998), Fiat Lux (1999), and The Parthenon (2004); he also

led the design of HDR Shop, the first high dynamic range image

editing program. At USC ICT, Debevec has led the developmentof a series of Light Stage devices used in Spider Man 2 and

Superman Returns. He is the recipient of ACM SIGGRAPH's first

Significant New Researcher Award and a co-author of the 2005

book High Dynamic Range Imaging from Morgan Kaufmann.

http://www.debevec.org


4/51

Speaker: Ramesh Raskar

Associate Professor, MIT Media Lab.

Previously at MERL as a Senior Research Scientist.

His research interests include projector-based graphics,

computational photography and non-photorealistic rendering.

He has published several articles on imaging and photographyincluding multi-flash photography for depth edge detection,

image fusion, gradient-domain imaging and projector-camera

systems. His papers have appeared in SIGGRAPH,

EuroGraphics, IEEE Visualization, CVPR and many other

graphics and vision conferences. He was a course organizer at

Siggraph 2002 through 2005. He was the panel organizer at

the Symposium on Computational Photography and Video inCambridge, MA in May 2005 and taught a graduate level class

on Computational Photography at Northeastern University, Fall

2005. He is a member of the ACM and IEEE.

http://raskar.info

http://www.media.mit.edu/~raskar


5/51

Overview

Unlocking Photography

Not about the equipment but about the goal

Capturing machine readable visual experience

Goes beyond what you can see through the viewfinder

Push the envelope with seemingly peripheral techniques and advances

Think beyond post-capture image processing Computation well before image processing and editing

Learn how to build your own camera-toys

Emphasis

Most recent work in graphics/vision (2006 and later)

Research in other fields: Applied optics, novel sensors, materials Review of 50+ recent papers and projects

What we will not cover

Minimum discussion of graphics/vision papers before 2006

Epsilon photography (improving camera performance by bracketing)

Film Cameras, Novel view rendering (IBR), Color issues, Traditional imageprocessing/editing


6/51

Traditional Photography

Courtesy: Shree Nayar

Lens

Detector

Pixels

Image


7/51

Traditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot:Single View, Single Instant, FixedDynamic range and Depth of fieldfor given Illumination in a Static

world


8/51

Computational Photography:Optics, Sensors and Computations

GeneralizedSensor

GeneralizedOpticsComputations

Picture

4D Ray Bender

Upto 4D

Ray Sampler

Ray Reconstruction

Merged braketed photos, Coded sensing


9/51

Computational Photography

Novel CamerasGeneralized

Sensor

Generalized

OpticsProcessing


10/51

Computational Photography Novel Illumination

Novel CamerasGeneralized

Sensor

Generalized

OpticsProcessing

Light Sources


11/51


Novel Cameras

Scene: 8D Ray Modulator

Generalized

Sensor

Generalized

OpticsProcessing

Light Sources


12/51


Novel Cameras


Display

Generalized

Sensor

Generalized

OpticsProcessing

Recreate 4D Lightfield

Light Sources


13/51


Novel Cameras


Display

Generalized

Sensor

Generalized

OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

RayReconstruction

GeneralizedOptics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field


14/51

What is Computational Photography?

Create photo that could not have been taken by a traditional Camera (?)

Goal: Record a richer, multi-layered visual experience

1. Overcome limitations of todays cameras

2. Support better post-capture processing

Relightable photos, Focus/Depth of field, Fg/Bg, Shape boundaries

3. Enables new classes of recording the visual signal

Moment [Cohen05], Time-lapse, Unwrap mosaics, Cut-views

4. Synthesize impossible photos

Wrap-around views [Rademacher and Bishop 1998]), fusion of time-lapsed events [Raskar et al 2004],

motion magnification [Liu et al 2005]), video textures and panoramas [Agarwala et al 2005].

5. Exploit previously exotic forms of scientific imaging

Coded aperture [Veeraraghavan 2007, Levin 2007], confocal imaging [Levoy 2004], tomography[Trifonov 2006]


15/51


1. Epsilon Photography Low-level vision: Pixels Multi-photos by perturbing camera parameters HDR, panorama, Ultimate camera

2. Coded Photography Single/few snapshot Reversible encoding of data Additional sensors/optics/illum Scene analysis: (Consumer software?)

3. Essence Photography Beyond single view/illum Not mimic human eye New art form


16/51

Epsilon Photography

Dynamic range Exposure bracketing [Mann-Picard, Debevec]

Wider FoV

Stitching a panorama Depth of field

Fusion of photos with limited DoF [Agrawala04]

Noise Flash/no-flash image pairs

Frame rate

Triggering multiple cameras [Wilburn04]


17/51

Goal: High Dynamic Range

Short Exposure

Long Exposure

Dynamic Range


18/51

Epsilon Photography

Dynamic range Exposure braketing [Mann-Picard, Debevec]

Wider FoV

Stitching a panorama

Depth of field

Fusion of photos with limited DoF [Agrawala04]

Noise

Flash/no-flash image pairs [Petschnigg04, Eisemann04]

Frame rate

Triggering multiple cameras [Wilburn05, Shechtman02]


19/51


1. Epsilon Photography Low-level Vision: Pixels Multiphotos by perturbing camera parameters HDR, panorama Ultimate camera

2. Coded Photography Mid-Level Cues:

Regions, Edges, Motion, Direct/global

Single/few snapshot Reversible encoding of data

Additional sensors/optics/illum Scene analysis

3. Essence Photography Not mimic human eye Beyond single view/illum

New artform


20/51

3D

Stereo of multiple cameras

Higher dimensional LF

Light Field Capture

lenslet array [Adelson92, Ng05], 3D lens[Georgiev05], heterodyne masks[Veeraraghavan07]

Boundaries and Regions

Multi-flash camera with shadows [Raskar08]

Fg/bg matting [Chuang01,Sun06]

Deblurring

Engineered PSF

Motion: Flutter shutter[Raskar06], Camera Motion [Levin08]

Defocus: Coded aperture [Veeraraghavan07,Levin07], Wavefront coding[Cathey95]

Global vs direct illumination

High frequency illumination [Nayar06]

Glare decomposition [Talvala07, Raskar08]

Coded Sensor

Gradient camera [Tumblin05]


21/51Marc Levoy

Digital Refocusing usingLight Field Camera

125 square-sided microlenses[Ng et al 2005]


22/51

3D



Light Field Capture





Deblurring

Engineered PSF


Defocus: Coded aperture [Veeraraghavan07,Levin07], Wavefront coding[Cathey95]




Coded Sensor



23/51

Depth

Edges

Left Top Right Bottom

Depth EdgesCanny Edges


24/51

3D



Light Field Capture





Deblurring

Engineered PSF


Defocus: Coded aperture [Veeraraghavan07,Levin07], Wavefront coding

[Cathey95]




Coded Sensor



25/51

Flutter Shutter CameraRaskar, Agrawal, Tumblin

[Siggraph2006]

LCD opacity switched

in coded sequence

C d d


26/51

Traditional

CodedExposu

re

Image ofStaticObject

DeblurredImage

DeblurredImage


27/51

3D



Light Field Capture





Deblurring

Engineered PSF


Defocus: Coded aperture [Veeraraghavan07,Levin07], Wavefront coding

[Cathey95]

Decomposition Problems

High frequency illumination, Global/direct illumination [Nayar06]


Coded Sensor



28/51

"Fast Separation of Direct and Global Components of aScene using High Frequency Illumination,"S.K. Nayar, G. Krishnan, M. D. Grossberg, R. Raskar,ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH),Jul, 2006.

S ti R fl t C t ith


29/51

normal imagecross-polarized

subsurfacecomponent

polarization difference(primarily)

specular component

Separating Reflectance Components withPolarization-Difference Imaging


30/51


1. Epsilon Photography Multiphotos by varying camera parameters HDR, panorama Ultimate camera: (Photo-editor)

2. Coded Photography

Single/few snapshot Reversible encoding of data Additional sensors/optics/illum Scene analysis: (Next software?)

3. Essence Photography High-level understanding

Not mimic human eye Beyond single view/illum

New artform


31/51


32/51

Blind Camera

Sascha Pohflepp,

U of the Art, Berlin, 2006


33/51

Capturing the Essence of Visual Experience

Exploiting online collections Photo-tourism [Snavely2006] Scene Completion [Hays2007]

Multi-perspective Images Multi-linear Perspective [Jingyi Yu, McMillan 2004]

Unwrap Mosaics [Rav-Acha et al 2008] Video texture panoramas [Agrawal et al 2005]

Non-photorealistic synthesis Motion magnification [Liu05]

Image Priors Learned features and natural statistics Face Swapping: [Bitouk et al 2008]

Data-driven enhancement of facial attractiveness [Leyvand et al 2008]

Deblurring [Fergus et al 2006, 2008 papers


34/51

Scene Completion Using Millions of PhotographsHays and Efros, Siggraph 2007


35/51










36/51

Andrew Davidhazy

Unwrap Mosaics + Video Editing


37/51

Unwrap Mosaics + Video Editing

Rav-Acha et al

Siggraph 2008


38/51










39/51

Motion Magnification

Liu, Torralba, Freeman, Durand, Adelson Siggraph 2005


40/51




41/51




42/51








Deblurring [Fergus et al 2006, 2007-2008 papers]


43/51

Face Swapping

Find Candidateface in DB andalign

Tune pose,lighting, color

and blend

Keep result withoptimizedmatching cost

[Bitouk et al 2008]


44/51


1. Epsilon Photography

Low-level vision: Pixels Multi-photos by perturbing camera parameters HDR, panorama, Ultimate camera

2. Coded Photography Mid-Level Cues:

Regions, Edges, Motion, Direct/global

Single/few snapshot Reversible encoding of data

Additional sensors/optics/illum

Scene analysis

3. Essence Photography High-level understanding

Not mimic human eye Beyond single view/illum

New artform


45/51

Submit your questions ..

1. Today

What makes photography hard?

What moments you are not able to capture?

2. Future

What do you expect in a camera orphoto-software you buy in 2020?

Please submit by break at 3:30pm

Panel Discussion at 5:10pm

Siggraph 2006


46/51

Siggraph 200616 Computational Photography Papers

Hybrid Images

Oliva et al (MIT)

Drag-and-Drop Pasting

Jia et al (MSRA)

Two-scale Tone Management forPhotographic Look

Bae et al (MIT)

Interactive Local Adjustment of Tonal Values

Lischinski et al (Tel Aviv)

Image-Based Material Editing

Khan et al (Florida)

Flash Matting Sun et al (Microsoft Research Asia)

Natural Video Matting using Camera Arrays

Joshi et al (UCSD / MERL)

Removing Camera Shake From a Single Photograph

Fergus (MIT)

Coded Exposure Photography: Motion Deblurring

Raskar et al (MERL)

Photo Tourism: Exploring Photo Collections in 3D

Snavely et al (Washington)

AutoCollage

Rother et al (Microsoft Research Cambridge)

Photographing Long Scenes WithMulti-Viewpoint Panoramas

Agarwala et al (University of Washington)

Projection Defocus Analysis for Scene Capture andImage Display

Zhang et al (Columbia University)

Multiview Radial Catadioptric Imaging for Scene Capture Kuthirummal et al (Columbia University)

Light Field Microscopy (Project)

Levoy et al (Stanford University)

Fast Separation of Direct and Global Components ofa Scene Using High Frequency Illumination

Nayar et al (Columbia University)


47/51

Siggraph 2007

19 Computational Photography Papers Image Analysis & Enhancement

Image Deblurring with Blurred/Noisy Image Pairs

Photo Clip Art

Scene Completion Using Millions of Photographs Image Slicing & Stretching

Soft Scissors: An Interactive Tool for Realtime High Quality Matting

Seam Carving for Content-Aware Image Resizing

Image Vectorization Using Optimized Gradient Meshes

Detail-Preserving Shape Deformation in Image Editing

Light Field & High-Dynamic-Range Imaging Veiling Glare in High-Dynamic-Range Imaging

Ldr2Hdr: On-the-Fly Reverse Tone Mapping of Legacy Video and Photographs Appearance Capture & Editing

Multiscale Shape and Detail Enhancement from Multi-light Image Collections Computational Cameras

Active Refocusing of Images and Videos

Multi-Aperture Photography

Dappled Photography: Mask-Enhanced Cameras for Heterodyned Light Fields and Coded ApertureRefocusing

Image and Depth from a Conventional Camera with a Coded Aperture Big Images

Capturing and Viewing Gigapixel Images

Efficient Gradient-Domain Compositing Using Quadtrees Video Processing

Factored Time-Lapse Video

Computational Time-Lapse Video (project page)

Real-Time Edge-Aware Image Processing With the Bilateral Grid


48/51

Siggraph 2008

19 Computational Photography Papers Computational Photography & Display

Programmable Aperture Photography: Multiplexed Light Field Acquisition

Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses

Light-Field Transfer: Global Illumination Between Real and Synthetic Objects

Deblurring & Dehazing

Motion Invariant Photography

Single Image Dehazing

High-Quality Motion Deblurring From a Single Image

Progressive Inter-scale and intra-scale Non-blind Image Deconvolution Faces & Reflectance

Data-driven enhancement of facial attractiveness

Face Swapping: Automatic Face Replacement in Photographs (Project)

AppProp: All-Pairs Appearance-Space Edit Propagation

Image Collections & Video

Factoring Repeated Content Within and Among Images

Finding Paths through the World's Photos

Improved Seam Carving for Video Retargeting (Project)

Unwrap Mosaics: A new representation for video editing (Project)

Perception & Hallucination

A Perceptually Validated Model for Surface Depth Hallucination

A Perception-based Color Space for Illumination-invariant Image Processing

Self-Animating Images: Illusory Motion Using Repeated Asymmetric Patterns

Tone & Color

Edge-preserving decompositions for multi-scale tone and detail manipulation

Light Mixture Estimation for Spatially Varying White Balance


49/51

Ramesh Raskar andJack Tumblin

Book Publishers: A K Peters

Siggraph 2008 booth: 20% off

Booth #821

A ti l


50/51

More .. Articles IEEE Computer,

August 2006 Special Issue

Bimber, Nayar, Levoy, Debevec, Cohen/Szeliski

IEEE CG&A, March 2007 Special issue

Durand and Szeliski

Science News cover story April 2007

Featuring : Levoy, Nayar, Georgiev, Debevec American Scientist

February 2008

Siggraph 2008 19 papers

HDRI, Mon/Tue 8:30am Principles of Appearance Acquisition and Representation

Bilateral Filter course, Fri 8:30am

Other courses .. (Citizen Journalism, Wedn 1:45pm)

First International Conf on Comp Photo, April 2009

Athale, Durand, Nayar (Papers due Oct 3nd)

Class: Computational Photography Advanced Topics


51/51

Module 1: 105 minutes

1:45: A.1 Introduction and Overview (Raskar, 15 minutes)

2:00: A.2 Concepts in Computational Photography (Tumblin, 15 minutes)

2:15: A.3 Optics: Computable Extensions (Raskar, 30 minutes)

2:45: A.4 Sensor Innovations (Tumblin, 30 minutes)

3:15: Q & A (15 minutes)

3:30: Break: 15 minutes

Module 2: 105 minutes

3:45: B.1 Illumination As Computing (Debevec, 25 minutes)4:10: B.2 Scene and Performance Capture (Debevec, 20 minutes)

4:30: B.3 Image Aggregation & Sensible Extensions (Tumblin, 20 minutes)

4:50: B.4 Community and Social Impact (Raskar, 20 minutes)

5:10: B.4 Panel discussion (All, 20 minutes)

Class: Computational Photography, Advanced Topics

Debevec, Raskar and Tumblin

Documents

Computational Imaging