Introduction to Camera Challenges - Ramesh Raskar

Marc Levoy

The CityBlock Project

Precursor to Google Streetview Maps

Image Fusion & ReconstructionImage Fusion & Reconstruction• Single photo:Single photo: forces narrow tradeoffs: forces narrow tradeoffs:

– Focus, Exposure, aperture, time, sensitivity, noise,Focus, Exposure, aperture, time, sensitivity, noise,

– Usual result: Incomplete visual appearance.Usual result: Incomplete visual appearance.

Multiple photosMultiple photos, assorted settings , assorted settings for Optics, Sensor, Lighting, Processingfor Optics, Sensor, Lighting, Processing

• Fusion:Fusion: ‘Merge the best parts’‘Merge the best parts’

• Reconstruction:Reconstruction:Detect photo changes; Detect photo changes; compute scene invariantscompute scene invariants

High Dynamic Range ImagingHigh Dynamic Range Imaging

• Cameras have limited dynamic range

Small Exposure image, dark inside

1/500 sec

Large exposure image, saturated outside

¼ sec

Images from Raanan Fattal

High Dynamic Range ImagingHigh Dynamic Range Imaging

• Combine images at different exposures• Exposure Bracketing• [Mann and Picard 95, Debevec et al 96]

Images from Raanan Fattal

How could we put all thisinformation into oneimage ?

Tone Map 20 bit image for 8 bit DisplayTone Map 20 bit image for 8 bit Display

input smoothed(structure, large scale)

residual(texture, small scale)

Gaussian Convolution

BLUR HALOS

Naïve Approach: Gaussian Blur

Impact of Blur and Halos

• If the decomposition introduces blur and halos, the final result is corrupted.

Sample manipulation:increasing texture

(residual 3)

input smoothed(structure, large scale)

residual(texture, small scale)

edge-preserving: Bilateral Filter

Bilateral Filter: no Blur, no Halos

input

increasing texturewith Gaussian convolution

H A L O S

increasing texturewith bilateral filter

N O H A L O S

Bilateral Filter on 1D Signal

BF

p

Our Strategy

Reformulate the bilateral filter– More complex space:

Homogeneous intensity Higher-dimensional space

– Simpler expression: mainly a convolution Leads to a fast algorithm

weightsappliedto pixels

Attenuate High GradientsAttenuate High Gradients

I(x)1

105

1

Intensity

I(x)1

105

Intensity

Maintain local detail at the cost of global range

Fattal et al Siggraph 2002

Attenuate High GradientsAttenuate High Gradients

I(x)1

105

G(x)1

105

Intensity Gradient

I(x)1

105

Intensity

Maintain local detail at the cost of global range

Fattal et al Siggraph 2002

Attenuate High GradientsAttenuate High Gradients

I(x)1

105

G(x)1

105

Intensity Gradient

I(x)1

105

Intensity

Keep low gradients

Fattal et al Siggraph 2002

Gradient Compression in 1DGradient Compression in 1D

Gradient Domain CompressionGradient Domain Compression

HDR Image L Log L

Gradient Attenuation Function G

Multiply 2D Integration

Gradients Lx,Ly

Grad X

Grad Y

New Grad X

New Grad Y

2D Integration

Intensity Gradient ManipulationIntensity Gradient Manipulation

Gradient Processing

A Common Pipeline

This Section

Next Section

Grad X

Grad Y

New Grad X

New Grad Y

2D Integration

Gradient Processing

Local Illumination ChangeLocal Illumination Change

Original gradient field:

Original Image: f

*f

Modified gradient field: v

Perez et al. Poisson Image editing, SIGGRAPH 2003

Ambient FlashSelf-Reflections and Flash HotspotSelf-Reflections and Flash Hotspot

Hands

Face

Tripod

ResultAmbient

Flash

Reflection LayerReflection Layer

Hands

Face

Tripod

Intensity Gradient Vector Intensity Gradient Vector ProjectionProjection[Agrawal, Raskar, Nayar, Li SIGGRAPH 2005][Agrawal, Raskar, Nayar, Li SIGGRAPH 2005]

Intensity Gradient Vectors in Flash and Ambient ImagesIntensity Gradient Vectors in Flash and Ambient Images

Same gradient vector direction Flash Gradient Vector

Ambient Gradient Vector

Ambient Flash

No reflections

Reflection Ambient Gradient Vector

Different gradient vector direction

With reflections

Ambient Flash

Flash Gradient Vector

Residual Gradient Vector

Intensity Gradient Vector Projection

Result Gradient Vector

Result Residual

Reflection Ambient Gradient Vector

Flash Gradient Vector

Ambient Flash

FlashProjection = Result

Residual = Reflection Layer

Co-located Artifacts

Ambient

Recovering foreground layerRecovering foreground layer– Find tensor based on background image– Transform gradient field of foreground image

Foreground maskImage Difference

Dark Bldgs

Reflections on bldgs

Unknown shapes

‘Well-lit’ Bldgs

Reflections in bldgs windows

Tree, Street shapes

Background is captured from day-time scene using the same fixed camera

Night Image

Day Image

Context Enhanced Image

Mask is automatically computed from scene contrast

But, Simple Pixel Blending Creates Ugly Artifacts

Pixel Blending

Pixel Blending

Our Method:Integration of

blended Gradients

Nighttime imageNighttime image

Daytime imageDaytime image Gradient fieldGradient field

Importance Importance image Wimage W

Fina

l res

ult

Fina

l res

ult

Gradient fieldGradient field

Mixed gradient fieldMixed gradient field

GG11 GG11

GG22 GG22

xx YY

xx YY

II11

I2

GG GGxx YY

Reconstruction from Gradient FieldReconstruction from Gradient Field

• Problem: minimize errorI’ – G|• Estimate I’ so that

G = I’

• Poisson equation

I’ = div G

• Full multigridsolver

I’I’

GGXX

GGYY

Rene Magritte, ‘Empire of the Light’

Surrealism

actual photomontageset of originals perceived

Source images Brush strokes Computed labeling

Composite

Brush strokes Computed labeling

• No Flash:No Flash: Candle warmth, but high noise Candle warmth, but high noise• Flash:Flash: low noise, but no candle warmth low noise, but no candle warmth

Photography: Full of Tradeoffs...Photography: Full of Tradeoffs...

No-flash Flash

Image A: Warm, shadows, but too Noisy(too dim for a good quick photo)

No-flash

Image B: Cold, Shadow-free, Clean(flash: simple light, ALMOST no shadows)

MERGE BEST OF BOTH: apply‘Cross Bilateral’ or ‘Joint Bilateral’

(it really is much better!)

Image Fusion & ReconstructionImage Fusion & Reconstruction• Single photo:Single photo: forces narrow tradeoffs: forces narrow tradeoffs:

– Focus, Exposure, aperture, time, sensitivity, noise,Focus, Exposure, aperture, time, sensitivity, noise,

– Usual result: Incomplete visual appearance.Usual result: Incomplete visual appearance.

Multiple photosMultiple photos, assorted settings , assorted settings for Optics, Sensor, Lighting, Processingfor Optics, Sensor, Lighting, Processing

• Fusion:Fusion: ‘Merge the best parts’‘Merge the best parts’

• Reconstruction:Reconstruction:Detect photo changes; Detect photo changes; compute scene invariantscompute scene invariants

The Media Lab Camera Culture

Epsilon Photography

Capture multiple photos, each with slightly different camera parameters.

• Exposure settings• Spectrum/color settings• Focus settings• Camera position• Scene illumination

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

NEARNEAR

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FARFAR

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

FUSION: Best-Focus DistanceFUSION: Best-Focus Distance

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

Source images

‘Graph Cuts’ Solution

FUSION

Agrawala et al., Digital PhotomontageSIGGRAPH 2004

What else can we extend? What else can we extend? Film-Like Camera Parameters: Film-Like Camera Parameters: • Field of View: image stitching for panoramasField of View: image stitching for panoramas• Dynamic Range: Dynamic Range: Radiance MapsRadiance Maps• Frame Rate: Interleaved VideoFrame Rate: Interleaved Video• Resolution: ‘Super-resolution’ methodsResolution: ‘Super-resolution’ methods

Visual Appearance & Content:Visual Appearance & Content:• Tone Map:Tone Map: Detail in every shadow and highlight Detail in every shadow and highlight• Color2grey:Color2grey: Keep Keep allall color changes in grayscale color changes in grayscale • Temporal Continuity: Space-time fusionTemporal Continuity: Space-time fusion• Viewpoint Constraints: Viewpoint Constraints:

Multiple COP images Multiple COP images and more…and more…

The Media Lab Camera Culture

Epsilon Photography

Capture multiple photos, each with slightly different camera parameters.

• Exposure settings• Spectrum/color settings• Focus settings• Camera position• Scene illumination

The Media Lab Camera Culture

Project Ideas

Marc Levoy

The CityBlock Project

Precursor to Google Streetview Maps

What is ‘interesting’ here? Social voting in the real world = ‘popular’

Vein Viewer Vein Viewer (Luminetx)(Luminetx)

Near-IR camera locates subcutaneous veins and project Near-IR camera locates subcutaneous veins and project their location onto the surface of the skin.their location onto the surface of the skin.

Coaxial IR camera Coaxial IR camera + Projector+ Projector

Focus Adjustment: Sum of Bundles

http://www.mne.psu.edu/psgdl/FSSPhotoalbum/index1.htm

Varying PolarizationVarying PolarizationYoav Y. Schechner, Nir Karpel 2005Yoav Y. Schechner, Nir Karpel 2005

Best polarization state

Worst polarization state

Best polarization state

Recovered image

[Left] The raw images taken through a polarizer. [Right] White-balanced results: The recovered image is much clearer, especially at distant objects, than the raw image

Varying PolarizationVarying Polarization• Schechner, Narasimhan, NayarSchechner, Narasimhan, Nayar

• Instant dehazing Instant dehazing of images using of images using polarizationpolarization

Spatial Augmented Reality | Raskar 2011

Pamplona , Mohan, Oliveira, Raskar, Siggraph 2010

NETRA: Near Eye Tool for Refractive Assessment

EyeNetra.com

90

Confocal Microscopy Examples

Slides by Doug Lanman

Beyond Visible SpectrumBeyond Visible Spectrum

CedipRedShift

MIT Media LabMIT Media Lab

Camera CultureCamera Culture

Ramesh RaskarRamesh Raskar

MIT Media LabMIT Media Labhttp:// CameraCulture . info/http:// CameraCulture . info/

Computational Camera & Computational Camera & Photography:Photography:

http://www.flickr.com/photos/pgoyette/107849943/in/photostream/

  Scheimpflug Scheimpflug principleprinciple

Ramesh Raskar, Computational Illumination

Computational Illumination

Edgerton 1930’sEdgerton 1930’s

Multi-flash Sequential Photography

Stroboscope(Electronic Flash)

Shutter Open

Flash Time

Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk

Mitsubishi Electric Research Labs (MERL), Cambridge, MAU of California at Santa Barbara

U of North Carolina at Chapel Hill

Non-photorealistic Camera: Non-photorealistic Camera: Depth Edge Detection Depth Edge Detection andand Stylized Stylized

Rendering Rendering usingusing Multi-Flash ImagingMulti-Flash Imaging

Depth Edges

Our MethodCanny

Flash MattingFlash Matting

Flash Matting, Jian Sun, Yin Li, Sing Bing Kang, and Heung-Yeung Shum, Siggraph 2006

DARPA Grand ChallengeDARPA Grand Challenge

The Media Lab Camera Culture

Epsilon Photography

Capture multiple photos, each with slightly different camera parameters.

• Exposure settings• Spectrum/color settings• Focus settings• Camera position• Scene illumination

The Media Lab Camera Culture

Lens Sensor

Camera

Static Scene

Image Destabilization[Mohan, Lanman et al. 2009]

The Media Lab Camera Culture

Static Scene

Lens Motion Sensor Motion

Camera

Image Destabilization[Mohan, Lanman et al. 2009]

MIT Media Lab Camera Culture

Our Prototype

MIT Media Lab Camera Culture

Adjusting the Focus Plane

all-in-focus pinhole image

MIT Media Lab Camera Culture

Defocus Exaggeration

destabilization simulates a reduced f-number

The Media Lab Camera Culture

Capturing Gigapixel Images[Kopf et al, 2007]

3,600,000,000 PixelsCreated from about 800 8 MegaPixel Images

The Media Lab Camera Culture

Capturing Gigapixel Images[Kopf et al, 2007]

Color Original Grayscale

New Method

Color2Gray: Color2Gray: Salience-Preserving Salience-Preserving

Color RemovalColor RemovalSIGGRAPH 2005

Gooch, Olsen, Tumblin, Gooch