MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Amit Agrawal, Ashok Veeraraghavan and Ramesh Raskar

Mitsubishi Electric Research Labs (MERL)MIT Media Lab

Cambridge, MA, USA

Reinterpretable Imager: Towards Variable Post-Capture Space, Angle & Time Resolution

in Photography

Captured Photo

Video from Single-Shot (Temporal Frames)

Captured Photo

Rotating Doll in Focus

Captured 2D Photo

Captured 2D Photo

In-Focus

High Resolution 2D Image

Static Scene Parts

Captured 2D Photo

In-Focus Out of Focus

High Resolution 2D Image

4D Light Field

Static Scene Parts

Captured 2D Photo

In-Focus Out of Focus In-Focus

High Resolution 2D Image

4D Light Field

Video

Static Scene Parts Dynamic Scene Parts

Captured 2D Photo

In-Focus Out of Focus In-Focus

High Resolution 2D Image

4D Light Field

Video

Static Scene Parts Dynamic Scene Parts

1D Parallax + Motion

Out of Focus

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Key Idea

• Resolution tradeoff for Conventional Imaging

– Fixed before capture

• video camera, lightfield camera– Scene independent

• Resolution tradeoff for Reinterpretable Imager– Variable in post-capture– Scene dependent– Different for different parts of the scene/captured photo

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Reinterpretable Imager

• Capture Time-Varying Light Field• Spatial dimensions = 2• Angular dimensions = 2• Temporal dimensions = 1

• We capture 4D subsets in single-shot

• Single Camera Design can act as– Still camera, Video camera, lightfield camera

Optical Design

Dynamic Aperture

Mask

Sensor

Static Mask

Sensor

Static Aperture

Mask

Sensor

Coded Aperture Optical Heterodyning Reinterpretable Imager

Veeraraghavan et al. SIGGRAPH 2007

Veeraraghavan et al. SIGGRAPH 2007

This Paper

Static Mask

Digital Refocusing

Dynamic Aperture

Mask

Sensor

Static Mask

Sensor

Static Aperture

Mask

Sensor

Coded Aperture Optical Heterodyning Reinterpretable Imager

Veeraraghavan et al. SIGGRAPH 2007

Veeraraghavan et al. SIGGRAPH 2007

This Paper

Static Mask

Digital Refocusing

Light Field Capture

Static Mask

Sensor

Static Aperture

Mask

Sensor

Coded Aperture Optical Heterodyning

Veeraraghavan et al. SIGGRAPH 2007

Veeraraghavan et al. SIGGRAPH 2007

This Paper

Dynamic Aperture

Mask

Sensor

Reinterpretable Imager

Static Mask

Digital Refocusing

Light Field Capture

Post-Capture Resolution Control

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Reinterpretable Imager

• Dynamic Aperture Mask

– Pinhole moved across the aperture during exposure time• Single shot video, lightfield, high res image

– Vertical slit moved across the aperture• 1D parallax + motion

• Near-Sensor Mask– Similar to Veeraraghvan et al. SIGGRAPH 2007

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Implementation

Camera Motor Wheel

Aperture Mask on Wheel

Shutter

Near-Sensor Mask

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Related Work

• Hand-held Light Field Camera• Single shot• Micro-lens Ng et al. 2005• Prims+lens Georgiev et al. 2006• Mask at sensor Veeraraghavan et al. 2007

• Light Field camera + Aperture Modulation– Horstmeyer ICCP 09– Polarization, Spectral

• Multiplexing techniques– Assorted Pixels– Illumination multiplexing, Schechner et al. ICCV 2003

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Recovering Full Resolution 2D ImageRecovering Full Resolution 2D Image

Sensor

p

Mask

No Mask: pixel value = I(p)

With Mask: pixel value = β(p)I(p)

In-focus static scene

Mask

For Static In-Focus Scene

Captured 2D Photo

Divide by Calibration Photo

High Resolution Image

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Recovering Full Resolution 2D ImageRecovering Full Resolution 2D Image

• For static in-focus scene

– Inserting Masks == Spatially Varying Image Attenuation

– Compensate using normalization photo β(p)

In Focus Out of Focus

In Focus Out of Focus

Captured Photo Normalization PhotoFull Resolution Image

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Recovering Light Fields for Static SceneRecovering Light Fields for Static Scene

• Capture Light Fields:– Map angular variations to spatial dimension– Angle To Space Mapping

• For static scenes

– Mask close to sensor == captures light field

• Heterodyning, Veeraraghavan et al. SIGGRAPH 2007

– Mask in aperture == no impact, only lose light

For Static Scenes

Captured 2D Photo

Compute 4DLight Field

Digital Refocusing

Sub-Aperture Views == Angular Samples

Captured Photo

(Static Scene)

Reconstructed Sub-Aperture Views (3 by 3 Light Field)

Angle

Angle

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Recovering Video from Single ShotRecovering Video from Single Shot

• In-focus dynamic scene

– Mask in aperture

• maps temporal variations to angular variations

– Mask close to sensor

• captures angular variations as a light field

• Mapping Time to Space Via– Time to Angle + Angle to Space

Static Mask

Moving Pinhole

K

K

Scene patch (t1)

Capturing In-focus Dynamic Scenes

Static Mask

K

K

Scene patch (t2)

Capturing In-focus Dynamic Scenes

Capturing In-focus Dynamic Scenes

Sensor

Static Mask

d

K

KspotScene patch (t3)

For Dynamic In-focus Scene

Captured 2D Photo

Compute 4DLight Field

Sub-Aperture Views == Temporal Frames

Captured Photo

Reconstructed Sub-Aperture Views (3 by 3 Light Field)

Time

Time

Rotating Doll

Reconstructed Sub-Aperture Views (3 by 3 Light Field)

Time

Time

For Rotating Doll

Angle

Angle

For Static Scene Parts

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Recovering 1D Parallax + MotionRecovering 1D Parallax + Motion

• Vertical slit moved across the aperture

– Map angular variations to vertical dimension

– Maps temporal variations to horizontal dimension

• Capture dynamic out-of-focus scene

– However, only 1D out-of-focus blur (bokeh)

For Dynamic Out-of-focus Scene

Captured 2D Photo

Compute 4DLight Field

Sub-Aperture Views == Temporal Frames (Horizontally)

Sub-Aperture Views == Angular Samples (Vertically)

Refocus Using Vertical Views

Captured Photo

Reconstructed Sub-Aperture Views (3 by 3 Light Field)

Time

Angle

Digital Refocusing on Moving Rubik’s Cube

Digital Refocusing on Moving Rubik’s Cube

Digital Refocusing on Moving Rubik’s Cube

Digital Refocusing on Moving Rubik’s Cube

Digital Refocusing on Moving Rubik’s Cube

Digital Refocusing on Moving Rubik’s Cube

Keeping Playing Card in Focus

Keeping Playing Card in Focus

Keeping Playing Card in Focus

Keeping Playing Card in Focus

Keeping Playing Card in Focus

Keeping Playing Card in Focus

Captured Photo

Static Object (in-focus)

Static Objects (Out-of-focus)

Moving Object (in depth)

Rotating Object (in focus)

Reconstructed Sub-Aperture Views (3 by 3 Light Field)

All Static and Dynamic Objects are sharp

(No focus blur, no motion blur)

Angle

Angle

For Static Objects

Time

Angle

For Moving Toy in Middle

Time

Time

For Rotating Toy on Right

Refocused on Static Toy

High Resolution Image

Digital Refocusing on Static Objects

Digital Refocusing on Static Objects

Digital Refocusing on Static Objects

Digital Refocusing on Static Objects

Digital Refocusing on Static Objects

Digital Refocusing on Static Objects

Digital Refocusing on Toy Moving in Depth

Digital Refocusing on Toy Moving in Depth

Digital Refocusing on Toy Moving in Depth

Digital Refocusing on Toy Moving in Depth

Digital Refocusing on Toy Moving in Depth

Digital Refocusing on Toy Moving in Depth

Video frames of Rotating ToyVideo for Rotating Toy in-focus

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Limitations

• Light Loss– To get extra information– Both at aperture and sensor– Micro-lens at sensor (Ng et al.) for lightfield capture

• Temporal Frames– No. of frames = Max angular resolution– Not independent as in a video camera– Large motions cause motion blur– Viewpoint shift– Ghosting artifacts across sub-aperture views

• Does not capture full 5D information– Video light field camera

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Future Work

• LCD’s for modulation– Benefit: Faster modulation– Issues: Contrast, Diffraction

• Using Computer Vision– No high/mid-level processing at present

• Adaptive (Active) Sampling

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

AcknowledgementsAcknowledgements

• Anonymous Reviewers

• MERL – Jay Thornton, Kojima Keisuke, Joseph Katz, John Barnwell,

Brandon Taylor, Clifton Forlines and Yuichi Taguchi

• Mitsubishi Electric, Japan– Haruhisa Okuda & Kazuhiko Sumi

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar

Google: ‘Reinterpretable Imager’

Captured 2D Photo

In-Focus Out of Focus In-Focus Out of Focus

High Resolution 2D Image

4D Light Field Video1D Parallax

+ Motion

Static Scene Parts Dynamic Scene Parts Dynamic Aperture

Mask

Sensor

Reinterpretable Imager

Static Mask