CORNAR: Looking Around Corners using Trillion FPS Imaging

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

MIT Media LabRamesh Raskar

Computational Light Transport:

CORNAR: Looking Around Corners using

Trillion FPS Imaging

http://raskar.info/cornar

What isaround the corner ?

Can you look around the corner ?

Can you look around the corner ?

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

Bit Hacking

Phot

on H

acki

ng

Computer Vision

Optics

Sensors

Signal Processing

Displays

Machine Learning

Computational Light Transport

Computational PhotographyIllumination

Co-designing Optical and Digital Processing

2nd Bounce

Multi-path Analysis

1st Bounce

3rd Bounce

CORNAR: Femto-PhotographyFemtoFlash

Trillion FPS camera

Computational Optics

Serious Sync

•2012: 3D around a corner (NatureComm, Velten, et. al.)•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)•2011: DARPA Young Faculty Award•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)•2008: Indirect depth (Hirsch, Raskar)•2008: Transient Light Transport (Raskar, Davis, March 2008)

With M Bawendi,

MIT Chemistry

Inverting Light Transport

Multiple Scattering Direct/Global

Dual Photography

[Seitz , Kutulakos, Matsushita 2005] [Nayar, Raskar et al 2006]

[Sen et al 2005]

LIDAR

[Atcheson et al 2008][Kutulakos, Steger 2005]

Collision avoidance, robot navigation, …

…, bronchoscopies, …

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

Streak Photo

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Multi-Dimensional Light Transport

5-D Transport

s1

Streak-camera

Occluder

C

3rd bounce

Why Pico-second Resolution?

s2ToF Diff = 0.15 mm

p1

p21st bounce

1cm

Curse of Pythagoras

Streak Photo

L

xz

s

S

Streak-camera

Laser beam

Occluder

CB

Echoes of Light

3rd bounce

R

Trillion FPS

ToF Streak Tube = Inverse of CRT

1D camera: Single scan line stretched vertically in time~2 ps resolution, 480 lines ~= 1 ns

But for small samples in biochemistry

Very accurate sync

Time-Image

Time Profile for a single pixel

R

Third Bounce (First bounce not shown)

Space, 640 pixels

Tim

e, ~

2ns e

ach

row

Time Image of a single point

Forward

Wavefront Propagation

Reconstruction

Non-linear Inversion

Invertibility

Scene Priors

Analysis

Resolution and dimensions

3D shape result from synthetic data

Steady State 4D

Impulse Response, 5D

[Raskar and Davis, 2007]

[Kajiya, 1986] [Seitz.., 2005]

Scene with hidden elements

Raw Time profiles

t

Signal Proc.

Novel light transport models and inference

algorithms

Photo, geometry, reflectance

beyond line of sight

3D Time images

Ultra fast illumination and camera

5D Capture

Time Resolved Multi-path Imaging

Third Bounce (First bounce not shown)

Third Bounce (First bounce not shown)

Third Bounce (First bounce not shown)

Third Bounce (First bounce not shown)

Third Bounce (First bounce not shown)

Capture Setup

Photos from Streak Camera

Hidden Scene

Capture Setup

Photos from Streak Camera

Hidden Scene OverlayReconstruction

Hidden 3D Shape Space-time Photos

Motion beyond line of sight

Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011

BRDF (reflectance) from Single Viewpoint and Time Images

Naik, Zhao, Velten, Raskar, Bala, (SIGGRAPH Asia 2011)

…, cardioscopies, …

Participating Media

Space-time Photos

Trillion Frames Per Second Imaging

http://raskar.info/trillionfps

Each frame = ~2ps = 0.6 mm of Light Travel

Camera Time

Camera Time

raskar.info/trillionfps

World Time

 Lorentz transformation: distances, velocities, ordering

raskar.info/trillionfps

Pixel Ray Wave Photons(ampli+phase)

Steady State TransientImpulse/Step•2012: 3D around corner (Nature Comm, Velten et al)•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)•2011: DARPA Young Faculty Award•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)•2008: Indirect depth (Hirsch, Raskar)•2008: Transient Light Transport (Raskar, Davis, March 2008)

FemtoFlash

Trillion FPS camera

Computational OpticsSerious Sync

• Collaborations Welcome• Dataset Available• Propose Configurations

raskar(at)media.mit.eduhttp://www.media.mit.edu/~raskar/cornar/

Femto-PhotographyLooking Around the Corner

BRDF Detection

Trillion FPS Movies

Space-time Transforms

http://raskar.info