Raskar 6Sight Keynote Talk Nov09

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Camera CultureAssociate Professor, MIT Media Lab

Photography Wishlist

http://raskar.info

Film-like Digital Photography

Fernald, Science [Sept 2006]

Shadow Refractive

Reflective

‘‘film-like’ Photographyfilm-like’ Photography

Lens

Detector

Pixels

Image

Slide by Shree NayarReproduce for the eye

Wish List TodayWish List Today

• ConsumersConsumers– Super-human visionSuper-human vision– Microscope like resolutionMicroscope like resolution– High speedHigh speed– See inside the body (health)See inside the body (health)– Auto-triggerAuto-trigger– Keep only ‘good’ picsKeep only ‘good’ pics– Find ‘relevant’ pics and better Find ‘relevant’ pics and better

archiving/accessarchiving/access

– Put photographer back in photo!Put photographer back in photo!

• CompaniesCompanies– CostCost– ResolutionResolution– Low-light sensitivity, HDRLow-light sensitivity, HDR– Stereo and 3DStereo and 3D– Mecha-free zoom/focusMecha-free zoom/focus– Auto-tagging for sharingAuto-tagging for sharing– RecognitionRecognition

Can you look around a corner ?

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Camera CultureAssociate Professor, MIT Media Lab

Computational Photography

Wish List

http://raskar.info

SynthesisLow Level Mid Level High

LevelHyper

realism

Raw

Angle, spectrum

aware

Non-visual Data, GPS

Metadata

Priors

Comprehensive

8D reflectance field

Digital

Epsilon

Coded

Essence

Computational Photography aims to make progress on

both axis

Camera ArrayHDR, FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

Transient Imaging

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

Phototourism

Resolution

Computational Photography

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Wish #1

Ultimate Post-capture Control

Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Marc Levoy

Zooming into the raw photo

Mask based Light Field CameraMask Sensor

[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan

Captured 2D Photo

MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan

x

y

u

v

121 Sub-aperture Views

MERL Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin

• Samples individual rays

• Predefined spectrum for lenses• Chromatic abberration

• High alignment precision

• For each lenslet, peripheral pixels are wasted

• Negligible Light Loss

• Samples coded combination of rays

• Supports any wavelength

• Reconfigurable f/#, Easier alignment

• No wastage• High resolution image for parts

of scene in focus

• 50 % Light Loss due to mask

Mask

Sensor

Microlens array

Sensor

Plenoptic Camera Heterodyne Camera

Motion Blur in Low Light

Fluttered Shutter CameraFluttered Shutter CameraRaskar, Agrawal, Tumblin Siggraph2006

Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

Motion Blur in Low Light

Ramesh Raskar, Computational Illumination

Compact Programmable

Lights ?

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Wish #1

Ultimate Post-capture Control

•Digital Refocus and Motion blur

•Emulate studio light from compact flash

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Wish #2

Freedom from Form

Convert LCD into a big flat camera?Beyond Multi-touch: 3D Gestures

Large Virtual Camera for 3D Interactive HCI and Video Conferencing

Matthew Hirsch, Henry HoltzmanDoug Lanman, Ramesh Raskar

Siggraph Asia 2009

BiDi Screen

Touch + Hover using Thin, Depth Sensing LCD Sensor

Sensing Depth from Array of Virtual Cameras in

LCD

Lots of glass; Heavy; Bulky; Expensive

Shallow DoF with Simple Lens

Image Destabilization

Lens Sensor

Camera

Static Scene

Image Destabilization

Static Scene

Lens Motion Sensor Motion

Camera

Mohan, Lanman,Hiura, Raskar ICCP 2009

Small Aperture Photo

all-in-focus

Destabilized Small Aperture

focused in the front using destabilization

Adjusting the Focus Plane

focused in the middle using destabilization

Adjusting the Focus Plane

focused in the back using destabilization

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Wish #3

Understand the World

Convert single 2D photo into 3D ?Snavely, Seitz, SzeliskiU of Washington/Microsoft: Photosynth

Exploit Community Photo CollectionsU of Washington/Microsoft: Photosynth

Can you ‘see’ around a corner ?

Femto-Photography: Higher Dimensional Capture

FemtoFlash

UltraFast Detector

Computational OpticsSerious Sync

Bokode

Defocus blur of Bokode

Coding in Angle

Mohan, Woo, Smithwick, Hiura, Raskar [Siggraph 2009]

camera

sensor

circle of confusion circle of information

Bokode(angle)

Quote suggested by Kurt Akeley

Encoding in Angle, not space, time or wavelength

Product labels

Street-view Tagging

MIT media lab camera culture bokode.com mohan, woo, hiura, smithwick, raskar

cell-phone cameraclose to the Bokode

(10,000+ bytes of data)

capturing Bokodes

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Wish #3

Understand the World

•Identify/recognize Materials•3D Awareness

•Interact with information

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Wish #4

Sharing Visual Experience

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Wish #4

Sharing Visual Experience

•LifeLog Auto-summary•Privacy in public and

authentication• Great Photo-frames

6D Photo Frames

One Pixel of a 6D Display = 4D Display

1 2

11

2D 2D 2D

Single Pixel of

6D Frame

6D Photo Frames: Respond to Viewpoint + Ambient Light

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Wish #4

Sharing Visual Experience

•LifeLog Auto-summary•Privacy in public and

authentication •Hyper-real Photo Frames

•Print ‘material’

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Wish #5

Capturing Essence

Essence Photography

• Beyond physical realism• New Visual Art Forms

Yu, McMillan

What are the problems with ‘real’ photo in conveying information ?

Why do we hire artists to draw what can be photographed ?

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Depth Edges with MultiFlashDepth Edges with MultiFlashRaskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004

Depth Discontinuities

Internal and externalShape boundaries, Occluding contour, Silhouettes

Our MethodCanny

Blind CameraBlind Camera

Sascha Pohflepp, U of the Art, Berlin, 2006

Scene Completion Using Millions of PhotographsHays and Efros, Siggraph 2007

Camera Culture

Ramesh Raskar

Wish #5

…………………

Photos of tomorrow: computed not recorded

http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/

Raskar, Camera Culture, MIT Media Lab

Questions• What will a camera look like in 10,20 years?• How will a billion networked and portable cameras change

the social culture? • How will online photo collections transform visual social

computing?• How will movie making/new reporting change?• computational-journalism.com

2nd International Conference on Computational Photography

Papers due November 2,

2009

http://cameraculture.media.mit.edu/iccp10

• Ramesh Raskar and Jack Tumblin

• Book Publishers: A K Peters

• ComputationalPhotography.org

Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info

• Post-capture control• Emulate studio lights with compact flash• Focus and motion blur

• New forms• Flat camera, large LCDs as cameras• Image destabilization for larger aperture

• Understand the world• Real or fake• Place 2D photo into 3D• Look around corner• Bokode: long distance barcode

• Sharing• Lifelogs auto summary• Privacy/Verification• 6D photoframes

• Essence• New visual arts• Multi-flash camera• Delta-camera and Blind-camera

Computational Photography Wish ListSensor