Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi...

Northeastern University, Fall 2005Northeastern University, Fall 2005 CSG242: Computational PhotographyCSG242: Computational Photography

Ramesh RaskarMitsubishi Electric Research Labs

Northeastern UniversityOct 19th, 2005

Course WebPage :

http://www.merl.com/people/raskar/photo/course/

Plan for TodayPlan for Today

• ““The Eye As a Camera” Michael SandbergThe Eye As a Camera” Michael Sandberg

• Computational Computational IlluminationIllumination

• Second Programming AssignmentSecond Programming Assignment

• Mid TermMid Term– Oct 26Oct 26thth

• Project Proposals DueProject Proposals Due– November 2November 2ndnd

• Paper reading Paper reading • 2 per student, 15 mins each, Reading list on the web2 per student, 15 mins each, Reading list on the web• Starts Nov 2Starts Nov 2ndnd

CreditsCredits• Assignments:Assignments:• Five project-oriented assignmentsFive project-oriented assignments• Requires programming in MatlabRequires programming in Matlab• 8 points each (Last assignment format is flexible)8 points each (Last assignment format is flexible)

• Mid-term ExamMid-term Exam• 20 points20 points•• Paper reading (two papers per student, 15 min presentation, 5pts each)Paper reading (two papers per student, 15 min presentation, 5pts each)• 10 points (Was Term Paper, 15 points)10 points (Was Term Paper, 15 points)

• Final ProjectFinal Project• Individual or in a group of 2Individual or in a group of 2• 20 points20 points

• Discretionary creditDiscretionary credit• 10 points (Was 5)10 points (Was 5)

Tentative ScheduleTentative Schedule

• Oct 26: Midterm examOct 26: Midterm exam

• Nov 2Nov 2ndnd Project Proposals Due Project Proposals Due

• Nov 9Nov 9thth Class ? Likely on 10 Class ? Likely on 10thth

• Nov 16Nov 16thth Class ? Class ?

• Nov 23Nov 23rdrd -> Likely on Nov 22 -> Likely on Nov 22ndnd

• Nov 30Nov 30thth

• Dec 7Dec 7thth

• Dec 15Dec 15thth (Exam week) Projects (Exam week) Projects

Mid-TermMid-Term

• Oct 26Oct 26thth at 6pm, at 6pm, • Duration: 90 minutesDuration: 90 minutes• Questions: Think, Explore, SolveQuestions: Think, Explore, Solve

– No need to remember all the formulas in detailNo need to remember all the formulas in detail– More concepts than math problemsMore concepts than math problems– Drawing diagrams to explain conceptsDrawing diagrams to explain concepts

• 20 points20 points• TopicsTopics

• All material covered till Oct 19All material covered till Oct 19thth

• Slides, assignments and in-class discussionsSlides, assignments and in-class discussions• Basics, Dynamic Range, Focus, IlluminationBasics, Dynamic Range, Focus, Illumination

FocusFocus

Computational Illumination

Synthetic LightingSynthetic LightingPaul Haeberli, Jan 1992Paul Haeberli, Jan 1992

Computational Computational PhotographyPhotography

Novel Illumination

Novel Cameras

Scene: 8D Ray Modulator

Display

GeneralizedSensor

Generalized Optics

Processing

Recreate 4D Lightfield

Light Sources

Photography Artifacts: Photography Artifacts: Flash HotspotFlash Hotspot

Ambient Flash

Flash Hotspot

Underexposed Reflections

Ambient Flash

Reflections due to FlashReflections due to Flash

Flash Brightness Falloff with DistanceFlash Brightness Falloff with Distance

Distant people underexposed

Combining Flash/No-flash Images Combining Flash/No-flash Images forfor High Dynamic Range (HDR) Imaging High Dynamic Range (HDR) Imaging

Need Both Ambient and Flash!! FlashAmbient

HDR Scene:

Underexposed

Well-lit in Flash

Well-lit in Ambient

Exposure Time

1/100 1/20 1/5 1 41/250

Conventional Exposure HDR:Conventional Exposure HDR:Varying Exposure TimeVarying Exposure Time

Flash HDR:Flash HDR:Varying Flash BrightnessVarying Flash Brightness

Scene distance dependence

Exposure Time

Flash-Exposure Flash-Exposure SamplingSampling

Flash-Exposure Flash-Exposure HDR:HDR:

Varying bothVarying both

Varying Exposure time Varying Flash brightness Varying both

Capturing HDR Image

Do We Need All Images ?Do We Need All Images ?

Regular 2D Sampling24 Pictures

Adaptive Sampling5 pictures

Next Best Picture ? Next Best Picture ?

Exposure Time

• Based on all previous picturesBased on all previous pictures• Maximize well-lit pixels over the imageMaximize well-lit pixels over the image• Exclude pixels already captured as well-exposedExclude pixels already captured as well-exposed

HDR Image using N images

HDR Image using N+1 images

Underexposed Still UnderexposedWell-exposedExposure Time

N+1th picture ?

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera

Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk

Mitsubishi Electric Research Labs (MERL), Cambridge, MAU of California at Santa Barbara

U of North Carolina at Chapel Hill

Non-photorealistic Camera: Non-photorealistic Camera: Depth Edge Detection Depth Edge Detection andand Stylized Stylized

Rendering Rendering usingusing Multi-Flash ImagingMulti-Flash Imaging

Car Manuals

What are the problems with ‘real’ photo in conveying information ?

Why do we hire artists to draw what can be photographed ?

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Shadows

Clutter

Many Colors

Highlight Edges

Mark moving parts

Basic colors

A New ProblemA New Problem

Why Non-photorealistic (NPR) Why Non-photorealistic (NPR) Images ?Images ?

• Easy to Understand• Easy to Display• Require not-so-rich (3D) data

Can we directly capture using a camera ?– Quick comprehensible images for the masses– Tools for the artists

Depth Edge Camera

Depth Discontinuities

Internal and externalShape boundaries, Occluding contour, Silhouettes

Depth Edges

Sigma = 9 Sigma = 5

Sigma = 1

Canny Intensity Edge Detection

Our method captures shape edges

Our MethodCanny

Our Method

Photo Result

Our Method

Imaging Geometry

Shadow lies along epipolar ray

Shadow lies along epipolar ray,

Epipole and Shadow are on opposite sides of the edge

Imaging Geometry

Shadow lies along epipolar ray,

Shadow and epipole are on opposite sides of the edge

Imaging Geometry

Depth Edge Camera

Light epipolar rays are horizontal or vertical

Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Input U{depth edges}

Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Negative transition along epipolar ray is depth edge

Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Negative transition along epipolar ray is depth edge

Plot U{depth edges}

% Max composite maximg = max( left, right, top, bottom);

% Normalize by computing ratio imagesr1 = left./ maximg; r2 = top ./ maximg;r3 = right ./ maximg; r4 = bottom ./ maximg;

% Compute confidence mapv = fspecial( 'sobel' ); h = v';d1 = imfilter( r1, v ); d3 = imfilter( r3, v ); % vertical sobeld2 = imfilter( r2, h ); d4 = imfilter( r4, h ); % horizontal sobel

%Keep only negative transitions silhouette1 = d1 .* (d1>0); silhouette2 = abs( d2 .* (d2<0) );silhouette3 = abs( d3 .* (d3<0) );silhouette4 = d4 .* (d4>0);

%Pick max confidence in eachconfidence = max(silhouette1, silhouette2, silhouette3, silhouette4);imwrite( confidence, 'confidence.bmp');

No magicparameters

Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi...

Documents

Raskar Npar04final

Raskar COSI invited talk Oct 2009

Compressive DIsplays: SID Keynote by Ramesh Raskar

Raskar UIST Keynote 2015 November

Raskar TEDMED 2013

Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005 Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric

Raskar Pocket Proj Fid2005presented

Raskar Mar09 Nesosa

Bimber & Raskar Siggraph 2005 Spatial Augmented Reality Spatial Augmented Reality Oliver Bimber University of Weimar Ramesh Raskar Mitsubishi Electric

Raskar Sig05 Display Panel July05

Raskar 2012, Idea Hexagon

Raskar ideahexagonapr2010

Raskar Emtech2010 Mar Final

Raskar Graphics Interface May05 Web

Raskar Coded Opto Charlotte

Raskar Thesis Guidance 2011

Raskar Banff

Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/.../AllSlidesraskarCodedPhotographyACCV2007.pdf · Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform,

Camera Culture - exponential.singularityu.org · Ramesh Raskar Innovating for Billions Ramesh Raskar Associate Professor MIT Media Lab . Sanchez, Heshmat, Reza, Romberg, Raskar 2016

Raskar Ilp Oct08 Web