Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005 Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric

Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005

Northeastern University, Fall 2005Northeastern University, Fall 2005 CSG242: Computational PhotographyCSG242: Computational Photography

Ramesh RaskarMitsubishi Electric Research Labs

Northeastern UniversityNov 20, 2005

Course WebPage :

http://www.merl.com/people/raskar/photo/course/


Plan for TodayPlan for Today

• Exam review (Avg 84.5, max 89 min 76)Exam review (Avg 84.5, max 89 min 76)

• Light FieldLight Field

• Assignment 4Assignment 4

• Tools: Gradient Domain and Graph CutsTools: Gradient Domain and Graph Cuts

• Paper reading Paper reading • 2 per student, 15 mins each, Reading list on the web2 per student, 15 mins each, Reading list on the web• Starts Nov 10Starts Nov 10thth

• Course feedbackCourse feedback


Reading Paper PresentationsReading Paper Presentations15 minutes: 10 minute presentation, 5 minutes for discussion15 minutes: 10 minute presentation, 5 minutes for discussionUse Powerpoint slides,Use Powerpoint slides,Bring your own laptop or put the slides on a USB drive, (print the slides to be safe)Bring your own laptop or put the slides on a USB drive, (print the slides to be safe)

Format:Format:MotivationMotivationApproach (New Contributions)Approach (New Contributions)ResultsResultsYour own view of what is useful, what are the limitationsYour own view of what is useful, what are the limitationsYour ideas on improvements to the technique or new applications (atleast 2 new ideas)Your ideas on improvements to the technique or new applications (atleast 2 new ideas)

It is difficult to explain all the technical details in 15 minutes. So focus on the key concepts andIt is difficult to explain all the technical details in 15 minutes. So focus on the key concepts andgive an intuition about what is new here. Ignore second order details in the paper, insteadgive an intuition about what is new here. Ignore second order details in the paper, insteaddescribe them in the context of the results.describe them in the context of the results.

Keep the description of the approach simple, a rule of thumb: no more than 3 equations in your presentation.Keep the description of the approach simple, a rule of thumb: no more than 3 equations in your presentation.

Most authors below have the powerpoint slides on their websites,Most authors below have the powerpoint slides on their websites,so feel free to use those slides and modify them. Be careful, do not simply present all their slides in so feel free to use those slides and modify them. Be careful, do not simply present all their slides in

sequence.sequence.

You should focus on only the key concepts and add your own views.You should focus on only the key concepts and add your own views.

If the slides are not available on the author website, copy paste images from the PDF to create your slides.If the slides are not available on the author website, copy paste images from the PDF to create your slides.Sometimes you can send email to the author, and s/he will send you the slides. Sometimes you can send email to the author, and s/he will send you the slides.


Tentative ScheduleTentative Schedule

• Nov 30Nov 30thth – Project UpdateProject Update– Special lecture: Video Special EffectsSpecial lecture: Video Special Effects

• Dec 2Dec 2nd nd (Friday)(Friday)– Hw 4 due MidniteHw 4 due Midnite

• Dec 7Dec 7thth

– In class exam (instead of Hw 5)In class exam (instead of Hw 5)– Special lecture: Mok3Special lecture: Mok3

• Dec 15Dec 15thth (Exam week) (Exam week) – Final Project PresentationFinal Project Presentation

Department of Computer and Information Science

The Matrix: A Real Revolution

The Matrix: A Real Revolution

Warner Brothers 1999

A Neo-Realism!


Assignment 4: Playing with Epsilon ViewsAssignment 4: Playing with Epsilon ViewsSee course webpage for detailsSee course webpage for details

• Resynthesizing images from epsilon views (rebinning of rays)Resynthesizing images from epsilon views (rebinning of rays)

• http://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdfhttp://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdf

In this assignment, you will use multiple pictures under slightly varying positionIn this assignment, you will use multiple pictures under slightly varying positionto create a large synthetic aperture and multiple-center-of-projection (MCOP) imagesto create a large synthetic aperture and multiple-center-of-projection (MCOP) images

You will createYou will create(i) An image with programmable plane of focus(i) An image with programmable plane of focus(ii) A see-through effect(ii) A see-through effect

------------------------------------------------------------------------------------------------------------------(A) Available set(A) Available sethttp://www.eecis.udel.edu/~yu/Teaching/toyLF.ziphttp://www.eecis.udel.edu/~yu/Teaching/toyLF.zipUse only 16 images along the horizontal translationUse only 16 images along the horizontal translation

(B) Your own data set(B) Your own data setTake atleast 12-16 pictures by translating a camera (push broom)Take atleast 12-16 pictures by translating a camera (push broom)The forground scene is a flat striped paperThe forground scene is a flat striped paperBackground scene is a flat book cover or paintingBackground scene is a flat book cover or paintingChoose objects with vibrant bright saturated colorsChoose objects with vibrant bright saturated colors

Instead of translating the camera, you may find it easier to translate the sceneInstead of translating the camera, you may find it easier to translate the scenePut the digital camera in remote capture time lapse interval mode (5 second interval)Put the digital camera in remote capture time lapse interval mode (5 second interval)

Effect 1: Programmable focusEffect 1: Programmable focusRebin rays to focus on first planeRebin rays to focus on first planeRebin rays to focus on back planeRebin rays to focus on back planeRebin rays to focus on back plane but rejecting first planeRebin rays to focus on back plane but rejecting first plane

Effect 2: MCOP imagesEffect 2: MCOP imagesRebin rays to create a single image with multiple viewsRebin rays to create a single image with multiple views

Useful linksUseful linkshttp://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdfhttp://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdf

http://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdf


Light Fields, Light Fields, Lumigraph, and Image-Lumigraph, and Image-

based Renderingbased Rendering


Pinhole CameraPinhole Camera

Ray Origin -o

Image Plane

•A camera captures a set of rays•A pinhole camera captures a set of rays passing through a common 3D point


Camera Array and Light FieldsCamera Array and Light Fields

• Digital cameras are cheap– Low image quality– No sophisticated aperture (depth

of view) effects

• Build up an array of cameras• It captures an array of images• It captures an array of set of

rays


Why is it useful?Why is it useful?

• If we want to render a new image– We can query each

new ray into the light field ray database

– Interpolate each ray (we will see how)

– No 3D geometry is needed


Key ProblemKey Problem

• How to represent a ray?• What coordinate system to use?• How to represent a line in 3D

space– Two points on the line (3D + 3D =

6D) Problem?– A point and a direction (3D + 2D =

5D)Problem?

– Any better parameterization?


Two Plane ParameterizationTwo Plane Parameterization

• Each ray is parameterized by its two intersection points with two fixed planes.

• For simplicity, assume the two planes are z = 0 (st plane) and z = 1 (uv plane)

• Alternatively, we can view (s, t) as the camera index and (u, v) as the pixel index


Ray RepresentationRay Representation

For the moment let’s consider just a 2D slice of this 4D ray space

Rendering new pictures = Interpolating raysHow to represent rays? 6D? 5D? 4D?


Light Field RenderingLight Field Rendering

• For each desired ray:– Compute intersection with (u,v) and

(s,t) plane– Blend closest ray– What does closest mean?

(s, t)

(u, v)

(s, t)

(u, v)(u1, v1)

(s1, t1)

(u2, v2)

(s2, t2)

tttt

ssss

vvv

uuu

21

21

21

21

,

,


Light Field RenderingLight Field Rendering

• Linear Interpolation

y1

y2

x1 x2

y?

x

12

12

1

)1( yyy

xx

xx

(s, t)

(u, v)

(s, t)

(u, v)(u1, v1)

(s1, t1)

(u2, v2)

(s2, t2)

tttt

ssss

vvv

uuu

21

21

21

21

,

,

• What happens to higher dimension? Bilinear, tri-linear, quadra-linear


Quadralinear InterpolationQuadralinear Interpolation

Serious aliasing artifacts


Ray StructureRay Structure

2D Light 2D Light FieldField

2D EPI2D EPI

t

v

v

tt1

v1

(v1, t1)

v2

t2

(v2, t2)

t3

v3

(v3, t3)

All the rays in a light field passing through a 3D geometry point


Why Aliasing?Why Aliasing?

• We study aliasing in spatial domain• Next class, we will study frequency

domain (take a quick review of Fourier transformation if you can)


Better Reconstruction Better Reconstruction MethodMethod

• Assume some geometric proxy• Dynamically Reparametrized

Light Field


Focal Surface Focal Surface ParameterizationParameterization


Focal Surface Focal Surface Parameterization -2Parameterization -2

• Intersect each new ray with the focal plane

• Back trace the ray into the data camera

• Interpolate the corresponding rays

• But need to do ray-plane intersection

• Can we avoid that?


Using Using

• Relative parameterization is hardware friendly– [s’, t’] corresponds to a particular texture– [u’, v’] corresponds to the texture coordinate– Focal plane can be encoded as the disparity

across the light field

Camera Plane

Image Plane

(s, t)

(u, v)

Focal Plane

(u1, v1)

(s1, t1)

(u2, v2)

(s2, t2)


ResultsResults

Quadralinear Focal plane at the monitor


Aperture FilteringAperture Filtering

• Simple and easy to implement• Cause aliasing (as we will see

later in the frequency analysis)• Can we blend more than 4

neighboring rays? 8? 16? 32? Or more?

• Aperture filtering


Small aperture sizeSmall aperture size


Large aperture sizeLarge aperture size


Using very large size Using very large size apertureaperture


Variable focusVariable focus


Close focal surfaceClose focal surface


Distant focal surfaceDistant focal surface

Marc Levoy

Synthetic aperture photography

Marc Levoy


Marc Levoy


Marc Levoy


Marc Levoy


Marc Levoy


Marc Levoy

Synthetic aperture videography

Marc Levoy

Long-rangesynthetic aperture photography

• width of aperture 6’

• number of cameras 45

• spacing between cameras 5”

• camera’s field of view4.5°

Marc Levoy

The scene

• distance to occluder 110’

• distance to targets 125’

• field of view at target 10’

Marc Levoy

People behind bushes

• close to sunset, so input images were noisy

• noise decreases as sqrt ( number of cameras )

→ camera arrays facilitate low-light imaging

Marc Levoy

Effective depth of field

• 35mm camera lens with equivalent field of view 460mm

• typical depth of field for an f/4 460mm lens 10’– allowing a 1-pixel circle of confusion in a 640 x 480 pixel image

• effective depth of field of our array 1’

125’

15’

6’ 10’

Marc Levoy

Synthetic aperture photographyusing an array of mirrors

• 11-megapixel camera (4064 x 2047 pixels)

• 18 x 12 inch effective aperture, 9 feet to scene

• 22 mirrors, tilted inwards 22 views, each 750 x 500 pixels

Marc Levoy

Marc Levoy


Perspective? Or Not?Perspective? Or Not?

It’s pretty cool, dude!


Multiperspective Camera?

Multiperspective Camera?



Assignment 4: Playing with Epsilon ViewsAssignment 4: Playing with Epsilon Views

• Resynthesizing images from epsilon views (rebinning of rays)Resynthesizing images from epsilon views (rebinning of rays)

• http://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdfhttp://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdf

In this assignment, you will use multiple pictures under slightly varying positionIn this assignment, you will use multiple pictures under slightly varying positionto create a large synthetic aperture and multiple-center-of-projection (MCOP) imagesto create a large synthetic aperture and multiple-center-of-projection (MCOP) images

You will createYou will create(i) An image with programmable plane of focus(i) An image with programmable plane of focus(ii) A see-through effect(ii) A see-through effect

------------------------------------------------------------------------------------------------------------------(A) Available set(A) Available sethttp://www.eecis.udel.edu/~yu/Teaching/toyLF.ziphttp://www.eecis.udel.edu/~yu/Teaching/toyLF.zipUse only 16 images along the horizontal translationUse only 16 images along the horizontal translation

(B) Your own data set(B) Your own data setTake atleast 12-16 pictures by translating a camera (push broom)Take atleast 12-16 pictures by translating a camera (push broom)The forground scene is a flat striped paperThe forground scene is a flat striped paperBackground scene is a flat book cover or paintingBackground scene is a flat book cover or paintingChoose objects with vibrant bright saturated colorsChoose objects with vibrant bright saturated colors

Instead of translating the camera, you may find it easier to translate the sceneInstead of translating the camera, you may find it easier to translate the scenePut the digital camera in remote capture time lapse interval mode (5 second interval)Put the digital camera in remote capture time lapse interval mode (5 second interval)

Effect 1: Programmable focusEffect 1: Programmable focusRebin rays to focus on first planeRebin rays to focus on first planeRebin rays to focus on back planeRebin rays to focus on back planeRebin rays to focus on back plane but rejecting first planeRebin rays to focus on back plane but rejecting first plane

Effect 2: MCOP imagesEffect 2: MCOP imagesRebin rays to create a single image with multiple viewsRebin rays to create a single image with multiple views

Useful linksUseful linkshttp://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdfhttp://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdf

http://groups.csail.mit.edu/graphics/pubs/siggraph2000_drlf.pdf


Computational IlluminationComputational Illumination• Presence or AbsencePresence or Absence

– Flash/No-flashFlash/No-flash• Light positionLight position

– Multi-flash for depth edgesMulti-flash for depth edges– Programmable dome (image re-lighting and matting)Programmable dome (image re-lighting and matting)

• Light color/wavelengthLight color/wavelength

• Spatial ModulationSpatial Modulation– Synthetic Aperture IlluminationSynthetic Aperture Illumination

• Temporal ModulationTemporal Modulation– TV remote, Motion Tracking, Sony ID-cam, RFIGTV remote, Motion Tracking, Sony ID-cam, RFIG

• Natural lighting conditionNatural lighting condition– Day/Night FusionDay/Night Fusion


A Night Time Scene: Objects are Difficult to Understand due to Lack of Context

Dark Bldgs

Reflections on bldgs

Unknown shapes


Enhanced Context :All features from night scene are preserved, but background in clear

‘Well-lit’ Bldgs

Reflections in bldgs windows

Tree, Street shapes


Background is captured from day-time scene using the same fixed camera

Night Image

Day Image

Result: Enhanced Image


Mask is automatically computed from scene contrast


But, Simple Pixel Blending Creates Ugly Artifacts


Pixel Blending

Our Method:Integration of

blended Gradients


Nighttime imageNighttime image

Daytime imageDaytime image

Gradient fieldGradient field

Importance Importance image Wimage W

Fin

al r

esu

ltF

inal

res

ult

Gradient fieldGradient field

Mixed gradient fieldMixed gradient field

GG11 GG11

GG22 GG22

xx YY

xx YY

II11

I2

GG GGxx YY


Reconstruction from Gradient Field

• Problem: minimize errorI’ – G|• Estimate I’ so that

G = I’

• Poisson equation

I’ = div G

• Full multigridsolver

I’I’

GGXX

GGYY


Image ToolsImage Tools

• Gradient domain operations, Gradient domain operations, – Applications in tone mapping, fusion and mattingApplications in tone mapping, fusion and matting

• Graph cuts, Graph cuts, – Applications in segmentation and mosaicingApplications in segmentation and mosaicing

• Bilateral and Trilateral filters, Bilateral and Trilateral filters, – Applications in image enhancementApplications in image enhancement


Intensity Gradient in 1DIntensity Gradient in 1D

I(x)1

105

G(x)1

105

Intensity Gradient

Gradient at x,G(x) = I(x+1)- I(x)

Forward Difference


Reconstruction from GradientsReconstruction from Gradients

I(x)1

105

Intensity

G(x)1

105

Gradient

??

For n intensity values, about n gradients



I(x)1

105

Intensity

G(x)1

105

Gradient

1D Integration

I(x) = I(x-1) + G(x)

Cumulative sum

?


Grad X

Grad Y


Gradient at x,y as Forward Differences Gx(x,y) = I(x+1 , y)- I(x,y)Gy(x,y) = I(x , y+1)- I(x,y)

G(x,y) = (Gx , Gy)


Intensity Gradient Vectors in ImagesIntensity Gradient Vectors in Images

Gradient Vector


Grad X

Grad Y

2D Integration


Given G(x,y) = (Gx , Gy)

How to compute I(x,y) for the image ?

For n 2 image pixels, 2 n 2 gradients !


Grad X

Grad Y

2D Integration


Recovering Original Image


Grad X

Grad Y

Intensity Gradient ManipulationIntensity Gradient Manipulation

New Grad X

New Grad Y

Gradient Processing

Recovering Manipulated Image


Gradient Processing

New Grad X

New Grad Y

2D Integration




Gradient Processing

New Grad X

New Grad Y

2D Integration




Grad X

Grad Y

New Grad X

New Grad Y

2D Integration


Gradient Processing

A Common Pipeline




Euler-Lagrange EquationEuler-Lagrange Equation


Application: Compressing Dynamic RangeApplication: Compressing Dynamic Range

How could you put all thisHow could you put all this

information into oneinformation into one

Image ?Image ?


Attenuate High GradientsAttenuate High Gradients

I(x)1

105

G(x)1

105

Intensity Gradient

I(x)1

105

Intensity

Maintain local detail at the cost of global range

Fattal et al Siggraph 2002


Basic AssumptionsBasic Assumptions

• The eye responds more to local intensity The eye responds more to local intensity differences (ratios) than global illuminationdifferences (ratios) than global illumination

• A HDR image must have some large A HDR image must have some large magnitude gradientsmagnitude gradients

• Fine details consist only of smaller magnitude Fine details consist only of smaller magnitude gradients gradients


Gradient Compression in 1DGradient Compression in 1D


Gradient Domain MethodGradient Domain Method


Basic MethodBasic Method

• Take the log of the luminancesTake the log of the luminances

• Calculate the gradient at each pointCalculate the gradient at each point

• Scale the magnitudes of the gradients with a Scale the magnitudes of the gradients with a progressive scaling function (Large progressive scaling function (Large magnitudes are scaled down more than small magnitudes are scaled down more than small magnitudes)magnitudes)

• Re-integrate the gradients and invert the log Re-integrate the gradients and invert the log to get the final imageto get the final image


Grad X

Grad Y

New Grad X

New Grad Y

2D Integration

Summary: Intensity Gradient ManipulationSummary: Intensity Gradient Manipulation

Gradient Processing


Graph and Images

Credits: Jianbo Shi


Brush strokes Computed labeling


Image objective

0 if red∞ otherwise

0 for any label


Graph Based Image Segmentation

Wij

Wij

i

j

V: graph node

E: edges connection nodes

Wij: Edge weight

Image pixel

Link to neighboring pixels

Pixel similarity

Segmentation = Graph partition


Minimum Cost Cuts in a graph

Cut: Set of edges whose removal makes a graph disconnected

Si,j : Similarity between pixel i and pixel j

Cost of a cut,

A

A


Brush strokes Computed labeling

Graph Cuts for Segmentation and Mosaicing

Cut ~ String on a height field

Modeling Projections

Modeling projection

The coordinate system• We will use the pin-hole model as an approximation

• Put the optical center (Center Of Projection) at the origin

• Put the image plane (Projection Plane) in front of the COP– Why?

• The camera looks down the negative z axis– we need this if we want right-handed-coordinates

–

Slide by Steve Seitz

Modeling projection

Projection equations• Compute intersection with PP of ray from (x,y,z) to COP

• Derived using similar triangles (on board)

• We get the projection by throwing out the last coordinate:


Homogeneous coordinates

Is this a linear transformation?

Trick: add one more coordinate:

homogeneous image coordinates

homogeneous scene coordinates

Converting from homogeneous coordinates

• no—division by z is nonlinear


Perspective ProjectionProjection is a matrix multiply using homogeneous coordinates:

divide by third coordinate

This is known as perspective projection• The matrix is the projection matrix• Can also formulate as a 4x4

divide by fourth coordinateSlide by Steve Seitz

Orthographic Projection

Special case of perspective projection• Distance from the COP to the PP is infinite

• Also called “parallel projection”• What’s the projection matrix?

Image World


Spherical Projection

What if PP is spherical with center at COP?

In spherical coordinates, projection is trivial:

dNote: doesn’t depend on focal length d!

Documents

Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005 Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric