157
Compositing & Blending Tamara Berg Computational Photography Many slides from: Alexei Efros & Derek Hoiem

Compositing & Blending

  • Upload
    leque

  • View
    239

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Compositing & Blending

Compositing & Blending

Tamara Berg Computational Photography

Many slides from: Alexei Efros & Derek Hoiem

Page 2: Compositing & Blending

Image Compositing & Blending

© NASA

Page 3: Compositing & Blending

This Class How do I put an object from one image into another?

Slide from: Derek Hoiem

Page 4: Compositing & Blending

Image Compositing

Page 5: Compositing & Blending

Recent news

Original

“Enhanced” Version

http://www.guardian.co.uk/world/2010/sep/16/mubarak-doctored-red-carpet-picture

Slide from: Derek Hoiem

Page 6: Compositing & Blending

Compositing Procedure 1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)

Composite by David Dewey

2. Blend them into the composite (in the right order)

Page 7: Compositing & Blending

Three methods

1.  Cut and paste

2.  Laplacian pyramid blending

3.  Poisson blending

Slide from: Derek Hoiem

Page 8: Compositing & Blending

Method 1: Cut and Paste

Slide from: Derek Hoiem

Page 9: Compositing & Blending

Method 1: Cut and Paste Method: •  Segment using intelligent scissors •  Paste foreground pixels onto target region

Slide from: Derek Hoiem

Page 10: Compositing & Blending

Method 1: Cut and Paste

Problems: •  Small segmentation errors noticeable •  Pixels are too blocky •  Won’t work for semi-transparent materials

Slide from: Derek Hoiem

Page 11: Compositing & Blending

Feathering Near object boundary pixel values come partly from foreground and partly from background

Slide from: Derek Hoiem

Page 12: Compositing & Blending

Method 1: Cut and Paste (with feathering)

Slide from: Derek Hoiem

Page 13: Compositing & Blending

Alpha compositing

+

Output = foreground*mask + background*(1-mask) Slide from: Derek Hoiem

Page 14: Compositing & Blending

Alpha compositing with feathering

Output = foreground*mask + background*(1-mask) Slide from: Derek Hoiem

Page 15: Compositing & Blending

Proper blending is key

Page 16: Compositing & Blending

Proper blending is key

Page 17: Compositing & Blending

Alpha Blending / Feathering

0 1

0 1

+

= Iblend = αIleft + (1-α)Iright

Page 18: Compositing & Blending

Setting alpha: simple averaging

Alpha = .5 in overlap region

Page 19: Compositing & Blending

Setting alpha: center seam

Alpha = logical(dtrans1>dtrans2)

Distance Transform bwdist!

Page 20: Compositing & Blending

Setting alpha: blurred seam

Alpha = blurred

Distance transform

Page 21: Compositing & Blending

Setting alpha: center weighting

Alpha = dtrans1 / (dtrans1+dtrans2)

Distance transform

Ghost!

Page 22: Compositing & Blending

Alpha Blending / Feathering

0 1

0 1

+

= Iblend = αIleft + (1-α)Iright

Page 23: Compositing & Blending

Affect of Window Size

0

1 left

right 0

1

Large windows -> lots of ghosting

alpha

Page 24: Compositing & Blending

Affect of Window Size

0

1

0

1

Small window –> seams are too obvious

Page 25: Compositing & Blending

Good Window Size

0

1

“Optimal” Window: smooth but not ghosted

Medium window -> just right!

Page 26: Compositing & Blending

What is the Optimal Window? To avoid seams

•  window = size of largest prominent feature

Page 27: Compositing & Blending

What is the Optimal Window? To avoid seams

•  window = size of largest prominent feature

To avoid ghosting •  window <= 2*size of smallest prominent feature

Page 28: Compositing & Blending

What is the Optimal Window? To avoid seams

•  window = size of largest prominent feature

To avoid ghosting •  window <= 2*size of smallest prominent feature

Natural to cast this in the Fourier domain

FFT

Page 29: Compositing & Blending

Fourier domain blending

FFT

Page 30: Compositing & Blending

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright)

FFT

Page 31: Compositing & Blending

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + …

FFT

Page 32: Compositing & Blending

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + … •  Feather corresponding octaves Fleft

i with Frighti

FFT

Page 33: Compositing & Blending

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + … •  Feather corresponding octaves Fleft

i with Frighti

•  Sum feathered octave images in frequency domain

FFT

Page 34: Compositing & Blending

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + … •  Feather corresponding octaves Fleft

i with Frighti

•  Sum feathered octave images in frequency domain •  Compute IFFT to get final blended result.

FFT

Page 35: Compositing & Blending

Fourier domain blending

Idea (Burt and Adelson) •  Compute Fleft = FFT(Ileft), Fright = FFT(Iright) •  Decompose Fourier image into octaves (bands)

–  Fleft = Fleft1 + Fleft

2 + … •  Feather corresponding octaves Fleft

i with Frighti

•  Sum feathered octave images in frequency domain •  Compute IFFT to get final blended result.

FFT

Actually better implemented in the spatial domain

Page 36: Compositing & Blending

Image Pyramids

Known as a Gaussian Pyramid [Burt and Adelson, 1983] •  In computer graphics, a mip map [Williams, 1983]

- Reminder

Page 37: Compositing & Blending

A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose

Figure from David Forsyth

Gaussian Pyramid

Page 38: Compositing & Blending

Laplacian Pyramid

Lowpass Images

Page 39: Compositing & Blending

Laplacian Pyramid

Bandpass Images

Lowpass Images

Page 40: Compositing & Blending

Laplacian Pyramid

Lowpass Images

Page 41: Compositing & Blending

Laplacian Pyramid

Lowpass Images

Page 42: Compositing & Blending

Laplacian Pyramid

Lowpass Images

Bandpass Images

Page 43: Compositing & Blending

Pyramid Blending

0

1

0

1

0

1

Left pyramid Right pyramid blend

•  At low frequencies, blend slowly •  At high frequencies, blend quickly

Page 44: Compositing & Blending

Method 2: Laplacian Pyramid Blending

Implementation: 1. Build Laplacian pyramids for each image 2. Build a Gaussian pyramid of region mask 3. Blend each level of pyramid using region mask

from the same level

4. Collapse the pyramid to get the final blended image

)1(2112iiiii RLRLL −⋅+⋅=

Burt and Adelson 1983

Region mask at level i of Gaussian pyramid

Image 1 at level i of Laplacian pyramid

Page 45: Compositing & Blending

laplacian level

4

laplacian level

2

laplacian level

0

left pyramid right pyramid blended pyramid

Page 46: Compositing & Blending

Pyramid Blending

Page 47: Compositing & Blending

Blending Regions

Page 48: Compositing & Blending

Horror Photo

© david dmartin (Boston College)

Page 49: Compositing & Blending

Season Blending (St. Petersburg)

Page 50: Compositing & Blending

Season Blending (St. Petersburg)

Page 51: Compositing & Blending

Simplification: Two-band Blending Brown & Lowe, 2003

•  Only use two bands: high freq. and low freq. •  Blends low freq. smoothly •  Blend high freq. with no smoothing: use binary alpha

Page 52: Compositing & Blending

Low frequency (λ > 2 pixels)

High frequency (λ < 2 pixels)

2-band Blending

Page 53: Compositing & Blending

Linear Blending

Page 54: Compositing & Blending

2-band Blending

Page 55: Compositing & Blending

Practice Quiz Poisson Image Editing: 1) “Solving the Poisson equation computes the function whose gradient is the closest, in the L2 norm, to some prescribed vector field under given boundary conditions” In blending: What does function refer to? What is an example prescribed vector field?

2) How might you implement texture flattening using Poisson techniques? Photomontage: 3) What are the 2 main techniques used in this paper?

4) What is an image objective? Name 3 possible objectives. Seam Carving: 5) What is the definition of a seam?

6) What algorithm would you use to efficiently find the optimal seam for removal in an image? Describe generally how to implement this.

Page 56: Compositing & Blending

Break

Page 57: Compositing & Blending

Related idea: Poisson Blending A good blend should preserve gradients of source region without changing the background

Perez et al. 2003

Page 58: Compositing & Blending

Related idea: Poisson Blending A good blend should preserve gradients of source region without changing the background

Perez et al. 2003

HW 2!

Page 59: Compositing & Blending

What’s with Poisson?

Solving Poisson – computes the function whose gradient is the closest, in the L2 norm, to some prescribed vector field under given boundary conditions

What’s that mean?

Given a guidance field (e.g. gradient field from source image, or from a composite image)

Calculate the image that best fits this vector field

In general no image exists whose gradient exactly matches input. Best-fit image in least squares sense can be calculated by solving Poisson equation.

Perez et al. 2003

Page 60: Compositing & Blending
Page 61: Compositing & Blending
Page 62: Compositing & Blending
Page 63: Compositing & Blending

HW2: Gradient Domain Editing

General concept: Solve for pixels of new image that satisfy constraints on the gradient and the intensity •  Constraints can be from one image (for filtering) or more (for

blending)

Page 64: Compositing & Blending

HW2: Poisson Blending

A good blend should preserve gradients of source region without changing the background

Treat pixels as variables to be solved

•  Minimize squared difference between gradients of foreground region and gradients of target region

•  Keep background pixels constant

Perez et al. 2003

Page 65: Compositing & Blending

Example

Source: Evan Wallace

Gradient Visualization

Page 66: Compositing & Blending

Source: Evan Wallace

+ Specify object region

Page 67: Compositing & Blending

Other results

Perez et al. 2003

Page 68: Compositing & Blending

What do we lose?

Perez et al. 2003

Foreground color changes Background pixels in target region are replaced

Page 69: Compositing & Blending

Blending with Mixed Gradients Use foreground or background gradient with

larger magnitude as the guiding gradient

Perez et al. 2003

Page 70: Compositing & Blending

Concealment

Page 71: Compositing & Blending

Transparent Objects

Page 72: Compositing & Blending

NPR

Page 73: Compositing & Blending

Things to remember

Three ways to blend/composite 1.  Alpha compositing

–  Need nice cut (intelligent scissors) –  Should feather

2.  Laplacian pyramid blending –  Smooth blending at low frequencies, sharp at high

frequencies –  Usually used for stitching

3.  Gradient domain editing –  Also called Poisson Editing –  Explicit control over what to preserve –  Changes foreground color (for better or worse) –  Applicable for many things besides blending

Page 74: Compositing & Blending

Don’t blend, CUT!

So far we only tried to blend between two images. What about finding an optimal seam?

Moving objects become ghosts

Page 75: Compositing & Blending

Davis, 1998 Segment the mosaic

•  Single source image per segment •  Pick a good seam (cut) so that when you merge the

segments is looks ok.

- Details about this next week!

Page 76: Compositing & Blending

Kwatra et al, 2003

Page 77: Compositing & Blending

Putting it all together Compositing images

•  Have a clever blending function –  Feathering –  Center-weighted –  blend different frequencies differently

•  Choose the right pixels from each image –  Graph-cuts

Page 78: Compositing & Blending

Interactive Digital Photomontage

Aseem  Agarwala,  Mira  Dontcheva,  Maneesh  Agrawala,  Steven  Drucker,  Alex  Colburn,  Brian  Curless,  David  Salesin,  Michael  Cohen  

Page 79: Compositing & Blending
Page 80: Compositing & Blending
Page 81: Compositing & Blending
Page 82: Compositing & Blending

Challenges Find good seams between parts of images so

they can be joined with few visible artifacts Blend along seams to reduce or remove any

artifacts remaining after joining

Page 83: Compositing & Blending
Page 84: Compositing & Blending

Seam Objectives Measures suitability of a seam between two image regions

e.g. •  Match colors across seams •  Match colors and color gradients across

seams •  Match colors across seams, but prefer seams

that lie along edges

Page 85: Compositing & Blending
Page 86: Compositing & Blending

User Selection

Page 87: Compositing & Blending

Result of Pasting

Page 88: Compositing & Blending

Gradient Smoothing

Remove any remaining artifacts after merging source image parts

View input images as sources of gradient information rather than color.

Using graph cut labeling, form composite whose gradients best match source gradient vectors.

Page 89: Compositing & Blending

Final Result after smoothing

Page 90: Compositing & Blending

Extended Depth of Field

Input  Images  

Page 91: Compositing & Blending

Source  Map  

Page 92: Compositing & Blending

Extended Depth of Field

Page 93: Compositing & Blending

Relighting

Input  Images  

Page 94: Compositing & Blending

Source  Map  

Page 95: Compositing & Blending

Relighted Result

Page 96: Compositing & Blending

Visualizing Movement

Page 97: Compositing & Blending

Visualizing Movement

Page 98: Compositing & Blending

Selective Composites

Page 99: Compositing & Blending

Selective Composites

Page 100: Compositing & Blending

Selective Composites

Page 101: Compositing & Blending

Selective Composites

Page 102: Compositing & Blending

Background    reconstruc?on  

Page 103: Compositing & Blending

Background  reconstruc?on  

Page 104: Compositing & Blending

Religh?ng  

Page 105: Compositing & Blending

Composi?ng  and  Obstruc?on  removal  

Page 106: Compositing & Blending

Fictional Composites

Page 107: Compositing & Blending
Page 108: Compositing & Blending

Photo Clip Art

Jean-François Lalonde, Derek Hoeim, Alexei A. Efros, Carsten Rother, John Winn and Antonio Criminisi. Photo Clip Art, ACM Transactions on Graphics (SIGGRAPH 2007),

Page 109: Compositing & Blending

Photo Clip Art

Page 110: Compositing & Blending
Page 111: Compositing & Blending
Page 112: Compositing & Blending
Page 113: Compositing & Blending
Page 114: Compositing & Blending

Photo Clip Art

Page 115: Compositing & Blending
Page 116: Compositing & Blending
Page 117: Compositing & Blending
Page 118: Compositing & Blending
Page 119: Compositing & Blending
Page 120: Compositing & Blending
Page 121: Compositing & Blending
Page 122: Compositing & Blending
Page 123: Compositing & Blending

Street accident

Page 124: Compositing & Blending

Bridge

Page 125: Compositing & Blending

Painting

Page 126: Compositing & Blending

Alley

Page 127: Compositing & Blending

Failure cases

Shadow transfer

Porous objects

Page 128: Compositing & Blending

Failure cases

Best matching objects

Page 129: Compositing & Blending

Inverse of previous topic Removing content (as opposed to adding it)

Next…

Page 130: Compositing & Blending

Seam Carving for Content-Aware Image Resizing Shai Avidan Ariel Shamir

Slides  from:  Kshitij  Padalkar  

Page 131: Compositing & Blending

In  One  Line…  

Method  for  Content-­‐Aware  Image  Resizing    

Page 132: Compositing & Blending

Need  

Page 133: Compositing & Blending

Different  Displays-­‐Different  Resolutions-­‐(Different  Images??)  

Page 134: Compositing & Blending

HTML  allows  automatic  layout  of  the  text,  but  not  the  images  

Page 135: Compositing & Blending

Solu?ons  and  Other  Work  Crop  and  Scale  Saliency  Segmentation    

Page 136: Compositing & Blending

Defini?on…  

Remove  unnoticeable  pixels.  Energy  Function:  

� Remove Pixels with least energy �  Sort pixels by Energy, remove smallest pixels (optimum) �  Remove Column with smallest total Energy (column) �  Remove Seam with smallest total Energy (seam)

Page 137: Compositing & Blending

Seam  

Vertical  Seam:     �  an 8-connected path of pixels in the image from

top to bottom, containing one, and only one, pixel in each row of the image

Page 138: Compositing & Blending

Seam  

Energy  of  a  Seam  

� Minimum Energy Seam

Page 139: Compositing & Blending

Pixel  Removal  Criteria  Optimal  

Page 140: Compositing & Blending

Pixel  Removal  Criteria  Column  

Page 141: Compositing & Blending

Pixel  Removal  Criteria  Seam  

Page 142: Compositing & Blending

Discrete  Image  Resizing  Aspect  Ratio  Change  

•  Optimal  Seam  Order  

Image  Enlarging  Content  Amplification  Object  Removal    

Page 143: Compositing & Blending

Aspect  Ra?o  Change  Change  the  image  from  (n  X  m)  to  (n  X  m’)  

•  (assume    m’  <  m)  

Remove  m-­‐m’=c  seams  successively  

Seam Carving

Page 144: Compositing & Blending

Aspect  Ra?o  Change  Change  the  image  from  (n  X  m)  to  (n  X  m’)  

•  (assume    m’  <  m)  

Remove  m-­‐m’=c  seams  successively  

Scaling

Page 145: Compositing & Blending

Op?mal  Seam  Ordering  

Change  the  image  from  (n  X  m)  to  (n’ X  m’)  •  (assume    m’  <  m  and  n’  <  n)  

What  to  do?  What  is  the  correct  order  of  seam  carving?  •  Remove  vertical  seams  first?    •  Horizontal  seams  first?    •  Or  alternate  between  the  two?  

Page 146: Compositing & Blending

Op?mal  Seam  Ordering  

Transport  map  •  Matrix  of  size  n  X  m  •  Each  element  T(r,c)  holds  the  minimal  cost  needed  to  obtain  an  

image  of  size  n−r  X m−c    

Page 147: Compositing & Blending

Discrete  Image  Resizing  Aspect  Ratio  Change  

•  Optimal  Seam  Order  

Image  Enlarging  Content  Amplification  Object  Removal    

Page 148: Compositing & Blending

Image  Enlarging  

�  compute  the  optimal  vertical  (horizontal)  seam  ‘s’  on  I  and  duplicate  the  pixels  of  s  by  averaging  them  with  their  left  and  right  neighbors  (top  and  bottom  in  the  horizontal  case)  

Page 149: Compositing & Blending

Image  Enlarging  

� To  reduce  the  width  by  ‘k’,  compute  first  ‘k’  vertical  seams  and  duplicate  them  

� All  seams  duplicated,  50%  enlargement.  For  more  do  it  in  steps.  

Page 150: Compositing & Blending

Discrete  Image  Resizing  Aspect  Ratio  Change  

•  Optimal  Seam  Order  

Image  Enlarging  Content  Amplification  Object  Removal    

Page 151: Compositing & Blending

Content  Amplifica?on  � Scale the image, this will scale everything, content as

well as non-content. � Shrink the scaled-image by seam carving, this will

carve out the non-content part.

Page 152: Compositing & Blending

Discrete  Image  Resizing  Aspect  Ratio  Change  

•  Optimal  Seam  Order  

Image  Enlarging  Content  Amplification  Object  Removal  

Page 153: Compositing & Blending

Object  Removal  

� User marks the target object to be removed. � Then seams are removed from the image until all

marked pixels are gone. � To regain the original size, seam insertion is

employed.

Page 154: Compositing & Blending

Object  Removal  �  Find  the  missing  shoe…  

Page 155: Compositing & Blending

Object  Removal  � Object marking to remove unwanted results

Page 156: Compositing & Blending

Limita?ons  

� Too much content � No space for seam to avoid content

Page 157: Compositing & Blending

Announcements HW2 up later today Readings for next week on web page