Fast Bilateral Filtering & Applicationspeople.csail.mit.edu/fredo/Classes/Comp_Photo_ENS/slides/05... · Fast Bilateral Filtering & Applications Frédo Durand MIT - EECS Many slides

6.815 Digital and Computational Photography 6.865 Advanced Computational Photography

Fast Bilateral Filtering & Applications

Frédo DurandMIT - EECS

Many slides by Sylvain Paris & Jiawen Chen

Tuesday, October 27, 2009

Demosaicking pset• Don’t forget to convert to double• See how to avoid loops on web page


Recap: HDR imaging• Multiple-exposure HDR capture

– calibrate response curve– combine multiple exposures

• Bilateral tone mapping– decompose luminance into large scale & detail

• use bilateral filter– reduce contrast of large scale only

• preserve detail• preserve colors


Alternative: exposure fusion

• One single step for both multiple-exposure merging & tone mapping• http://research.edm.uhasselt.be/~tmertens/exposure_fusion/

4Tuesday, October 27, 2009

http://research.edm.uhasselt.be/~tmertens/exposure_fusion/

http://research.edm.uhasselt.be/~tmertens/exposure_fusion/

Back to bilateral tone mapping

Detail

Color

Intensity Large scale

Bilateral Filter

Reducecontrast

Detail

Large scale

Color

Preserve!

Input HDR image Output


Bilateral Filter: Weighted Average of Pixels

space rangenormalization

space

range

p

q

• Depends on spatial distanceand intensity difference– Pixels across edges have almost no influence


Review: Gaussian (Bell curve)

• σ (standard deviation) determines width• Can be normalized

– here, to be 1 at 0– or to make area under curve 1 (multiply by )

• Nice smooth way to have high influence around center and then decrease rapidly beyond

Gσ(x) = e−x2

2σ2

1σ√

2π


Brute-force Implementation of Bila.

For each pixel pFor each pixel q

Compute

8 megapixel photo: 64,000,000,000,000 iterations!

V E R Y S L O W !More than 10 minute per image


Idea: Far away pixels are negligible, truncate the Gaussian

– usually truncate at 3σ or 4σ

For each pixel p– For each pixel q such that || p – q || < constant × σs

Better Brute-force Implementation

looking at all pixels looking at neighbors only


Discussion

• Complexity:

• Fast for small kernels: σs ~ 1 or 2 pixels

• BUT: slow for larger kernels

neighborhood area


Questions?


Bilateral Grid: basic motivation• When we smooth, we reduce complexity of image

=> we should be able to do it at a lower resolution

[Paris and Durand 06, Chen et al. 07]

x

y

x

y

Downsampling & Simple blur


• When we smooth, we reduce complexity of image => we should be able to do it at a lower resolution

• However, the bilateral filter preserves sharp edges and a low resolution image does not

• Idea: add a 3rd dimension to the image so that intensity difference are handled well

x

yBilateral Grid

Bilateral Grid: basic motivation[Paris and Durand 06, Chen et al. 07]

y x

intensity


Recall other view• The bilateral filter uses the 3D distance• With the bilateral grid, this becomes a 3D blur


• Idea 2.0: The product of spatial and intensity Gaussian defines a 3D Gaussian in x, y, I

p

space

range (intensity)

Bilateral Grid [Paris and Durand 06, Chen et al. 07]


Fast bilateral filter idea• Represent image in

low-resolution 3D grid

• 3D blur combines space and intensity terms

y x

intensity

sprange


Questions?


Overview• Convert image to bilateral grid

– (x, y) pixel goes to x, y, I(x,y)• Blur the grid

– 3D Gaussian combines 2D x,y term f and I term g• Convert back to 2D image space

x

y

Bilateral Grid

y x

intensit


Bilateral Filter on the Bilateral GridImage scanline

Intensity plot

Bilateral Grid


Bilateral Grid


Intensity plot

Bilateral Grid

Blurred bilateral grid (3D blur) Blurred bilateral grid

Filtered scanline

Query grid with input image


x

Grid creation• Convert image to bilateral grid

– (x, y) pixel goes to x, y, I(x,y)• Note that not all grid cells receive the same # of pixels

– empty cells shown in blue here– store a weight to keep track of #pixel– will give us normalization factor k in bilateral filter

x

y

Bilateral Grid


Bilateral Grid data structure• 3D array indexed by x, y, intensity• Each cell stores

– a value (either RGB or just intensity)– a weight (keeps track of #pixels)

• Resolution depends on application – For bilateral filter, depends on σs and σr

(σ should be ~ the width of a cell)

xTuesday, October 27, 2009

Implementation details• Probably a good idea to have helper functions to

index grid directly from image x, y and I– i.e. do the downsampling with appropriate scale

factors (here σs and σr)

x, y, I →�

x

σs

�,

�y

σs

�,

�I

σr

�

where [] denotes integer truncation


Blurring the grid• Same as in 2D• Each cell replaced by Gaussian-weighted average of

neighbors– if v is the value in grid, the blurred output b is:

y x

intensit

b(x, y, i) =�

x�,y�,i�

Gσf (x− x�)Gσf (y − y�)Gσg (i− i�)v(x, y, i)


Even smarter: separable• Blur one axis at a time• works because our blurring kernel is separable

(defined as product along axes)• e.g. blur along x axis:

• If we have chosen sf = 1 cell width, then the Gaussian is simply [1 4 6 4 1]/16

b(x, y, i) =�

x�

Gσs(x− x�)v(x, y, i)

1 4 6 4 1


•JUSTIFY SEPARABLE


Blurring• Blur BOTH the values and the weights• Recall original bilateral filter formulas

€

J(x)

€

=

€

1k(x)

€

ξ

∑

€

f (x,ξ)

€

g(I(ξ) − I(x))

€

I(ξ)

€

ξ

∑

€

f (x,ξ)

€

g(I(ξ) − I(x))k(x) =


[Tomasi and Manduchi 1998]

• k(x)=

€

J(x)

€

=

€

1k(x)

€

ξ

∑

€

f (x,ξ)

€

g(I(ξ) − I(x))

€

I(ξ)

€

ξ

∑

€

f (x,ξ)

€

g(I(ξ) − I(x))


Slicing: critical step• Read the grid at locations specified by input image

– Output at pixel x, y, is read from grid cell x, y, I(x, y)• Trilinear reconstruction

– because the grid is downsampled

Blurred bilateral grid

Filtered scanline

Query grid with input image


Linear reconstruction• Say we only have values v at integer x• We want to reconstruct at real-valued x’• Linear reconstruction:

x x+1x’

(1-x’+x)v(x, y)+(x’-x)v(x+1, y)]


BiLinear reconstruction• Say we only have values v at integer x & y• We want to reconstruct at real-valued x’, y’• Bilinear reconstruction:

x,y x+1,y

x+1,y+1x,y+1

x’,y’

(1-y’+y)[(1-x’+x)v(x, y)+(x’-x)v(x+1, y)] +(y’-y)[(1-x’+x)v(x, y+1)+(x’-x)v(x+1, y+1)]


Bilinear: order does not matter• Linear along x followed by linear along y

• Linear along y followed by linear along x

• Reduces to the same terms

(1-y’+y)[(1-x’+x)v(x, y)+(x’-x)v(x+1, y)] +(y’-y)[(1-x’+x)v(x, y+1)+(x’-x)v(x+1, y+1)]

(1-x’+x)[(1-y’+y)v(x, y)+(y’-y)v(x, y+1)] +(x’-x)[(1-y’+y)v(x+1, y)+(y’-y)v(x+1, y+1)]



Intensity plot

Bilateral Grid

RECAP



Intensity plot

Bilateral Grid

Blurred bilateral grid: both values & weights Blurred bilateral grid

Filtered scanline

Query grid with input image: output =

blurred value/blurred weight

RECAP


Pseudo code

For each pixel x, y

add I(x,y) to grid cell x/σ, y/σ, I(x,y)/σ add 1 to weight of grid cell x σ, y σ, I(x,y) σBlur values & weights along X axis

Blur values & weights along Y axis

Blur values & weights along I axis

For each pixel x, y

output = value(x/σ, y/σ, I(x,y)/σ) / weight(x/σ,y/σ,I(x,y)/σ)

35

Grid creation

Grid Blur

Slicing


Questions?


input


brute-forceimplementation


bilateral gridvisually similar


PerformanceImage size: 2 MPixels

• Brute force: 10 minutes• CPU Bilateral grid: 1 second

• GPU bilateral grid• 2004 card (NV40): 28 ms (36 Hz)• 2006 card (G80): 9 ms (111 Hz)



Discussion

• Complexity:

• Fast for medium and large kernels– Can be ported on Graphics Processing Units

(graphics cards) [Chen 07]: always very fast

• Can be extended to color images but slower

• Visually similar to brute-force computation

numberof pixels

numberof 3D cells

| R | : number of gray levels


Surprising behavior

• Faster when spatial footprint gets bigger•Because we can downsample more aggressively



More bilateral grid operations


Edge-aware brush

• Classical paint brush• Ignores edges

• Our edge-aware brush• Respects edges

Stroke with classical brush

Stroke with bilateral brush

Input image


Bilateral Grid PaintingImage scanline

space

inte

nsityBilateral

Grid


Bilateral Grid Painting

Input image Bilateral Grid

• When mouse is held down, paint only at intensity level of initial mouse click


Scribble-based Selection• User scribbles to specify selection [Lischinski 06]

• Piecewise-smooth interpolation to get full selection• Respects intensity discontinuities

Input with scribbles Our interpolated selectionInput image


Scribble-based Selection

Hard constraintsin grid

Image scanline

Bilateral Grid

space

inte

nsity


Scribble-based SelectionImage scanline

space

inte

nsity

space

inte

nsitySmooth

interpolation

Interpolated selection

Slice: query grid with input

image

Hard constraintsin grid

More about the use of scribbles & optimization next week


Many Operations and Applications• Local histogram equalization• Interactive tone mapping

• Video abstraction[Winnemoller 06, DeCarlo 02]

• Photographic style transfer [Bae 06]

Model Input Output


Questions?


Discussion• Respects luminance edges• Color bilateral grid would be 5D

• high memory cost• Might not fit on current graphics hardware• Luminance edges are often sufficient

• Grid resolution depends on the operator• E.g., for edge-aware brush:

space sampling rate ~ brush radiusintensity sampling rate ~ edge-awareness

Bilateral brush crosses thin lines

Edge-aware brushTuesday, October 27, 2009

Summary: the Bilateral Grid• 3D representation for 2D data

• Intelligent downsampling

• Many edge-aware operations• Painting, scribble interpolation,

bilateral filter, local histogram equalization

• Real-time for HD video


Refs

• A Gentle Introduction to Bilateral Filtering and its Applications – http://people.csail.mit.edu/sparis/bf_course/

• A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. Sylvain Paris and Frédo Durand. International Journal of Computer Vision (IJCV'09) – http://people.csail.mit.edu/sparis/publi/2009/ijcv/

Paris_09_Fast_Approximation.pdf• Real-time Edge-Aware Image Processing with the Bilateral Grid. Jiawen

Chen, Sylvain Paris, Frédo Durand. In ACM Transactions on Graphics (Proceedings of the ACM SIGGRAPH 2007 conference)– http://groups.csail.mit.edu/graphics/bilagrid/

• Fast Bilateral Filtering for the Display of High-Dynamic-Range Images. Frédo Durand and Julie Dorsey. SIGGRAPH 2002– http://people.csail.mit.edu/fredo/PUBLI/Siggraph2002/


http://people.csail.mit.edu/sparis/publi/2009/ijcv/Paris_09_Fast_Approximation.pdf




http://groups.csail.mit.edu/graphics/bilagrid/

http://groups.csail.mit.edu/graphics/bilagrid/

http://people.csail.mit.edu/fredo/PUBLI/Siggraph2002/

http://people.csail.mit.edu/fredo/PUBLI/Siggraph2002/

Questions ?


Alternative

• Merge exposures & tone map in one single step

• Burt


EXTRA MATERIAL

• Another acceleration technique


Box Kernel

• Bilateral filter with a square box window

• The bilateral filter can be computed only from the list of pixels in a square neighborhood.

[Weiss 06]

[Yarovlasky 85]

box windowrestrict the sum

independent of position q


Box Kernel• Idea: fast histograms of square windows

[Weiss 06]

input:full histogram is known

update:add one line, remove one line

Tracking one window


Box Kernel• Idea: fast histograms of square windows

[Weiss 06]

input:full histograms are known

update:add one line, remove one line,

add two pixels, remove two pixels

Tracking two windows at the same time


Discussion

• Complexity: – always fast

• Only single-channel images

• Exploit vector instructions of CPU

• Visually satisfying results (no artifacts)– 3 passes to remove artifacts due to

box windows (Mach bands)

1 iteration

3 iterations


input


brute-forceimplementation


box kernelvisually different,

yet no artifacts


Documents

Fast Bilateral Filtering & Applicationspeople.csail.mit.edu/fredo/Classes/Comp_Photo_ENS/slides/05... · Fast Bilateral Filtering & Applications Frédo Durand MIT - EECS Many slides