Fast Edge-Preserving/Aware Filters for Image/Video Processing

Yu Huang Sunnyvale, California

yu.huang07@gmail.com

Outline • Gaussian filter as weighted average • Bilateral filter (Note: anisotropic diffusion filter is equivalent)

– Separable bilateral filtering (range and spatial kernels);

• Fast bilateral filter – Acceleration by piecewise linear approximation;

• Splatting (downsample), blurring (FFT) and slicing (upsample); – Bilateral grid (importance sampling); – Gaussian KD-tree; – Pemutohedral Lattice.

• Fast Gauss (Transform) bilateral filtering; – Constant time bilateral filtering O(1).

• Fast median (distributed histogram) filter • Smooth local histogram filter and its acceleration • Accelerated trilateral filter (gradient preserving) • Domain Transform (DT) as a real-time edge preserving filter • Patch Match • Coherent Sensitivity Hashing (CSH) • Propagation-assisted KD-tree • References

Gaussian Filter as Weighted Average • Weight of x depends on distance to x

),( xxf )(xI

output input

)(xJ x

xx

x

Bilateral Filter [Tomasi’98] • Spatial Gaussian f • Gaussian g on the intensity difference • Edge preserving filter

)(xJ ),( xxf ))()(( xIIg x )(xIx)(

1

xk

output input

x

Cross-Bilateral Filter

q

p

output input

p

S

IIIGGW

IBFq

qqp

p

p qp ||||||1

][rs

[Eisemann’04] [Petschnigg’04]

Edge Preserving vs. Edge Blurring

Input Image Edge-Preserving Base Image Edge-Blurring Base Image

Edge-Preserving Enhanced Image Edge-Blurring Enhanced Image

Halo Artifacts

Anisotropic Diffusion Filter

c(p, t) is large when p is not a part of an edge c(p, t) is small when p is a part of an edge

• Diffusion [Perona & Malik 1990]: – Diffusion is smoothing: pixel ;

– Anisotropic kernel preserve edge: strong edge reduces kernel size;

• Equivalent to bilateral filter, but not easy to accelerate it.

I

Separable Bilateral Filtering [Pham’05]

Pros: extremely fast (fixed spatial weight + LUT for tonal weight)

Cons: effective filtering kernel is slightly distorted

Acceleration of Bilateral Filters • Discretize the set of possible I(x) – sparse in intensity; • Piecewise linear approximation;

• Perform linear Gaussian blur (FFT); • Sub-sampling in spatial domain; • Linear interpolation in between.

• k(x) is treated similarly • [Durand, Siggraph 2002]

)(xJ ),( xxf ))()(( xIIg x )(xIx)(

1

xk

higher dimensional functions

Gaussian convolution

division

slicing

w i w

D O W N S A M P L E

U P S A M P L E

“A Fast Approximation of the Bilateral Filter using a Signal Processing Approach”

• 3-D histogram;

• Splatting (spatial tonal space in high dim);

• Blurring (convolution); – 3D FFT.

• Division & Slicing (interpolation).

[Pham, ECCV’06]

Bilateral Grid

Bilateral Grid (importance sampling) •Bilateral grid = 3D array

–x and y correspond to pixel position

–z corresponds to pixel intensity

–Euclidean distance accounts

for edges

• space distance (x, y) and

intensity distance (z)

•Grid can be coarsely sampled; • 70 x 70 x 10 for an 8 MB pixel image

•Bilateral filter = high dim. kernel + … • Gaussian convolut. applied to grid;

• Sampling and normalization;

• Downsample, blur, then upsample.

Standard 2D Image

x

y [Chen, Siggraph’07]

Bilateral Filter on the Bilateral Grid

space

inte

nsi

ty

Image scanline

Filtered scanline

Slice: query grid with input image

Bilateral Grid

Gaussian blur grid values

space

inte

nsi

ty

Bilateral Grid for Color Image

Tiiiiii

i

i

i

bgryxp

ip

iv

iv

pixel ofposition :

pixel of ueoutput val :ˆ

pixel of ueintput val :

Image

High-Dimensional Grid (5d grid)

High Memory Usage Cost

Bilateral Filtering Based on Luminance Bilateral Filtering Based on Color

Gaussian KD-Tree

x

y

r,g,b

Input Image

Gaussian KD Tree

High-Dimensional Space

Resolution Reduction

•Monte-Carlo Sampling •Weighted Importance Sampling

Different Weighting to Leaf Nodes

Inner Node

Leaf Node

[Adams, Siggraph’09]

• Low Computational Complexity

• Low Memory Usage

• Building The Tree • Querying The Tree

• Gaussian KD-tree is also Important Sampling • High-dimension Gaussian filter: sampling s neighborhood

• Important sampling with Gaussian KD-tree: evaluating samples as near as possible s

• High dimension kernel + sampling on leaf nodes • Splatting (downsample points to leaf nodes)

• Blurring (Gaussian blurring on leaf nodes)

• Slicing (upsample from leaf nodes to points)

Difference of Gaussian Kd-Tree, Bilateral grid

Naïve Bilateral Filtering

5-D Bilateral Grid 3-D Bilateral Grid

KD-Tree

• Patch-based (not pixel-based like bilateral filter);

• Averaging with nearby pixels of similar texture;

– Pixel-wised

– Patch-wised

• More discriminative, still blurry in finer details.

• Note: Intel OpenCV implements NLM for denoising.

Nonlocal Means Filter [Buadess’05]

Non-Local Means by Gaussian Kd-Tree

Gaussian KD Tree

Inner Node

Leaf Node

Image Pixel

Different Weighting to Leaf Nodes

High-Dimensional Space

Querying

Permutohedral Lattice for High Dim • A framework for high-dim Gaussian filtering (Gauss KD-tree)

– Embedding the input values at positions in a high-dimensional space (splatting): bary centric coord. interpolation;

– Performing a Gaussian blur in that space (blurring);

– Sampling the space at the original positions (slicing).

• It tessellates the high-dimensional space with uniform simplices (HD tetrahedra);

• Less sparse than Gaussian kd-tree, but less memory consumption than bilateral grid, efficient for a high dim filter.

[Adams, Eurographic’10]

Permutohedral Lattice for High Dim

Speed Comparison of Permutohedral Lattice with Bilateral Grid and Gaussian Kd-tree

Gauss Transform and its fast version • Discrete Gauss transform

– v’(x) = Σ’y v(y) f(p(x) - p(y)); – Take a high-dimensional signal; – Put it into a data structure; – Perform a Gaussian blur really fast; – Read out its values.

• Fast Gauss Transform [Greengard &Strain’91]: reduces computation from O(MN) to O(M + N) for N Gaussians at M points in d dimensions; – The more general “Fast Multipole Method (FMM)”.

• Improved by [Yang’03]: for high dimension d, use a multivariate Taylor expansion scheme combined with the space subdivison using the k-center algorithm; – Employ “Farthest-Point Clustering”.

Fast Gauss Bilateral Filtering • Bilateral filtering as convolution: a weighted

sum of Gaussians;

• Bilateral filtering as Gauss Transform: a ratio of two (n+1)-d Gauss Transform;

• Dimensional elevation [Paris’06]: clustering and Taylor series expansion;

• Fast Gauss Transform: a fast multi-pole method strategy, O(N).

[Yoshizawa’10]

Comparison of Fast Edge Preserving Filters

Constant Time Bilateral Filtering O(1)

• [Porikli’08] • Use of integral histogram for box spatial kernel; • Use of linear filters of image powers for polynomial range kernel; • Linear filter decompositions by Taylor series for Gaussian range kernel .

Real-time Bilateral Filtering O(1) • [Yang et al.’09] • Bilateral filter can be decomposed into a number of O(1) spatial filters for

each pixel value (quantization is moved into a range function); • Output as linear interpolation of each closest principle bilateral filtered

image component pair ( I(x) = k, k+1 ); – Box filtering ->integral images; – Gaussian filtering -> recursive filtering; – Polynomial filtering -> a set of integral images; – Median filtering -> integral image.

Fast Median and Bilateral Filtering • Median filter:

– Huang’s O(r): adjacent sliding windows overlap to some extent; • Makes use of this sequential overlap to consolidate the redundant

calculations, reducing the computational complexity to O(r)

– Distributive Histograms O(log(r)): If multiple columns are processed at once, the redundant calculations become sequential. • Histogram distributivity: for disjoint image regions A and B, their

union’s window W compute its histogram by HW=HA + HB; • The union window’s median comes from scanning HW, splicing

together from HA and HB on the fly.

– High depth median filter: storage consideration • Ordinal transform: reduce the histogram size; • Compound histogram: Hc

• Coarse to fine recursion

[Weiss, Siggraph’06]

Fast Median and Bilateral Filtering

O(r) O(logr)

Fast Median and Bilateral Filtering • Bilateral filter:

– box spatial kernel: weighting function becomes constant – 3-d histogram: – logarithmic bilateral:

[Weiss, Siggraph’06]

Smoothed Local Histogram Filter

• Image filter from local histogram: – Median, Dilation/Erosion filter: 50% ,100%/0% point of the local histogram;

• How to accelerate local histogram? – [Huang75] O(n), [Weiss06] O(log(n)), [Porikli05] O(1); – Center-weighted isotropic histogram neighborhood;

• KDE (Parzen window) as a smoothed locally weighted histogram

– K determines the frequency content of as a function of s; – W determines the spatial frequency content; – For W arbitrary kernel, the convolution can performed at the cost of O (log(n))

operations per output pixel using 2D FFT (for n neighborhood size) ; – If K, W are both Gaussian, the convolution can be done in O(1) constant time,

independent of neighborhood size per pixel.

• Designed new filters: – Closest mode filter: restricted mean shift; – Dominant mode filter: choose the mode corresponding to the largest population.

)(ˆ sf p

[Kass, ACM T-Graphics’10]

Smoothed Local Histogram Filter

Accelerating Spatially Varying Gaussian Filters • Drawback of Bilateral Filter

– piecewise-flat regions, false edges, blooming at high contrast regions; – spurious detail at color edges;

• Trilateral filter: Besides, the filter window is “tilted” by the gradient vector; • Acceleration schemes of S. V. (spatially varying) filters: importance sampling

– Directly evaluate anisotropic filter kernel by integration for Gaussian kd-tree; – Embeds data in higher dim space to make kernels separable, i.e. dimensional

elevation M:(x,y) (x, y, x+y, x-y); – Segments dataset into overlapping regions of constant anisotropic kernel to

accelerate in the permutohedral lattice; – Kernel sampling sparsely and segment by clustering, then interpolate result finally.

[Baek, Siggraph Asia’10]

noisy signal

bilateraled

kernels along the gradient

l

x

y

2D Gaussian kernel 3D Gaussian kernel ?

Spatially varying Gaussian filtered

The trilateral filter is a “gradient-preserving” filter

Comparison of Bilateral and Trilateral

Bilateral filter

Trilateral filter

Comparison of Bilateral and Trilateral Bilateral

Kernel sampling

Trilateral

Blooming & false edge

Kernel sampling

Domain Transform [Gastal, Siggraph’11] • Isometry (distance preserving) between curves on 2d manifold and real lines; • 1D filtering on transformed multiple-dimension domain e.g. a case of 2D

filter on a color image (XY, RGB) • Real-time high-quality edge aware filtering of image/video based on a

dimensionality reduction

Smoothing on transformed domain Ωw ≡ Edge-preserving on Ω

Ωw is ct(u)

Domain Transform

Domain Transform [Gastal, Siggraph’11] • Implementation for image filtering with iterations

– Horizontal pass along each image row

– Vertical pass along each image column

• Normalized Convolution (NC)

• Interpolated Convolution (IC)

• Recursive Filtering (RF)

)()ˆ( pctpt

d ↗ ⇨ ad 0 .. stop propagation and preserve edge

Halve σH at each iteration. Stop while σH is small enough )/2exp( Ha

Filter Quality Comparison with Bilateral Filter, WLS, Anisotropic Diffusion and Edge-Avoid Wavelet

Guided Image Filtering [He et al.’10]

• Problems in bilateral filter – Complexity – Gradient distortion: preserve edge, not gradient;

• Guided filter: in all local windows , compute the linear coefficients and the average of in all local windows that covers pixel

• Gradient preserving: q has an edge only if I has an edge • Integral images

– O(1) time – Non-approximate

Naturally O(N) time independent of the window radius

mean

variance

Guided Image Filtering [He et al.‘10]

Linear regression

Bilateral/joint bilateral filter does not have this linear model!

Noise/texture

Patch Match [Barnes’09]

• Nearest neighbor field: nearest neighbor over all possible patch candidates for some distance function between two patches;

• Naive brute force search is expensive: O(mM^2);

• Dimensionality reduction is a limited solution;

• Popular methods: KD-trees, Locality Sensitive Hashing (LSH); – These methods can be very accurate, but are relatively slow;

• Patch Match is an iterative randomized algorithm: iteratively improve the nearest neighbor field, based on local propagation and mostly local search, until convergence; – Heavily relies on the fact that images are generally coherent;

– Search space: patch offsets vs. patches;

– Neighboring pixels have coherent matches;

– Large number of random sampling will yield some good guesses.

Patch Match [Barnes’09] • Each iteration, for each pixel:

– Randomly located patches (with the prob. that drops exp. with distance from current match);

– Check if the offsets from neighboring patches give a better matching patch; If so, adopt neighbor’s patch offset;

– Search for better patch offsets within a concentric radius around the current offset; search radius halved each time till 1 (started with the image size).

Generalized Patch Match [Barnes’10] • Dealing with higher dimensional spaces;

• Scales and Rotations: from (x, y) to (x, y, θ, s);

• Any distance function (pairwise comparisons) e.g. – SIFT, a popular computer vision descriptor;

– Patches invariant to lighting changes;

• k-Nearest Neighbors: heap-based. – Store k correspondences in max-heap;

– If new candidate better, pop heap, push new candidate.

• Outperform KD-tree and LSH by two orders of magnitude.

Ground truth Gaussian noise NLM PM Generalized PM

Coherence Sensitive Hashing [Korman & Avidan ’11]

• CSH extend LSH and Patch Match: – LSH relies on hashing by mapping similar patches to the same bin;

– PM relies on images are coherent, to propagate good matches to neighbors;

– CSH relies on hashing to seed the initial patch matching and on image coherence to propagate good matches (3~4 times faster than PM).

• Indexing and Search: Input: images A, B, Output: nearest patch map ANNF

Coherence Sensitive Hashing [Korman & Avidan ’11]

• Hash all patches of both images (with hash function g) – Keep k (=2) representatives of each image in each entry

p

Image ‘A’ Image ‘B’ k k

Hash Table

g

g

g-1 g-1

CSH Indexing

Coherence Sensitive Hashing [Korman & Avidan ’11]

Type 1 Type 2

Type 3

Type 1 Type 2 Type 3

Uses appearance

Uses coherence

# candidates 2 4 - 6 2

Estimated contribution

20% 50% 30%

(LSH) (> PM)

CSH Indexing – Candidate (potential matching patches) Types

appearance only

appearance and coherence appearance only

Coherence Sensitive Hashing [Korman & Avidan ’11]

CSH Recap Preprocessing: • Project A’s and B’s patches on Walsh-Hadamard vectors

Iterate (typically 1-3 times): • Hash patches into a table

– Recall the hash function;

– hi(v) is a binned projection of v on one of WH kernels;

– Heuristics adapt to these kernels in:

• number of bins (per projection);

• spacing of bins.

• For each patch in A

– Create candidate patches in B

– Pick the best one

A B

)()()( 1 vhvhvg n

r

bvavh ii

i )(

Propagation-Assisted KD-Tree [He & Sun ’12]

• Jointly exploit distrib. of candidates and dependency of queries;

• Build a kd-tree using all the patches in the Candidate Set. Each leaf contains no more than m candidates.

• Scan the image A in raster order. For each patch pA(x, y) in A, do propagation-assisted kd-tree search as follows: – Descend the tree to a leaf (Leaf #0);

– Propagate a leaf from left/upper, using the already matched result of pA(x-1, y) and pA(x, y-1);

– Find the NN of pA(x, y) in all the above leaves.

• Check for each query its own leaf and extra leaf propagated from nearby queries after organizing the candidates in a KD-tree;

• Check fewer candidates and has no backtracking;

• 10~20 times than PM, 2~5 times than CSH.

Propagation-Assisted KD-Tree [He & Sun ’12]

The joint distribution P(e0, e1) in different search strategies

Propagation in Patch Match Propagation-assisted KD-tree

References • Tomasi, C., Manduchi, R. “Bilateral filtering for gray and color images”. ICCV’98, pp839–

846. • F. Durand and J. Dorsey, “Fast bilateral filtering for the display of high-dynamic-range

images,” ACM T-Graphics, 21(3), pp. 257–266, (SIGGRAPH 2002). • Choudhury, P., Tumblin, J., "The Trilateral Filter for High Contrast Images and Meshes,"

Eurographics Symp. on Rendering, pp. 186-196, 2003 • T. Pham, L. Vliet, “Separable bilateral filtering for fast video preprocessing,” IEEE ICME,

2005 • S. Paris and F. Durand, “A fast approximation of the bilateral filter using a signal

processing approach,” ECCV, 2006. • B. Weiss, “Fast median and bilateral filtering,” ACM T-Graphics, 25(3), ACM SIGGRAPH,

2006. • Porikli, F. “Constant time o(1) bilateral filtering”, IEEE CVPR’08. • Q Yang, K Tan and N Ahuja, “Real-time O(1) Bilateral Filtering”, IEEE CVPR 2009. • J. Chen, S. Paris, and F. Durand, “Real-time edge-aware image processing with the

bilateral grid,” ACM Transactions on Graphics (SIGGRAPH 07). • A Adams, N. Gelfand, J. Dolson, and M. Levoy, “Gaussian KD-Trees for Fast High-

Dimensional Filtering,” ACAdams M Transactions on Graphics (SIGGRAPH 2009).

References • S Yoshizawa, A Belyaev, H Yokota, Fast Gauss Bilateral Filtering , Computer Graphics

forum’10. • M Kass, J Solomon, Smoothed Local Histogram Filters, ACM T-Graphics, 2010. • A Adams, J Baek, A Davis, “Fast High-Dimensional Filtering Using the Permutohedral

Lattice”, Eurographics 2010. • Baek, J., Jacobs, D. E. “Accelerating Spatially Varying Gaussian Filters”. SIGGRAPH

Asia 2010. • E Gastal and M. Oliveira, Domain Transform for Edge-Aware Image and Video

Processing, SIGGRAPH 2011.

• C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. “Patchmatch: a randomized correspondence algorithm for structural image editing”. SIGGRAPH, 2009.

• C. Barnes, E. Shechtman, D. B. Goldman, and A. Finkelstein. “The generalized patchmatch correspondence algorithm”. ECCV, pages 29–43, 2010.

• Kaiming He, Jian Sun, and Xiaoou Tang, “Guided image filtering”, ECCV 2010.

• S. Korman and S. Avidan. “Coherency sensitive hashing”. ICCV, 2011.

• K He, J Sun, “Computing Nearest-Neighbor Fields via Propagation-Assisted KD-Trees”, CVPR, 2012.

Thanks!