Upload
chiamin-hsu
View
1.623
Download
4
Tags:
Embed Size (px)
Citation preview
04/12/2023 Domain Transform
DOMAIN TRANSFORM FOR EDGE-AWARE IMAGE AND VIDEO PROCESSINGEduardo S. L. Gastal and Manuel M. Oliveira
Instituto de Inform´atica – UFRGS
SIGGRAPH 2011
04/12/2023 Domain Transform
Edge-aware filter• e.g. Bilateral filter
• Weighting F of spacial distance and (color) range distance
Spacial distance is fixed.Fast
Range distance and kernel value has to be calculated Bottleneck !
How about L1 distance ?
How about replacing 2D filter with 1D vertical and horizontal filter ?
How to speed up?No way in most cases !
pq0
||PQ||1 = |xp-xq| + |yp-yq| , ||PQ||2= √((xp-xq)2+(yp-yq)2)
04/12/2023 Domain Transform
Abstract• 1D filtering on transformed multiple-dimension domain
e.g. a case of 2D filter on a color image (XY, RGB)
This paper provides high quality edge-preserving smoothing after
• (x, RGB) ct(x). Apply 1D horizontal filter on ct(x)
• (y, RGB) ct(y). Apply 1D vertical filter on ct(y)
• Real-time at arbitrary scale
• Many applications of this real-time edge preserving filter • Depth-of-field effects, stylization, recoloring, colorization, detail
enhancement, and tone mapping.
04/12/2023 Domain Transform
Contribution• Real-time high-quality edge aware filtering of image/video
based on a dimensionality reduction • Smoothing on ct(X,RGB) and ct(Y, RGB) ≡ edge preserving on 2D
color image
• Perform anisotropic edge-preserving on curves of 2D image manifold using 1D linear filters• Isotropic kernel on ct(x) ≡ anisotropic kernel on x
• 2D edge-preserving filter as a sequence of 1D linear filter
• The first real-time edge-preserving smoothing• Any scale kernel• Process time independent of the filter parameters• Can control kernel shape• Handle color image correctly
1D
04/12/2023 Domain Transform
RELATED WORK
04/12/2023 Domain Transform
It’s have been well known that …
2D Smoothing ≡ 1D Edge-preserving
However, replace 1D Bilateral with 2D Gaussian …. High Cost !
04/12/2023 Domain Transform
2D Smoothing ≡ 1D Edge-preservingBilateral grid
[Chen et al. SIG07]
Gaussian KD-tree
[Adams et al. SIG09]
higher dimensional functions
Gaussian convolution
division
slicing
w i w
limited for small kernels, coarse problem not real-time
down-sample
04/12/2023 Domain Transform
Anisotropic Diffusion Filter (AD)
• [Perona and Malik 1990] suggested ppt
• Diffusion is smoothing : pixels diffuse ∆I • Anisotropic kernel preserves edge : strong edges reduce kernel size
Unknown iterations
c(p, t) is large when p is not a part of an edge c(p, t) is small when p is a part of an edge
04/12/2023 Domain Transform
Edge-avoid Wavelets (EAW)• [Raanan Fatta SIG2009]
Constrain the size of the smoothing kernel to 2^n
translation-invariant trans. Edge-avoiding trans.
Edge-avoiding scaling func.
inversely related to the pixel difference.
04/12/2023 Domain Transform
DOMAIN TRANSFORM
04/12/2023 Domain Transform
Domain transform ct(u)
Smoothing on transformed domain Ωw ≡ Edge-preserving on Ω
Ωw is ct(u)
04/12/2023 Domain Transform
I’
|I’|
04/12/2023 Domain Transform
|I’|
ct(u) is accumulation of each L1 distance of (x, I(x))
04/12/2023 Domain Transform
Domain Transform ct(u)
RG B
ct(u) : (u,I(u)) 1 : 1
ct(u) : (u,I(u)) 1 : 1
ct(u) is accumulation of each L1 distance of (u, I(u))
04/12/2023 Domain Transform
Domain Transform ct(u)
RG B
ct(u) : (u,I(u)) 1 : 1
ct(u) : (u,I(u)) 1 : 1
ct(u) is accumulation of each L1 distance of (u, I(u))
04/12/2023 Domain Transform
Ωw = ct(u)pq
04/12/2023 Domain Transform
Ωw = ct(u)
edge edgeedgeedge
p
q
04/12/2023
Smoothing Ωw ≡ Edge-preserving Ω
Ωw is ct(u)
WHY ?
Smoothing Ωw ≡ Edge-preserving Ω
04/12/2023
Smoothing Ω can not preserve edge …
Smoothing Ωw ≡ Edge-preserving Ω
04/12/2023
Smoothing Ωw ≡ Smoothing neighbors in L1
Sample Ωw uniformly ≡ sample more densely in high |I ’| (edge) Sample Ωw uniformly ≡ along curve in Ω ≡ adaptive kernel Ω |(spatial, color)|-1
L1
sample Ωw uniformly
sampling along the curve
Smoothing on Ω
Smoothing Ωw ≡ Edge-preserving Ω
04/12/2023 Filter Attributes
Kernel size of Bilateral filter – σs, σr • σs
2 = variance of spatial kernel. σs radius of spatial kernel
• σr2
= variance of range kernel. σr radius of range kernel
large σs
σr
x
I
pkernel in Ωw
What do σs and σr mean in Ωw ?
04/12/2023 Filter Attributes
Kernel size of Bilateral filter – σs, σr • σs
2 = variance of spatial kernel. σs radius of spatial kernel
• σr2
= variance of range kernel. σr radius of range kernel
small σs
σr
x
I
p
What do σs and σr mean in Ωw ?
kernel in Ωw
04/12/2023 Domain Transform
Kernel size of Bilateral filter – σs, σr • σs
2 = variance of spatial kernel. σs radius of spatial kernel
• σr2
= variance of range kernel. σr radius of range kernel
small σs
σr
x
I
pscaled kernel in Ωw
|pq| L1 = |px-qx| + |pr-qr|
q non-scaled kernel in Ωw
04/12/2023 Filter Attributes
Kernel size of Bilateral filter – σs, σr • σs
2 = variance of spatial kernel. σs radius of spatial kernel
• σr2
= variance of range kernel. σr radius of range kernel
large σs
σr
x
I
p
|pq| L1 = |px-qx| + |pr-qr|
scaled kernel in Ωw
non-scaled kernel in Ωw
04/12/2023 Filter Attributes
Scaling Factors – σs, σr • σs
2 = variance of spatial kernel. σs radius of spatial kernel
• σr2 = variance of range kernel. σr radius of range kernel
• σH2 = variance of transformed domain kernel. σH radius of transformed
domain kernel
04/12/2023 Filter Attributes
Relationship to σs, σr and I
no longer edge-preserving
unbounded smoothing as input
as input
σs
σr
x
I
04/12/2023
FILTERING IN THE TRANSFORMED DOMAINNormalized Convolution (NC)
Interpolated Convolution (IC)
Recursive Filtering (RF)
04/12/2023
Filtering in Ωw
• Sample uniformly in Ωw is unnecessary
• It’s easy to determine who is neighbor in Ωw
I
I Ω
Ωw
p
ct(p)
ab
c
ab
c
ct(p)-ct(a) < r a is b’s neighborct(c)-ct(p) > r c is not b’s neighbor
04/12/2023
Normalized Convolution (NC)
𝑡 (�̂� )=𝑐𝑡 (𝑝 )
𝑡 (�̂� )
moving-average [Dougherty 94]
[Knutsson and Westin 93]
r
04/12/2023
Normalized Convolution (NC)
𝑡 (�̂� )=𝑐𝑡 (𝑝 )
moving-average [Dougherty 94]
[Knutsson and Westin 93]
why cost time can be independent on filter attributes ( or ) by moving average?
Bilateral : large costs much time NC : Tlarge σs is smilar to Tsmall σs
Σ I
Ω
√3𝜎𝑠0 √3𝜎𝑠1
04/12/2023
Interpolated Convolution (IC)[Piroddi and Petrou 04]
continuous, not discrete !
04/12/2023
NC vs. IC
𝑡 (�̂� ) 𝑡 (�̂� )
Normalized convolution (NC) Interpolated convolution (IC) NC, IC
04/12/2023
Recursive Filtering (RF)
a ∈ [0, 1] is a feedback coefficient
An infinite impulse response (IIR) with exponential decay
d a↗ ⇨ d 0 .. stop propagation and preserve edge
[Smith 07]
04/12/2023
FILTERING 2D SIGNALSReplacing 2D filter with multiple iterations of 1D vertical and horizontal filter
04/12/2023 Filtering 2D signals
Filtering 2D Signals
• Is it possible to apply domain transform on (x,y) ? • ct(p) is the accumulated L1 distance along (p, I(p))
• Only exists in surface with zero Gaussian curvature [O’Neill 06]•
• Implementation• Several iterations
• Horizontal pass along each image row• Vertical pass along each image column
04/12/2023 Filtering 2D signals
How to Determine #Iterations ? • Stripes are only present along the last filtered dimension
• The length of the stripes size of the filter in the last pass
1st iteration 2nd iteration
stripes
Halve σH at each iteration. Stop while σH is small enough
04/12/2023 Filtering 2D signals σH = σs = 40, σr = 0.77
σH = σs = 50, σr = 0.5
2 itr.
04/12/2023
COMPARISON TO OTHER APPROACHES
04/12/2023
Impulse ResponseCase : the left impulse without strong edge; the right impulse with strong edge
NC:Normalized convolution; IC:Interpolated convolution; RF:Recursive convolution; BF:Bilateral ; AD:Anisotropic Diffusion; WLS:Weighted Least Squares
The NC and IC filters have Gaussian-like response, similar to AD and BF.
Strong edges : IC ~ AD. NC has a higher response near strong edges : pixels near
edges have less neighbors in the same population, and will weight their contribution strongly.
Impulse : RF ~ WLS
#iteration = 3
Ωw
Ω
strong edge
04/12/2023
Smooth Quality
04/12/2023
Performance
Filtering on CPU
• 2.8 GHz Quad Core PC • 8GB memory• NC, RF by C++• IC in MATLAB
• 1M RGB * #3 = 0.16~0.06 sec.• 10 M RGB * #3 = 1.6 ~ 0.6 sec.
• Linearly with image size • Independent on σs or σr
• Faster than EAW, PLBF, CTBF
Filtering on GPU
• NVIDIA GeForce GTX280• CUDA
• 1M RGB * #3 = 0.007 sec.• Domain tx : 0.7 mx• Each iteration : 2 ms
04/12/2023
Comparison
• Bilateral Grid [Chen et al. 07]• Faster than domain tx. • Luminance only• Due to down-sampling process, not possible for small
spatial and range kernel
• NVIDIA GeForce GTX280
This paper PLBF WLS
1M RGB * #3 = 0.007 sec
0.5 M RGB = 0.1 sec
1 M gray = 1sec
04/12/2023
REA-TIME APPLICATIONS OF EDGE-PRESERVING FILTER (EPF) Detail Manipulation Tone Mapping Stylization Joint Filtering Colorization Recoloring
04/12/2023
Detail Manipulation
• EPF decomposes image into level-of-detail : J0, J1, .. Jk
• I = J0 , Di = Ji – Ji+1
input I using D0 by IC (σs=20, σr=.08) [Farbman et al. 08]
EAW by [Fattal 09]
04/12/2023
Tone Mapping• Edge-aware tone mapping avoids haloing and artifacts due to compression• The compressed luminance channel Lc in HDR [Farbman et al. 08]
This paper (RF) WLS
12 msec on log-luminance by RF ! J1 : σs= 20, σr=0.33 J2 : σs= 50, σr=0.67 J3 : σs=100, σr=1.34
I = J0 , Di = Ji – Ji+1 μ is the mean of Blocal min B = 0local max B = 1
04/12/2023
Stylization 1/2
• Abstract low-contrast region and preserving high-contrast features
edge + this edge-aware filtered image
NC filtering
04/12/2023
Stylization 2/2
• Assign each output pixel a scaled version of the value of the normalization factor Kp in NC
0.11 Kp
Kp : #points in the moving windowsOn the edge : #point dark↘
On the smooth area : #point light↗
04/12/2023
Joint Filtering• Alpha can be combined with other map to create some
localized or selective stylization
alpha map
04/12/2023
Joint Filtering• Smooth image A based on the
edges information of image B simulate depth-of-field (DoF)
depth-of-field effect
𝑐𝑡 (𝑢)=∫0
𝑢
1+𝜎 𝑠
𝜎𝑟
|𝐵 ′ (𝑥)|𝑑𝑥
B is an alpha matte
04/12/2023 Domain Transform
Joint FilteringCanny edge identifies edges
04/12/2023
Colorization• Propagate user color S by blurring them using the edge
information
using this RF (σs=100, σr=.03) [Levin et al. 04]User color scribbles
𝐶𝑜𝑙𝑜𝑟𝑝=~𝑆(𝑝 )/~𝑁 (𝑝)
S : user scribbles N : normalization function : blurred S : blurred N
04/12/2023
Recoloring
• Soft segmentation – region Ri defined by color Ci
𝐶𝑜𝑙𝑜𝑟𝑝=~𝑁𝑅𝑖 (𝑝 )/∑
𝑗
~𝑁𝑅𝑗 (𝑝)
04/12/2023
Conclusions and Future Work• High-quality edge-preserving filtering of image and videos
in real time• First real-time edge-preserving filter on color image at arbitrary
scale • 1D smoothing filter on transformed domain ≡ Edge-preserving
filter • High quality 2D edge-preserving filtering by iterating 1D-filtering• independent of the filter parameters(σs, σr) (due to moving average)
• Future work• Edge-preserving applications
• Limitation• Not rotationally invariant
04/12/2023
Q & A