Upload
ulla
View
36
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Unified Loop Filter for High-performance Video Coding. Yu Liu and Yan Huo ICME2010, July 19-23, Singapore. Outline. Introduction Proposed Unified Loop Filter Experimental Results Conclusion. Introduction. Conventional Video Coding Standard Block-based DPCM coding - PowerPoint PPT Presentation
Citation preview
Unified Loop Filter for High-performance Video Coding
Yu Liu and Yan Huo
ICME2010, July 19-23, Singapore
Outline
2
• Introduction• Proposed Unified Loop Filter• Experimental Results• Conclusion
Introduction
• Conventional Video Coding Standard– Block-based DPCM coding
• Transform, quantization, ME/MC, in-loop deblocking filter, entropy coding
• In-loop Deblocking Filter– A bank of fixed low-pass filters to alleviate blocking artifacts
• Assume smooth image model– thus singularities, such as edges and textures, are not handled correctly– an analysis on the gradients across the boundary is performed to check whether the
filtering should be skipped to preserve image sharpness.
• Pre-defined filter coefficients– do not retain the frequency-selective properties or have the ability to suppress
quantization noise optimally
3
Introduction
• Adaptive Wiener Filter [4-7]– Well-known optimal linear filter: adaptive post/loop filter
• Improve the quality of reconstructed picture degraded by compression
– Pros:• Guarantee the optimized objective quality restoration
– Cons:• Can’t efficiently improve the subjective quality if used alone; thus• Have to be utilized on top of in-loop deblocking filtered picture to achieve
both improved objective and subjective quality
4
Introduction
• Proposed Unified Loop Filter– Is there a way to combine the advantages of deblocking
filter and Wiener filter into a unified filtering framework?– Motivation
• The fact: multiple sources of information loss in current video coding standards, such as H.264/AVC
• Quantization: the only source of information loss, prior to H.264/AVC• Deblocking loop filter in H.264/AVC: another source of information loss• Adaptive loop filter in KTA:
– may not be able to reach the capability upper-bound of picture restoration– due to additional information loss brought by the deblocking filtering
– Unified Loop Filter• Reduces the number of the sources causing information loss• Thus further improves the capability of picture restoration
5
Introduction
• Block Diagram of Conventional Video Codec
6
T Q
Intra-Pred
Inter-Pred
ME
Reference Buffer
Adaptive Loop Filter
DeblockingLoop Filter
Q-1+T-1
Entropy Coding
+
+
Adaptive Post Filter
Video
-
Display
Introduction
• Block Diagram of Video Codec with Unified Loop Filter
7
T Q
Intra-Pred
Inter-Pred
ME
Reference Buffer
Unified Loop Filter
Q-1+T-1
Entropy Coding
+
+Video
-
Display
Proposed Unified Loop Filter
• Order Statistics Filter– Filters utilizing order statistics information [9], improved on median filters, can
effectively remove the blocking artifacts and ringing artifacts, while retaining the sharpness of edges.
– Although order statistics filter is a nonlinear filter, it can be optimized to minimize the mean square error by using linear combination of ordered statistics.
– Here, the order statistics filter is used to combine nonlinear enhancement filter and linear restoration filter into one unified filtering framework.
• Unified Loop Filter– Suppose that X=(x1, x2, …,xc,… , xN)T is a support vector
containing N pixels of the reconstructed picture arranged by the spatial order surrounding the central pixel xc.
– The unified loop filter is constructed as follows:
8
...
... xc
...
x1
x2
...
...
xN
...
...
...
...
Proposed Unified Loop Filter
• Step 1: Vector of Similarity Statistics– The support vector X is converted to form a vector of similarity statistics X’=(x’1,
x’2, … , x’N)T by using the following equation:
where f(xc,xi) is the similarity function. Real-valued similarity functions have to satisfy the following constrains:
– In this paper, the following similarity function is adopted:
where σ is the spread parameter controlling the strength of similarity function.
9
iiccici xxxfxxxfx ),()],(1['
. if),,(),(.3
,0),(lim.2
,1),(lim.1
22112211
||
0||
bababafbaf
baf
baf
ba
ba
)/)(10(1),( 22 ba,,Clipbaf
Proposed Unified Loop Filter
• Step 2: Similarity-Ordered Statistics Filter– The vector of similarity statistics X’ is further ordered to form a vector of
similarity-ordered statistics Xn=(x’(1),x’(2), …, x’(N))T by using the following rule:
– Then the output of non-linear similarity-ordered statistics filter becomes
where Wn is the vector of N optimized filter coefficients.
10
cici xxxx )1()( ''
nTnn XWy
• Step 3: Unified Loop Filter– In order to improve the coding efficiency, linear Wiener filter should also be
incorporated into the unified loop filter. Suppose that Xl=(x1, x2, …,xc,… , xM)T is the support vector of Wiener filter, the output of Wiener filter becomes
where Wl is a vector of M optimized filter coefficients. – Generally speaking, Wiener filter is also a kind of order statistics filter, called as
spatially ordered statistics filter, because its support vector Xl is constructed by arranging the M pixels in a spatial order. Therefore, nonlinear similarity-ordered statistics filter is concatenated with linear spatially ordered statistics filter, aka Wiener filter, to form the unified loop filter:
where Xu =(x’(1), x’(2), … , x’(N) , x1, x2, … , xM )T and Wu is a vector of M+N optimized filter coefficients.
Proposed Unified Loop Filter
11
uTuu XWy
Nonlinear Part Linear Part
lTll XWy
• Optimization of Unified Loop Filter– The optimization of the unified loop filter falls into the classical optimization
framework of least mean square error (LMSE). The solution can be obtained by solving the Wiener-Hopf equations:
where xo is the original video frame. The minimization problem can be solved by the Wiener-Hopf equation, which is given by
where Ru,u(k,l) is the auto-correlation function of xu, which is defined as
and Ro,u(l) is the cross-correlation function between xo and xu , which is defined as
Proposed Unified Loop Filter
12
2
1~
2
~))()((minarg)(minarg
~
NM
kuuo
Wuo
Wu kxkwxEyxEW
uu
NMlforlRlkRkwNM
kuouuu
1),(),()(1
,,
)]()([),(, lxkxElkR uuuu
)]([)(, lxxElR uouo
Proposed Unified Loop Filter
• Filter Design– Two considerations in the filter design of unified loop filter:
• not only the subjective enhancement (for removing the blocking and ringing artifacts)• but also the objective restoration (for improving the coding efficiency)
– For Luma Component (Y)• In unified loop filter, nonlinear part consists of one 12-tap diamond filter, and linear part consists
of four kinds of different taps (1-tap, 13-tap, 25-tap, and 41-tap) diamond filters with central point symmetry
13
2
5 6
9
0
1
4
8
11
10
7
3
x
0
or or
or
Nonlinear part Linear part
2
5 6
2
0
1
4
3
0
1
4
3
5
2
5 6
2
0
1
4
3
0
1
8
3
7
11 1210 10119 9
7 68 45
2
5 6
2
0
1
4
3
0
1
8
3
7
11 1210 14139 15
7 68 45
19 2018 181917 1716 16
13 1214 101115 9
Proposed Unified Loop Filter
• Filter Design – For Chroma Components (Cr/Cb)
• In unified loop filter, nonlinear part consists of one 4-tap diamond filter, and linear part consists of two kinds of different taps (1-tap and 13-tap) diamond filters with central point symmetry
• Selection of Filter Tap Type– The tap type of linear part in unified loop filter is decided by rate-distortion optimization
selection within the whole frame:
– The filter side information includes the filter tap type and filter coefficient quantization bits, which are encoded and transmitted to the decoder side.
14
0
or
Nonlinear part Linear part
2
0
1
3
x
2
5 6
2
0
1
4
3
0
1
4
3
5
RDJ filter
Experimental Results
• Test Conditions– The proposed unified loop filter has been implemented within JM11.0KTA2.4r1 reference
software. The test conditions are listed as follows:
Table 1. Test conditions
15
Test Sequence 1280x720 progressiveCoding Structure IBBPBBP…
Intra Frame Period Only the first frameEntropy Coding CABAC
R-D Optimization ON
QP I(22, 27, 32, 37) P(23, 28, 33, 38) B(24, 29, 34, 39)
Transform Size 4x4, 8x8Reference Frame 4
Search Range ± 64Frame Number 61
• Objective Performance Comparison
Experimental Results
Table 2. Coding gain comparison of different coding schemes, compared with H.264/AVC High Profile, in BD bitrate reduction (BR) and BD-PSNR gain (for Luma)
16
SequenceALF w/o DLF ALF + DLF QALF + DLF ULF
Δ BR (%) Δ PSNR (dB) Δ BR (%) Δ PSNR (dB) Δ BR (%) Δ PSNR (dB) Δ BR (%) Δ PSNR (dB)
Bigships -4.82 0.13 -6.93 0.19 -8.38 0.23 -9.95 0.28City -15.49 0.51 -15.43 0.52 -16.32 0.55 -17.27 0.58
Crew -7.39 0.2 -10.74 0.28 -14.43 0.39 -12.12 0.33Harbour -12.64 0.49 -14.26 0.55 -14.21 0.55 -14.37 0.56
Jet -5.51 0.12 -6.46 0.15 -10.64 0.28 -11.27 0.30Night -1.46 0.05 -5.52 0.21 -6.32 0.24 -6.54 0.25Optis -5.75 0.15 -5.84 0.14 -7.87 0.20 -12.28 0.32
Sailormen -7.63 0.20 -10.24 0.27 -11.14 0.30 -11.77 0.31Sheriff -5.89 0.16 -7.72 0.22 -8.90 0.25 -10.64 0.30
ShuttleStart -7.73 0.23 -7.27 0.21 -9.36 0.27 -9.54 0.28SpinCalendar -4.11 0.09 -7.38 0.19 -14.31 0.36 -11.03 0.29
Average -7.13 0.21 -8.89 0.27 -11.08 0.33 -11.53 0.34
Experimental Results
• Subjective Quality Comparison
17
(a) Anchor (b) ALF w/o DLF
(c) ALF + DLF (e) ULF Figure. Part (128x128) of the reconstructed SpinCalendar sequence at the 54th frame with QP=38.
• Complexity Comparison– Compared with “ALF+DLF”, extra complexity introduced by ULF includes:
• Similarity computation: lookup-table (LUT) technique • Sorting process: counting sort algorithm with linear complexity O(n).
– With the help of the two fast algorithms• ULF only increases the execution time 13.70% and 26.52% on average for the encoder and the
decoder, respectively, compared with “ALF+DLF”
Experimental Results
Avg. Exec. Time (s)
Anchor ALF w/o DLF ALF+DLF QALF+DLF ULF
Enc Dec Enc Dec Enc Dec Enc Dec Enc Dec
Bigships 1019.695 12.332 1116.576 16.809 1116.710 19.074 1507.433 16.715 1368.247 23.549
City 1102.105 12.789 1198.029 16.813 1197.776 19.137 1611.106 17.707 1467.945 23.794
Crew 1010.664 11.746 1109.336 15.347 1109.758 18.145 1501.561 15.902 1173.826 23.790
Harbour 1158.656 14.234 1251.545 18.152 1253.102 21.360 1654.652 21.055 1396.649 26.969
Jet 914.743 11.840 1011.421 13.864 1012.875 15.496 1300.529 13.352 1104.618 19.751
Night 1101.743 12.445 1199.932 16.985 1197.756 19.219 1580.087 16.715 1292.450 26.666
Optis 995.519 12.461 1095.312 15.683 1093.754 17.532 1450.333 16.035 1259.534 21.738
Sailormen 1102.997 14.032 1197.463 18.000 1199.272 20.309 1612.117 18.324 1352.911 25.752
Sheriff 1006.427 12.945 1103.943 16.508 1105.292 18.793 1468.728 17.258 1258.969 23.923
ShuttleStart 874.494 10.856 973.000 12.438 973.945 14.828 1246.393 13.008 1085.115 17.301
SpinCalendar 1117.885 14.329 1214.541 16.945 1217.469 20.535 1614.543 17.219 1426.570 25.406
Total Average 1036.812 12.728 1133.736 16.140 1134.337 18.584 1504.316 16.663 1289.712 23.513
18
Table 3. Average execution time comparison of different coding schemes
Conclusion
• Unified Loop Filter (ULF)– Combine the advantages of linear filter and nonlinear filter to achieve both
objective and subjective quality optimization– The deblocking loop filter in conventional video codec can be removed, and thus
replaced by the proposed unified loop filter– The proposed unified loop filter can be used in any hybrid video coding system
• Classification-based Unified Loop Filter (CULF)– Global v.s. Local: different regions have different quantization error characteristics– It is better to classify pixels into different groups:
• one group for boundary pixels with blocking artifacts• one group for non-boundary and boundary pixels without blocking artifacts• other group for non-filtering pixels
– Different unified loop filter with different characteristics is applied to each group– Additional 1.57% bitrate reduction is achieved, compared with ULF
19
Thank You!
Q&A
20