Recent Advances in Video Coding.thomas Wiegand 2011

Thomas Wiegand

Heiko Schwarz

� Video Coding – State-of-the-Art

� High-Efficiency Video Coding• Motion representation• Motion compensation

• Transform coding• Intra coding

• Entropy Coding

slid

e 3

Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding

“It has been customary in the past to transmit successive

complete images of the transmitted picture.”

[...]

“In accordance with this invention, this difficulty is avoided by

transmitting only the difference between successive images of

the object.”

slid

e 4


1989: Digital TV – Digital Broadcast, DVD

1999: TML-1 of H.264/AVC

Today: more than 1 Billion devices with H.264/AVC

Every HDTV Receiver

Every Blu-Ray Player

Most Internet Video

Countless Mobile Video

slid

e 5


Source: http://techcrunch.com/2010/05/01/h-264-66-percent-web-video/

slid

e 6


EntropyCoding

Scaling & Inv. Transform

Motion-Compensation

ControlData

Quant.Transf. coeffs

MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation

Transform/Scal./Quant.-

InputVideoSignal

Split intoMacroblocks16x16 pixels

Intra-frame Prediction

De-blockingFilter

OutputVideoSignal

slid

e 7


Efficiency of Motion CompensationEfficiency of Motion Compensation• Intraframe coding: only spatial correlation exploited

� DCT [Ahmed, Natarajan, Rao 1974], JPEG [1992]

• Motion compensation (MC): integer-pel accuracy, 16x16� H.261 [1991]

• Half-pel accurate MC� MPEG-1 [1993], MPEG-2/H.262 [1994]

• 16x16 & 8x8 block size MC, quarter-pel� H.263 [1996], MPEG-4 [1999]

• Quarter-pel & 16x16 - 4x4 block size MC and multiple reference pictures

� H.264/AVC [2003]

• 64x64 – 4x4 block sizes and new filters for MC� HEVC [start 2010]

Complexityincrease

slid

e 8


Development of Video CodingDevelopment of Video Coding

0 100 200 300

28

30

32

34

36

38

40

Rate [kbit/s]

PSNR [dB] Half-pel

motion compensation(MPEG-1 1993MPEG-2 1994)

Integer-pelmotioncompensation(H.261, 1991)

Variable block size(16x16 – 8x8)(H.263, 1996) +quarter-pelmotion compensation(MPEG-4, 1998)

Variable block size(16x16 – 4x4) +quarter-pel +multi-framemotion compensation(H.264/AVC, 2003)

Intra frameDCT coding(JPEG, 1990)

Bit-rate Reduction: 75%

35

Foreman10 Hz, QCIF100 frames

slid

e 9


Development of Video CodingDevelopment of Video Coding

0 100 200 300

28

30

32

34

36

38

40

Rate [kbit/s]

PSNR [dB] Half-pel

motion compensation(MPEG-1 1993MPEG-2 1994)

Integer-pelmotioncompensation(H.261, 1991)

Variable block size(16x16 – 8x8)(H.263, 1996) +quarter-pelmotion compensation(MPEG-4, 1998)

Variable block size(16x16 – 4x4) +quarter-pel +multi-framemotion compensation(H.264/AVC, 2003)

Intra frameDCT coding(JPEG, 1990)

Foreman10 Hz, QCIF100 framesVisual Comparison

slid

e 10


Foreman, QCIF, 10 Hz, 100 kbit/sJPEG H.264/AVC

slid

e 11


� Video encoder needs to decide between many options denoted with parameter vector p

� Unconstrained Lagrangian Formulation:

min

pD(p) + λ ⋅R(p)

D - Distortion

R - Rate

RT - Target Rate

p - Parameter Vector

with λ controlling the rate-distortion trade-off

min

pD(p) s.t. R(p) ≤ RT

� Minimization tests the various modes in video coding[Wiegand, et al., 1996]

� Constrained Problem:

slid

e 12


[Shoham & Gersho, 1989]

slid

e 13


Next Steps in Video CodingNext Steps in Video Coding

0 100 200 300

28

30

32

34

36

38

40

Rate [kbit/s]

PSNR [dB] H.264/AVC (2003)?

slid

e 15


� New Joint Collaborative Team on Video Coding (JCT-VC)of ISO/IEC MPEG and ITU-T VCEG

� April 2010: Call for Proposals evaluated by JCT-VC• 27 proposals received• Significant video quality improvements relative to

H.264/AVC High Profile for high and low delay• JCT-VC decided to create Test Model under Consideration (TMuC)

� October 2010: Creation of first Test Model

� Early 2013: intended finalization of standard

slid

e 16


EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

8x8

0

4x8

0 10 1

2 3

4x48x4

1

08x8Types

0

16x16

0 1

8x16MB

Types

8x80 1

2 3

16x8

1

0

slid

e 17


� Division of a picture into square blocks

� Blocks are assigned to quadtrees

� Maximum block size is signalled (e.g. 64x64)

� Quadtree-based subdivision oftree block into prediction and transform blocks

� Rectangular shapes

� Geometric partitions

slid

e 18


CurrentPicture

4 Prior Decoded Picturesas Reference

∆ = 2

1. Extend motion vector by reference picture index ∆

2. Provide reference pictures at decoder side

Flexible buffering of reference pictures generalizes B pictures:• B pictures can be reference pictures (decoupling of concepts)• Reference picture for a B picture can be any prior decoded picture

∆ = 4

∆ = 1

3. In case of bi-predictive coding (B pictures): decode 2 sets of motion parameters

[Wiegand, et al., 1999]

slid

e 19


MotionMotion--Compensated InterpolationCompensated Interpolation

� Longer Interpolation Filters• In H.264/AVC: 6-tap filters + bi-linear

• In HM: 12-tap interpolation filters

� Generalized Interpolation• MOMS basis functions

• IIR pre-filtering of reference image (2-tap)

• Short FIR filters (4-taps)

� Adaptive Interpolation filters• Transmission of filter coefficients in slice header

• Transmission of an index into a list of predefined filters

� Directional interpolation filters• Low-complexity interpolation

slid

e 20


Motion Representation

� Quadtree-based splitting of blocks

� Motion Partition Merging

L

T

X

B

R

slid

e 21


Motion Vector Coding

� Adaptive Motion Vector Coding• Signaling of motion vector predictor

current block

toptop

righttopleft

left

bottomleft

slid

e 22


EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

Transform CodingTransform Coding

slid

e 23


1. Partitioning into prediction blocks (intra or inter prediction)

2. Partitioning of prediction blocks into transform blocks(specifying the transform sizes)

Partitioning of a „tree block“ into prediction blocks

Partitioning of a prediction block into transform blocks

slid

e 24


Variable Transform Block Sizes Variable Transform Block Sizes � Transform sizes range from 4x4 to 32x32

� Requirement on fast algorithms

� Additional new directional transforms proposed

slid

e 25


EntropyCoding


Motion-Compensation

MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

Intra PredictionIntra Prediction� H.264/AVC directional prediction

� New directional predictions and transforms

1

2

3456

7

8

0Q A B C D E F G HI a b c dJ e f g hK i j k lL m n o p

slid

e 26


EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

Entropy CodingEntropy Coding

slid

e 27


� Entropy coding: reversible mapping from one data representation to another, more compact representation

• Entropy is lower bound on transmission rate [Shannon, 1948]

� Problem complicated in video coding

� Video encoder generates sequence of symbols

� Sequence of symbols contains a mix of data types: control data, motion vectors, transform coefficients

slid

e 28


� Symbols in video codec correspond to a mix of data type: control data, motion vectors, transform coefficients

� Huffman coding: create a code for each data type

� Arithmetic coding: binarize data and arithmetically encode each binary symbol

� Examples1,s2 ,...,sN{ }→ b1,b2,b3 ,b4 ,...,bB{ }

Symbol Binary Symbol

0 0

1 10

2 11

slid

e 29


� Simple construction method for entropy codes [Huffman, 1951]

� Minimizes average codeword length

with constraint

� Bounds on Rate

� Example

Redundancy 17.8 %

slid

e 30


� Alternative construction method for entropy codes [Rissanen, 1976], [Pasco, 1976]

� Achieves entropy bound asymptotically for

� In video coding: context-adaptive binary arithmetic coding(CABAC), [Marpe, Schwarz, Wiegand, 2003]

Context Modeling

Binarization Probability Estimation

Coding Engine

update probability estimation

Adaptive binary arithmetic coderBottleneck !

slid

e 31


� Arithmetic coding represents probability of binary symbols

� New concept: PIPE coding [Marpe, Schwarz, Wiegand 2010]

slid

e 32


0,0

0,2

0,4

0,6

0,8

1,0

0 0,1 0,2 0,3 0,4 0,5probability p

exp

ecte

d ra

te p

er b

in [

bit

]

entropy limitapproximation by

interval discretization

probabilityinterval

intervalrepresentative

� Uniform probability distribution of binary probabilities

� With slope of binary entropy function

� Rate increase

Average Rate Increase (relative to entropy limit)

f( p ) K=1 K=2 K=4 K=8 K=12 K=16

uniform 12.47% 3.67% 1.01% 0.27% 0.12% 0.07%

slid

e 33


� Assume fixed probability for each interval

� Create combined events of binary symbols

� Probability of symbol p

� Leaf node probability is

bin sequence codeword 000 11

001 001

01 01

10 10

11 000

0.216

0.144

0.240

0.240

0.160

11

001

01

10

000p

p2

qp

pq

qq2

q3

q2p

p = 0.4q = 1− p

prob. codewords

slid

e 34


� Coding efficiency of arithmetic coding

� Complexity of Huffman coding

0.000

0.001

0.002

0.003

0.004

0.005

0.006

0 0.1 0.2 0.3 0.4 0.5

probability p

aver

age

cod

ewo

rd le

ng

th in

crea

se [

bit

s]

optimal code assumption (theoretical)

real V2V codes (with up to 65 codewords)

0.12 %

0.24 %

slid

e 35


EntropyCoding


Motion-Compensation

ControlData


MotionData

Intra/Inter

CoderControl

Decoder

MotionEstimation


InputVideoSignal



De-blockingFilter

OutputVideoSignal

InIn--Loop FilteringLoop Filtering

slid

e 36


InIn--Loop FilteringLoop Filtering

� Deblocking filter

� Adaptive Loop Filter• Wiener filter• Transmission of filter coefficients and indication for regions

slid

e 37


Coding EfficiencyCoding Efficiency

slid

e 38


� On average, current gains seem to be around 30% bit rate reduction

� Goal: 50% bit rate reduction against H.264/AVC HP at same quality

� Finalization: January 2013

Coding EfficiencyCoding Efficiency

slid

e 39


� H.264/AVC: everywhere today

� High-Efficiency Video Coding

• New project of MPEG and VCEG

• Started in April 2010

• First Test Model in Oct. 2010

• Currently roughly 30% bit rate reduction

• Goal: 50% bit rate reduction vs. H.264/AVC HP

• Finalization intended for early 2013

slid

e 40


HHI & TUB members and research associates� D. Marpe� H. Schwarz� K. Müller� P. Kauff� R. Schäfer� other members of the HHI/TUB team

Documents

Recent Advances in Video Coding.thomas Wiegand 2011