High Efficiency Video Coding (HEVC) · 2017-10-24 · High Efficiency Video Coding (HEVC) 5. The latest draft video coding standard developed by a joint team of experts from ISO/IEC

1

High Efficiency Video Coding (HEVC)

2The MPEG Vision

Three years ago in 2009, it was expected

-- Ultra-HD (e.g., 4kx2k) video will emerge

-- Mobile HD applications will become popular-- Video bitrate using current technology will go

up faster than the network infrastructure

Now, we see …

3London 2012 Olympic Games

Immersive experience in Super Hi-Vision16 times the resolution of Full HD (1080p)

Pioneered by NHK & BBC 7680x4320

4Go beyond Full HD

DVD (720x480)

SDTV (1280x720)

HDTV (1920x1080)

Digital Cinema 4K (4096x2160)

Digital Cinema 2K (2048x1080)

Super Hi-Vision / Ultra HD (7680x4320)

VIDEO RESOLUTIONCOMPARISONFROM 480P TO 4320P

5High Efficiency Video Coding (HEVC)

The latest draft video coding standard developed by ajoint team of experts from ISO/IEC MPEG and ITU-TVCEG

Goal: substantially better performance than theH.264/AVC standard, especially in coding HD and Ultra-HD video

International Standard (IS) in April 2013

6Timeline

2005-: MPEG/VCEG exploration activities

2010/01: Joint Collaborative Team (JCT-VC) Final Call-for-Proposals (CfP)

2010/04: 1st JCT-VC Meeting (Dresden, Germany) 27 proposals received

2010/10: Working Draft 1.0 (WD1.0) and HM-1.0

2013/01: Final Draft International Standard (FDIS)

2013/04: International Standard (IS)

7HEVC Version 1

Press letter of 103rd Geneva Meeting (N13253) –The next major milestone in MPEG video history isachieved“ISO/IEC JTC1/SC29/WG11 MPEG is proud to announcethe completion of the new High Efficiency Video Coding(HEVC) standard which has been promoted to FinalDraft International Standard (FDIS) status at the 103rdMPEG meeting.”

8Subjective Assessment

J. R. Ohm, G. J. Sullivan, F. Bossen, T. Wiegand, V. Baroncini, M. Wien, and J. Xu,“JCT-VC AHG report: HM subjective quality investigation (AHG22)”, JCTVC-H0022, San José, CA, Feb., 2012.

Class B (HD): -67% Class C (SD): -49%

Over -50%

HM 5.0JM 18.2

MO

S

Bit Rate [Kbps]

9New milestone since year 2003

J.-R. Ohm, G. J. Sullivan, H. Schwarz, T. K. Tan, and T. Wiegand, “Comparison of the Coding Efficiency of Video Coding Standards—Including High Efficiency Video Coding (HEVC)”, IEEE Trans. CSVT, Dec., 2012

Park Scene, 1920x1080, 24Hz

H.264/MPEG-2 (MP)

MPEG-4 (ASP)

H.263 (HLP)

H.264/MPEG-4 AVC (HP)

HEVC (MP)

YUV-

PSN

R (d

B)

42

41

40

39

38

37

36

35

34

33

32

310 2 4 6 8 10 12 14

Bitrate (Mbps)

HEVC

H.264/AVC

MPEG-4

H.263

MPEG-2

35.4%

10HEVC vs. AVC (1/2)Te

st C

ondi

tion

0 10 20 30 40 50BD-rate Saving (%)

B. Li, G. J. Sullivan, and J. Xu,“Comparison of Compression Performance of HEVC Draft 10 with AVC HighProfile,” JCTVC-M0329, Incheon, April, 2013.

HM10.0 (Main) vs. JM18.4 (High)

36.4%

ALL INTRA

RANDOM ACCESS

LOW DELAY

23%

33.5%

11HEVC vs. AVC (2/2)

Class ResolutionY BD-Rate (%)

All Intra Radom Access Low Delay

A 2500x1600 -23.6 -36.6

B 1080p -22.7 -39.8 -42.1

C 480p -19.7 -30.3 -32.7

D 240p -16.4 -28.0 -29.9

E 720p -28.8 -44.1

F 480p, 720p -28.6 -31.2 -33.8(Minus sign means coding gain)

12Encoding TimeTe

st C

ondi

tion


97%

20% 40% 60% 80% 100% 120%Encoding Time Ratio (%)

ALL INTRA

RANDOM ACCESS

LOW DELAY71%

109%


13Decoding TimeTe

st C

ondi

tion

20% 40% 60% 80% 100% 120%Decoding Time Ratio (%)


ALL INTRA

RANDOM ACCESS

LOW DELAY

107%

42%

56%


14Multi-thread Decoding

0 0.5K 1K 1.5K 2K 2.5K

Vertical Resolution

Fram

e Ra

te (f

ps)

908070605040302010

0

720p1080p

3840x2160(12Mbps)

T. Tan, Y. Suzuki, and F. Bossen, “On software complexity: decoding 4K60p content on a laptop,” JCTVC-L0098, Geneva, CH, Jan., 2013.

HM-9.0 (RA-Main)

30fps

60fps

15Few weeks later …

Samsung UNPACKED 2013 - http://www.youtube.com/watch?v=Yaw6CSaPnfk

Display 4.99 inch, 1920x1080

MPEG4, H.264, H.263, DivX, VC-1, VP8, WMV7/8, Sorenson Spark, HEVC

Codec

Camera 13 Mega Pixels

Video 1080p@30fps

1616

HEVC TOOL FEATURES

17HEVC Tool Features

Current Frame

FrameBuffer

Intra Prediction

Inter Prediction

Entropy Coding

IQ IDCT

DCT Q

Inter

Intra

-+

In-loop filter +

+

Bitstream• Asymmetric Motion Partitioning• Merged Skip / Motion Merging• Advanced MV Prediction• DCT-based Interpolation Filter

• More Directions• Pre-/Post-filtering• Direct Chroma

• Residual Quad-tree Trans.• Transform Skipping• Adaptive Coeff. Scanning

• Deblocking Filter• Sample Adaptive Offset

• CABAC• Tiles• Wavefront

18Coding Unit (CU)

Basic unit for coding, conceptually similar tomacroblock but now can be of variable size

MB MB16

16

64

64

CUCU

CU CU

CU CU

CU CU

CTU0 CTU1

H.264/AVC HEVC

19Prediction Unit (PU)

PU PU PU

PU

PU

PU PU

PU PU

PU

PUPU

PU

PU PUPU PU

16x16MB

CU (Only for SCU) Asymmetric

Sub-MB Partition

20Transform Unit (TU)

Residual Quad-tree Transform (RQT)Transform can cross PU boundaries

TU

TU TU

TU TU

TU TUCU RQT TUTU TU

TU TU

16x16MB

TU TU

TU TU

8

8

PU Aligned

May beskipped if 4x4

TU TU TU TU 4

4

TU TU TU TU

TU TU TU TU

TU TU TU TU

21Intra Prediction

More directions (up to 33)Adaptive pre-filtering of reference pixelsBoundary smoothing for DC/Ver./Hor. modesDirect mode for Chroma

TR

LBLB

TR

1: Planar0: DC

Reference Sample Mean

17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2

18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34Vertical

Horizontal

22DCT-based Interpolation Filter

P-3 P-2 P-1 P0 P1 P2 P3 P4Pa

Spatial Domain Samples{P-3, P-2, …, P4}

DCT Domain Coefficients{C-3, C-2, …, C4}

Forward DCT

M Integer-pelsFraction-pel

Inverse DCT

DCT Coefficients

( )2 1

1

0 2 1 2cos

2 4a k

M

k

k M aM

π−

=

− + = +

∑Cp C

M Integer-pels

23Motion Merging (1/2)

Optional

5 Candidatesat most

24Motion Merging (2/2)

mv0

mv2 mv3

mv6 mv7

mv1

mv4 mv5

Current PU 0

Current PU1

mv0

mv2 mv3

mv6 mv7

mv1

mv4 mv5

mv7

mv5

Before Merging After Merging

Irregular Motion PartitionsMerge w/ Nearby Motion

25Temporal Merge Candidate Derivation

Current

Current

3Co-located

L0Ref0

L0 L1Ref0

L1

1

2

26Advanced Motion Vector Prediction

B1B2

A1 Current PU

T1

T0A0Find

Firs

t Av

aila

ble

Find First Available

B0

If LCU boundary, exclude A0 & T0

2 Candidatesat most

1

2

3

Optional

27

Operable in 4x4 TUsSwitchable with DST (intra) / DCT (inter)

Transform Skipping

(QP37, 608.4Kbps, 34.8dB)

TS EnabledTS Disabled

(QP36, 600.8Kbps, 36.1dB)

28Significance Map Scanning

TU Size Prediction Type Scanning Order4x4, 8x8 Intra (Vertical, Horizontal) Vertical, HorizontalAll Intra, Inter 4x4 Sub-diagonal

4

4

4x4 Sub-diagonalHorizontalVertical

29Sample Adaptive Offset (1/3)

Band Offset (BO): intensity-based sample classificationEdge Offset (EO): edge-based sample classification

0 MAX4 Bands[BO]

[EO]C C

C C

LCUs in a Frame

EO Off Off EO

BO BO BO

BO BO BO

EO

EO

EO EO EOOff


Pixel Indexx-1 x x+1

Pix

el L

evel

x-1 x x+1 x-1 x x+1

x-1 x x+1

Pix

el L

evel

x-1 x x+1Pixel Index

x-1 x x+1

I. Positive Edge Offset

II. Negative Edge Offset


SAO Disabled SAO Enabled

(Frame15, 32.6dB, 3.0Mbps) (Frame15, 32.7dB, 3.1Mbps)

32Parallel-friendly Design

LiveContent

Chunk encodingusing 16 cores

Chunk re-ordering2 entry points

SequentialBitstream

Transcoding toWPP

Chunksre-ordering

G. Clare, F. Henry, and S. Pateux,“Wavefront and Cabac Flush: Different Degrees of Parallelism Without Transcoding”, JCTVC-F275, Torino, IT, July, 2011.

33Wavefront Parallel Processing (WPP)

FlushCABAC States

[n Waves in a Slice]

Parsing Decompressing


DecodedPixels

Dependent

Parallel Parallel

SyntaxElements

DecodedPixels

DependentSyntax

Elements

Dependent Dependent

Pass CABAC states to next Wave

34Tiles

Parsing


DecodedPixels

DecodedPixels

[4 Tiles in a Slice/Frame]

Parallel Parallel

SyntaxElements

Decompressing

Prediction can’t cross Tiles

SyntaxElements

35Parallel Merge Group

CUs/PUs within Groupm,n

PUs/CUs within a group run in parallel

Left Inferable

Groupm-1,n-1 Groupm+1,n-1Groupm,n-1

Groupm-1,n

Above Inferable

36Profiles & Tool Summary

Tool \ Profile Main Still Picture Main Main 10Bit Depth 8 8 8, 9, 10

CU Size 16x16~64x64

PU Partition Symmetric Symmetric, AsymmetricTU Partition Residual Quad-tree Transform

MV Prediction - AMVP, MRG, MRG-Skip

Interpolation Filter - DCT-IF

Intra Prediction DC, Planar, 33 Directions, DMTransform DCT 4x4~32x32, Skip 4x4, DST 4x4 (Intra)In-loop Filter De-blocking, SAO

Entropy Coding CABAC (Tiles, Wavefront)

* Support picture resolution ranging from 128x96 to 8192x4320.

37Information

http://phenix.int-evry.fr/jct/(Website)

http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc(Subscribe)

https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/(SVN, Software Manual JCTVC-J0470)

Text Specification Draft 9 (JCTVC-K1003)

http://phenix.int-evry.fr/jct/

http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc

https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/

3838

RANGE EXTENSIONS TO HEVC

39Range Extensions

Goal: minimum changes to HEVC V1 to support non-4:2:0 chroma formats and bit depths beyond 8 bits andto improve lossless coding

Finalizes in July 2014

4040

RANGE EXTENSIONS TOOL FEATURES

41RExt Tool Features

Current Frame

FrameBuffer

Intra Prediction

Inter Prediction

Entropy Coding

IQ IDCT

DCT Q

Inter

Intra

-+

In-loop filter +

+

Bitstream

• Residual DPCM• Large TS block• Residual rotation• Cross component prediction

• One separate CABAC for significant map coding for TS block

42Residual DPCM

Second-order residual predictionVertical/horizontal predictionImplicit for Intra, explicit for Inter

Residual Block

a-p b-pp c-p d-p

sub. sub. sub.

Parallel Encoding

Parallel Decoding

Residual DPCM

a-p b-ap c-b d-c

add add add

Processed at parsing stage

Short-distanceresidual prediction

Reference Samples

43Intra Residual Rotation

Reverse scan order for intra residual blocks when transform skip is in use

Much similar to the pattern of

coefficients

Residual Block Rotated Block

tend to be

zero

tend to be

non-zero

tend to be

non-zero

tend to be

zeroRotate

180 degree

44Cross Component Prediction

Predict chroma residuals from luma residualsApply to intra DM and inter (if luma residual exists)Adapt α∈{-8,-4,…,8} at TU level for U/V separately

Chroma

Recon. Chroma Residual

Luma𝜶𝜶𝟖𝟖∗

Recon. LumaResidual

2nd-order Chroma

Decoded 2nd-order Chroma Residual

45Common Test Conditions (AHG5)

Coding Structure QP Range (Lossy)

• All Intra• Random access• Low delay B

• MT (22, 27, 32, 37)• HT (17, 22, 27, 32)• SHT (12,17, 22, 27)

Test Sequence

Color Format Bit Depth Resolution (# of Seqs)

RGB 4:4:4 8, 10,12 2560x1600 (1), 1920x1080 (7)

YUV 4:4:4 10 2560x1600 (1), 1920x1080 (6)

YUV 4:2:2 10 2560x1600 (1), 1920x1080 (6)

46HM-14.0+RExt-7.0 vs. JM-18.6

Format

Y BD-Rate (%)

All Intra Radom Access Low Delay B

MT HT SHT MT HT MT HT

RGB 4:4:4 -32.6 -25.1 -19.6 -36.0 -25.1 -36.2 -25.0

YUV 4:4:4 -22.4 -19.0 -14.8 -35.1 -29.8 -39.8 -32.9

YUV 4:2:2 -19.7 -15.8 -11.7 -30.2 -27.8 -35.9 -31.0

B. Li, J. Xu, and Gary. J. Sullivan, “Comparison of Compression Performance of HEVC 4:4:4 RangeExtensions Test Model 7 and HEVC Screen Content Coding Extensions Test Model 1 with AVC High 4:4:4Predictive profile,” JCTVC-R0101, Sapporo, June, 2014.

47Information


http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc(Subscribe)

https://hevc.hhi.fraunhofer.de/svn/svn_SHVCSoftware/(SVN)

Edition 2 Draft Text of High Efficiency Video Coding (HEVC), Including Format Range (RExt), Scalability (SHVC), and Multi-View (MV-HEVC) Extensions (JCTVC-R1013)



https://hevc.hhi.fraunhofer.de/svn/svn_SHVCSoftware/

4848

SCREEN CONTENT CODING (SCC)

49Screen Content Coding (SCC)

Screen content video, usually a mixture of text, graphicsand nature scene images, exhibits very different characteristics from camera-captured video

50Screen Sharing

Share & Play Together by TM

51Cloud Gaming

52Camera-captured vs. Screen Contents

Camera-capture video– Continuous-tone with camera noise– Smooth edges, complicated texture, thick lines with rich

colors

Screen content video– Discontinuous-tone with less/no noise– Sharp edges, simple shapes, thin lines (e.g. 1-pixel wide) with

few colors

53

Missing details and annoying artifacts

Why SCC is Challenging?

SCM-1.0Original

(All Intra, QP27, 2.9Mbps, 42.5dB)

54Requirements

From the discussions on the reflector, SCC shouldaddress– 4:4:4 chroma sampling format (RGB, YUV)– Up to 10-bit for each color component– Low latency/complexity on encoder & decoder– Temporal stability– Subjectively lossless– Mathematically lossless for some application– Low bitrate for discontinuous-tone contents

55Screen Content Coding

Future HEVC extensions in coding screen content targeting at coding of 4:4:4 8-bit sequences

2014/01: Final Call-for-Proposals

2014/03: Evaluation of Proposals

2014/04: SCC Test Model 1.0

2014/07: SCC Test Model 2.0

2015/02: Proposed Draft Amendment (PDAM)

2015/10: Final Draft Amendment (FDAM)

56SCC Tool Features

Current Frame

FrameBuffer

Intra Prediction

Inter Prediction

Entropy Coding

IQ IDCT

DCT Q

Inter

Intra

-+

In-loop filter +

+

Bitstream

• IBC mode• Palette mode

• Adaptive color transform

57Intra Block Copy (IBC)

Similar to inter motion compensation, except– using the current frame as reference– referring to un-deblocked samples

58Palette Mode

Adaptive color quantization for PCM pixelsRepresent a block by major colors & an index mapSpecify the major color a pixel mapped to by index

[Block Samples]

Intensity Value

0 1 2 3 4 5 6 7 8 9

Prob

abilit

y

p(x)

x

Index

Major Color

[Index Map]

0000

0000

11 1

1

111 1

Analysis Quantization

0 1

59Other tools

Palette mode– Major color coding (e.g. stuffing, merging,…)– Index map coding (e.g. transition mode, dictionary,…)

Intra string matching– PCM sample coding– Index coding

Intra line copy

60Major Color Coding (1/2)

Major color table (MCT) propagation – Infer major colors from left or above CU (or previously decode

ones when unavailable)

Major color merge (left or above CU)Long-term palette prediction– Signal a long-term palette

at the slice header

Triplet palette coding– Three component-wise palette

Current CU

AboveReference

CU

LeftReference

CU

MCT propagation

MCT propagation

61Major Color Coding (2/2)

Palette stuffing (predictor propagation)

62Index Map Coding (1/2)

Find a match inside current CU– Run mode (copy-left, copy-above)– Transition copy

63Index Map Coding (2/2)

Search a match outside current CU– 1-D/2-D String matching– Re-quantization required on reconstructed pixels

Search Range

Search Range

64Dictionary-based String Matching

Patterns repeat frequently within text and graphics regions (e.g. text, icons, lines etc.)

65Reconstruction-based String Matching

Preserve block structureHybrid 1-D/2-D string matching for index codingMore flexible than IBC, higher data dependency

66Intra Line Copy

Split a PU equally into 1xN/Nx1 linesLower data dependency than string matching; more flexible than PU-based IBC

Search Range

2N. . .

PU0

Line(size=1)

PU1

Search Range

BV0, PU0

BV0, PU1

BV1, PU0. . .

67Common Test Conditions (SCC)

Coding Structure QP Range

• All Intra (AI)• Random access (RA)• Low delay B (LB)

• Lossy: 22, 27, 32, 37• Lossless

Test Sequence

Category Resolution (# of Seqs)

Text and graphics with motion 1920x1080 (6), 1280x720 (8)

Mixed content 2560x1440 (4), 1920x1080 (2)

Animation 1280x720 (2)

Camera-captured content 1920x1080 (4)

68SCM-1.0 vs. JM-18.6

CodingStructure

Y BD-Rate (%)Text, Graphics Mixed Content Anima. Camera

1080p 720p 1440p 1080p 720p 1080pRGB Sequences

AI -85.8 -71.6 -69.5 -74.9 -36.8 -45.1RA -78.8 -65.3 -60.5 -70.1 -39.6 -49.8LB -78.4 -61.3 -55.3 -62.0 -42.9 -46.6

YUV SequencesAI -77.2 -56.6 -54.0 -63.8 -23.6 -26.9RA -69.4 -54.0 -48.3 -62.0 -32.6 -40.1LB -68.8 -52.8 -45.9 -56.1 -39.1 -40.0

B. Li, J. Xu, and Gary. J. Sullivan, “Comparison of Compression Performance of HEVC 4:4:4 RangeExtensions Test Model 7 and HEVC Screen Content Coding Extensions Test Model 1 with AVC High 4:4:4Predictive profile,” JCTVC-R0101, Sapporo, June, 2014.

69Information


http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc(Subscribe - AHG7: Screen Content Coding)

https://hevc.hhi.fraunhofer.de/svn/svn_SHVCSoftware/(SVN)

HEVC Screen Content Coding Draft Text 1(JCTVC-R1005)



https://hevc.hhi.fraunhofer.de/svn/svn_SHVCSoftware/

70TCSVT Special Issue on HEVC

IEEE Transaction on Circuits and Systems for Video Technology (TCSVT), vol. 22, no. 12, Dec., 2012

Special Section: HEVC Standard– G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the High

Efficiency Video Coding (HEVC) Standard”– J.-R. Ohm, G. J. Sullivan, H.Schwarz, T. K. Tan, and T. Wiegand,

“Comparison of the Coding Efficiency of Video Coding Standards—Including High Efficiency Video Coding (HEVC)”

Special Issue: Emerging Research and Standards in Next Generation Video Coding (HEVC)

71

ThankYou ~

Documents

High Efficiency Video Coding (HEVC) · 2017-10-24 · High Efficiency Video Coding (HEVC) 5. The latest draft video coding standard developed by a joint team of experts from ISO/IEC