Topics in Signal ProcessingCourse ID: EE5359

Project Proposal: HEVC Lossless Coding

and Improvements

SUBMITTED BY: SUJATHA GOPALAKRISHNAN STUDENT ID: 1001024145

Table of ContentsACRONYMS:..................................................................................................................................3HEVC..............................................................................................................................................5Block Diagram HEVC.....................................................................................................................5HEVC Lossless Coding...................................................................................................................6Block Diagram HEVC Lossless Coding..........................................................................................7Basic Definitions.............................................................................................................................7

H.264[1][2]..................................................................................................................................7Inter Frame...................................................................................................................................7Intra Frame...................................................................................................................................7Loop Filters..................................................................................................................................7

De-blocking Filter....................................................................................................................8Sample Adaptive Offset...........................................................................................................8

Block-Based Angular Intra Prediction.........................................................................................8Sample-Based Angular Intra Prediction......................................................................................8Coding Tools...............................................................................................................................8

LCU/CTU................................................................................................................................9Parallel Processing...................................................................................................................9Entropy Coding........................................................................................................................9Motion Estimation...................................................................................................................9Motion Compensation...........................................................................................................10

Algorithm of Sample Based Angular Intra Prediction..................................................................10Proposed Project............................................................................................................................12Projected Enhancement in the Project...........................................................................................12References......................................................................................................................................12

ACRONYMS:

2D Two dimension3D Three dimensionACM MoVid

Association for Computer Machinery Mobile Video

AHG Ad Hoc GroupsAIF Adaptive Interpolation FilterALF Adaptive Loop FilterAMVP Advanced Motion Vector PredictionAPIF Adaptive Pre-Interpolation FilterASIC Application-Specific Integrated CircuitAVC Advanced Video CodingAVS Audio Video StandardBBC British Broadcasting CorporationBD Bjontegaard DistortionBL Base Layerbpp Bits per pixelBS Boundary StrengthCU Coding UnitCI Confidence IntervalCABAC Context Adaptive Binary Arithmetic CodingCPU Central Processing UnitCRA Clean Random AccessCSVT Circuits and Systems for Video TechnologyCU Coding UnitDCT Discrete Cosine TransformDCTIF Discrete Cosine Transform Interpolation FiltersDDCT Directional Discrete Cosine TransformDSP Digital Signal ProcessingDST Digital Sine TransformEC Error ConcealmentFIR Finite Impulse ResponseFPGA Field Programmable Gate Arrayfps Frames per secondGPU Graphics Processing UnitHDR High Definition RangeHEVC High efficiency video codingHEVStream High Efficiency Video StreamHTTP Hyper Text Transfer ProtocolICIEA IEEE Conference on Industrial Electronics and ApplicationsIEEE Institute of Electrical and Electronics EngineersINTDCT Integer Discrete Cosine Transformintra HE Intra high efficiencyIPTV Internet Protocol Television

IS & T Information Systems and TechnologyISO International Organization for StandardizationITU-T Telecommunication Standardization Sector of the International

Telecommunications UnionIVMSP Image, Video, and Multidimensional Signal ProcessingJCTVC Joint Collaborative Team on Video CodingJM Joint ModelJPEG Joint Photographic Experts GroupJPEG-XR JPEG extended rangeJSVM Joint Scalable Video ModelJTC Joint Technical CommitteeLR Low ResolutionMbit/s Megabit per secondMC Motion CompensationMDDCT Modified Directional Discrete Cosine TransformMDDT Mode-Dependent Directional TransformME Motion EstimationMJPEG Motion JPEGMMSP Multimedia Signal ProcessingMPEG Moving Picture Experts GroupMpixel MegapixelMpm Most Probable ModesMV Motion VectorNAB National Association of BroadcastersNAL Network Abstraction LayerPCM Pulse Code ModulationPSNR Peak-to-peak signal to noise ratioPU Prediction UnitQP Quantizer parameterRD Rate DistortionRDOQ Rate-distortion optimized quantizationRDPCM Residual Differential Pulse Code ModulationROT Rotational TransformRTP Real-time Transport ProtocolSAO Sample adaptive offsetSHVC Scalable High Efficiency Video CodingSVC Scalable Video CodingSELC Sample based weighted prediction for Enhancement Layer CodingSIP Signal and Image ProcessingSSVC Spatially Scalable Video CodingTB Transform BlockTU Transform Unit

HEVC High Efficiency Video Coding (HEVC) [1] [2] is a video compression standard, a

successor to H.264/MPEG-4 AVC [22]. HEVC is said to double the data compression ratio compared to H.264/MPEG-4 AVC [1] at the same level of video quality [2].

The design of most video coding standards is primarily aimed at having the highest coding efficiency

HEVC benefits from the use of larger Coding Tree Unit (CTU) sizes. The HEVC video coding layer uses the same "hybrid" approach used in all modern video

standards, starting from H.261 [1], in that it uses inter-/intra-picture prediction and 2D transform coding.

The main goal of the HEVC standardization effort is to enable significantly improved compression performance relative to existing standards, in the range of 50% bit rate reduction for equal perceptual video quality [10] [11].

Block Diagram HEVC

Figure 1: HEVC Encoder [2]

Figure 2: HEVC Decoder Block Diagram [3]

Some differences in HEVC [1][2] are coding tree units instead of macro blocks, single entropy coding methods-Context Adaptive Binary Arithmetic Coding (CABAC) [15] method and features like tiles , wave front parallel processing and dependent slices to enhance parallel processing.

HEVC Lossless Coding The lossless coding mode of HEVC main profile bypasses transform quantization and in-

loop filters as shown in the fig.2 [4][19]. Comparing it with non-lossless coding mode, it has smallest quantization parameter

value. Lossless coding mode provides perfect fidelity and average bit rate reduction. Outperforms the existing lossless compression solution such as JPEG-2000 [22] and

JPEG-LS [22]. It employs Sample Angular-based Intra-Prediction (SAP) [4].

Same prediction mode signaling method. Same interpolation method of HEVC. Uses adjacent neighbors as reference shown in fig.7.

Block Diagram HEVC Lossless Coding

Figure 3: HEVC lossless Algorithm Block Diagram [4]

The blocks that are marked bypass are not being used when implementing a HEVC [1][2] lossless algorithm, thereby providing average bit rate reduction.

Basic DefinitionsH.264 [1][2]H.264/MPEG-4 AVC [1] [2] [22] is a block-oriented, motion-compensation based video compression standard.

Inter FrameAn inter frame is a frame in a video compression stream which is expressed in terms of one or more neighboring frames. The "inter" part of the term refers to the use of Inter frame prediction.

Intra FrameThe term intra-frame refers to the various lossless and lossy compression techniques that happens relative to information which is contained only within the current frame and not relative to any other frame in the video sequence.

Loop FiltersHEVC [1] specifies two loop filters that are applied sequentially; the de-blocking filter (DBF) [4] applied first and the sample adaptive offset (SAO) filter applied afterwards. Both loop filters

are applied in the inter-picture prediction loop, i.e. the filtered image is stored in the decoded picture buffer (DPB) as a reference for inter-picture prediction.

De-blocking Filter The DBF is similar to the one used by H.264/MPEG-4 AVC [1] [2], but with a simpler

design and better support for parallel processing. DBF first apply horizontal filtering for vertical edges to the picture and only after that

does, it apply vertical filtering for horizontal edges to the picture. This allows for multiple parallel threads to be used for the DBF [1].

Sample Adaptive OffsetThe SAO filter is applied after the DBF and is designed to allow for better reconstruction of the original signal amplitudes by applying offsets stored in a lookup table in the bit stream.

Block-Based Angular Intra PredictionIt is a method of computing predicted samples produced by PU when lossless coding is not enabled. It is defined to exploit spatial sample redundancy in intra coded CUs. As shown in the fig.4, a total of 33 angles are defined for the angular prediction, which can be categorized into two classes: vertical and horizontal angular predictions as illustrated [14].

Figure 4: Block Based Angular Intra Prediction in HEVC [4]

Sample-Based Angular Intra PredictionIt is a method of computing predicted samples produced by PU when lossless coding is enabled. It is explained detail in the following.

Coding ToolsCoding efficiency is the ability to encode video at the lowest possible bit rate while maintaining a certain level of video quality. This could be achieved with the following coding tools.

LCU/CTUCoding tree unit (CTU) is the basic processing unit of the HEVC video standard and conceptually corresponds in structure to macroblock units, which were used in several previous video standards. CTU is also referred to as largest coding unit (LCU) [1] [2]. In the HEVC, one frame is divided into a series of non-overlapped Coding Tree Unit (CTU) [9] [12].

Figure 5: Division of a CTB into CBs and transform blocks TB [2]

Parallel ProcessingA picture is divided into tiles. Main purpose of these tiles is that, they can be decoded /encoded individually in a simultaneous way called parallel processing. Parallel computing is basically a technique in which multiple computation tasks are assigned to multiple processes and process the job simultaneously. The basic approach for parallel processing is to break the task into multiple smaller tasks and further assign each task to each of the thread which performs required operations in parallel. Parallelization can sometimes get complicated due to race conditions, data dependency, synchronization and communication among different threads [13].

Entropy CodingHEVC uses a context-adaptive binary arithmetic coding (CABAC) algorithm that is fundamentally similar to CABAC in H.264/MPEG-4 AVC. CABAC is the only entropy encoder method that is allowed in HEVC while there are two entropy encoder methods allowed by H.264/MPEG-4 AVC. CABAC and the entropy coding of transform coefficients in HEVC are designed for a higher throughput than H.264 while maintaining higher compression efficiency for larger transform block sizes relative to simple extensions [15][16]. These techniques include reducing context coded bins, grouping bypass bins, grouping bins with the same context, reducing context selection dependencies, reducing total bins, and reducing parsing dependencies. It also describes reductions to memory requirements that benefit both throughput and implementation costs.

Motion EstimationMotion estimation [5] is an essential process in many video coding standards like MPEG-2, H.264/AVC and HEVC [1] [2]. Motion estimation has been used at the encoder. Motion

Estimation itself consumes more than 50% coding complexity or time to encode. To reduce the computation time, many fast motion estimation Algorithms were proposed and implemented [5]. HEVC allows for two MV modes which are Advanced Motion Vector Prediction (AMVP) and merge mode. AMVP uses data from the reference picture and can also use data from adjacent prediction blocks. The merge mode allows for the MVs to be inherited from neighboring prediction blocks. Merge mode in HEVC is similar to "skipped" and "direct" motion inference modes in H.264.

Motion estimation process in HEVC consumes more than 50% coding complexity or time to encode with equal perceptual quality [6] [7]. Many block based motion estimation algorithms [8] [9] are proposed and also implemented to reduce the computation time.

Figure 6: Illustration of Motion Estimation process [5]

Motion CompensationThe interpolation of fractional luma sample positions HEVC uses separable application of one-dimensional half-sample interpolation, with an 8-tap filter or quarter-sample interpolation with a 7-tap filter [5]. While H.264/MPEG-4 AVC[1][2] uses a two-stage process that first derives values at half-sample positions, using separable one-dimensional 6-tap interpolation followed by integer rounding; then applies linear interpolation between values at nearby half-sample positions to generate values at quarter-sample positions. HEVC has improved precision due to the longer interpolation filter and the elimination of the intermediate rounding error. As in H.264/MPEG-4 AVC [1] [2], a scaling and offset operation may be applied to the prediction signal(s) in a manner known as weighted prediction [1].

Algorithm of Sample Based Angular Intra PredictionThe SAP [4] is designed to better exploit the spatial redundancy in the lossless coding mode by generating intra prediction samples from adjacent neighbors. The design principle here is very similar to the sample-based DPCM in [21][4] H.264/MPEG-4 AVC [20][4] lossless coding, but SAP[4] is fully harmonized with the HEVC block-based angular intra prediction, and can be applied to all the angular intra prediction modes specified in HEVC [4].

As shown in the fig.7 SAP is performed sample by sample. The adjacent neighboring samples,   of the current sample   in the current PU are used for prediction. That is, the reference samples used for prediction are not limited to those boundary reference samples from the left and upper neighboring PUs. The SAP has to be processed in a predefined order to ensure the availability of these adjacent neighbors for prediction.

Figure 7: Algorithm of SAP [4]

Proposed Project

The aim of this proposal is to analyze HEVC lossless coding. Hence if a scheme is designed that will reduce the computation time using HEVC lossless coding it will further increase its performance and reduce encoding time without much increase in complexity [21].

The simulation will be conducted using HM Software 16.0 [18] [26], with different video sequences [29], search range, block sizes and number of frames using GPU multicore computing. Bit rate differences, Compression ratio, Distribution of intra-coded ratio (ratio between intra-coded and coded blocks), and results based on it, will be calculated for different algorithms using various search patterns.

Projected Enhancement in the Project

Performance analysis of HEVC Lossless Mode using HM16.0 [18] software. Performance analysis of SAP [4] using HM16.0 [18] software. Complete Study of HEVC Lossless Coding [4].

References[1] G.J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Trans. CSVT, vol. 22, pp.1649-1668, Dec 2012.

[2] G.J. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, vol.7, pp.1001-1016, Dec 2013.

[3] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC) document JCTVC- J0292r1, July 2012.

[4] M. Zhou et al, “HEVC lossless coding and improvements”, IEEE Trans. CSVT, vol.22, pp.1839-1843, Dec 2013.

[5] N. Purnachand et al, "Fast Motion Estimation Algorithm for HEVC", IEEE Second International Conference on Consumer Electronics-Berlin (ICCE-Berlin), vol.11, pp.34-37, Sep 2012.

[6] P. Hanhart et al, “ Subjective quality evaluation of the upcoming HEVC video compression standard”, SPIE Optical Engineering+ Applications. International Society for Optics and Photonics, vol. 8499, pp.84990v-84990v, Aug 2012.

[7] M. Horowitz et al, “Informal subjective quality comparison of video compression performance of the HEVC and H.264/MPEG - 4 AVC standards for low delay applications”, SPIE Optical Engineering+ Applications. International Society for Optics and Photonics, vol.84990, pp.84990w-84990w, Aug 2012.

[8] A. Abdelazim, W. Masri and B. Noaman., "Motion estimation optimization tools for the emerging high efficiency video coding (HEVC)", IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, vol. 9029, pp. 902905-902905, Feb 17 2014.

[9] M. Jakubowski and G. Pastuszak, “Block-based motion estimation algorithms-a survey”, Journal of Opto-Electronics Review, vol. 21, pp.86-102, Mar 2013.

[10] B. Bross et al, “High Efficiency Video Coding (HEVC) Text Specification Draft 10”, Document JCTVC-L1003, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC), Jan. 2013, available on http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=7243

[11] J. Ascenso et al, "Improving Frame Interpolation with Spatial Motion Smoothing for Pixel Domain Distributed Video Coding", 5th EURASIP Conference on Speech and Image Processing, Multimedia Communications and Services, pp.1-6, Smolenice, Slovak Republic, July 2005. 

[12] X. Wang et al, “Paralleling Variable Block Size Motion Estimation of HEVC on Multicore CPU plus GPU platform”, IEEE International Conference on Image Processing (ICIP), vol.22, pp. 1836-1839, Sep. 2013.

[13] Introduction to parallel computing https://computing.llnl.gov/tutorials/parallel_comp/#Whatis

[14] L. Zhao et al, “Group-Based Fast mode decision algorithm for intra prediction in HEVC”, IEEE Eighth international Conference on Signal Image Technology and Internet based Systems. Article no.6115979, pp. 225-229, Nov 2011.

[15] V. Sze and M. Budagavi, "High Throughput CABAC Entropy Coding in HEVC", IEEE Transactions on Circuits and Systems for Video Technology, vol.22, no.12, pp.1778-1791, Dec 2012.

[16] T.Nguyen et al, "Transform Coding Techniques in HEVC", IEEE Journal of Selected Topics in Signal Processing, vol.7, pp.978–989, Dec 2013.

[17] HEVC tutorial by I.E.G. Richardson: http://www.vcodex.com/h265.html

[18] HEVC Reference Software HM16.0. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.0rc1/

[19] B. Bross et al,“High Efficiency Video Coding (HEVC)Text Specification Draft 8”, JCT-VC document, JCTVC-J1003, Stockholm, Sweden, Jul. 2012.http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=6465

[20] Joint Video Team, “Advanced Video Coding for Generic Audiovisual Services”, ITU-T Rec. H.264 and ISO/IEC, 14496-10 (MPEG-4) AVC, pp.H.100-H.869, Feb 2014.

[21] Y.L. Lee et al, "Improved lossless intra coding for H.264/MPEG-4 AVC", IEEE Trans. Image Process., vol.15, no.9, pp.2610-2615, Sep 2006. [22]K.R. Rao, D.N. Kim and J.J Hwang, “High Efficiency Video Coding (HEVC) Revised/Updated Chapter from the book Video Coding Standards”–Springer 2014.

[23] ITU-T website: http://www.itu.int/ITU-T/index.html

[24] JCT-VC documents are publicly available at http://ftp3.itu.ch/av-arch/jctvc-site and http://phenix.it-sudparis.eu/jct/

[25] V.Sze, M.Budagavi, and G.J. Sullivan “High Efficiency Video Coding (HEVC)” Springer, 2014.

[26] Software reference manual for HM:https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/branches/HM-9.2-dev/doc/software-manual.pdf         

[27] M. Wien, “High efficiency video coding: Tools and specification”, Springer, 2015.

[28] I.E. Richardson, “Coding video: A practical guide to HEVC and beyond”, Wiley, 11 May 2015

[29] Video Sequences:http://ultravideo.cs.tut.fi/