68
Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate Student The University of Texas at Arlington Advisor Dr. K. R. Rao, EE Dept, UTA Committee Members Dr. W. Alan Davis, EE Dept, UTA Dr Kambiz Alavi, EE Dept, UTA

Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Embed Size (px)

Citation preview

Page 1: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Implementation and Analysis of Directional Discrete Cosine

Transform in H.264 for Baseline Profile

Shreyanka Subbarayappa

Electrical Engineering Graduate Student

The University of Texas at Arlington

Advisor

Dr. K. R. Rao, EE Dept, UTA

Committee Members

Dr. W. Alan Davis, EE Dept, UTA

Dr Kambiz Alavi, EE Dept, UTA

Page 2: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

AgendaAgendaIntroduction to the field of researchMotivation for the researchOverview of H.264Overview of DDCTImage Quality measuresH.264 JM 18.0 settingsExperimental ResultsConclusionsFuture workReferences

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 2

Page 3: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Introduction to the field of Introduction to the field of researchresearch

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 3

Page 4: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

IntroductionIntroduction Importance of video Need for compression

◦ High bandwidth requirements◦ Remove inherent redundancy

Need for standardization◦ Ensures interoperability

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 4

Year

Coding

Efficiency

Network

awareness

Complexity20052005

20102010

19991999

19941994

MPEG4MPEG4

H.264H.264

19921992MPEG1MPEG1

Video Conferencing

H.26H.2633

20032003

Mobile Phone

Hand PC

Mobile TV

SVCHDTV

MPEG2MPEG2

H.265/HECH.265/HEC/ NGVC/ NGVC

VC-1

NEED FOR IMAGE OR

VIDEO COMPRESION

2011-2013

Page 5: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Lossless or Lossy Lossless or Lossy CompressionCompression

Lossless compression

◦ There is no information loss, and the image can be reconstructed exactly the same as the original

◦ Applications: Medical imagery, Archiving

Lossy compression

◦ Information loss is tolerable.

◦ Applications: commercial distribution (DVD) and rate constrained environment where lossless methods cannot provide enough compression ratio

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 5

Page 6: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Motivation for the researchMotivation for the research

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 6

Page 7: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Motivation for implementing DDCT in Motivation for implementing DDCT in H.264H.264

Choice of codecs◦ Prevalence of H.264

Need for DDCT in H.264◦ New concept in the

transform domain◦ Better coding gain◦ Better image quality◦ Implemented in the other

upcoming standards like H.265

◦ Larger applications for H.264 in communication fields, data storage and streaming.

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 .7

Broadcast

Streaming

Content Server

Internet

Link

Mobile

Storage

H.264

ISO media file format

Page 8: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Overview of H.264Overview of H.264

April 16, 2012Implementation and Analysis of Directional Discrete Cosine Transform in H.264 8

Page 9: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

H.264: OverviewH.264: Overview

Latest block-oriented motion-compensation-based codec.

Good video quality at substantially lower bit rates.

Better rate-distortion performance and compression efficiency than MPEG-2 [42].

Simple syntax specifications, very flexible.Network friendly.Wide variety of applications such as video

broadcasting, video streaming, video conferencing, D-Cinema, HDTV.

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 9

Page 10: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

H.264 – Encoder [1]H.264 – Encoder [1]

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 10

Page 11: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

H.264 – Decoder [1]H.264 – Decoder [1]

April 16, 2012Implementation and Analysis of Directional Discrete

Cosine Transform in H.264 11

Page 12: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Profiles in H.264 [1]Profiles in H.264 [1]

April 16, 2012Implementation and Analysis of Directional Discrete

Cosine Transform in H.264 12

Page 13: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

13 Implementation and Analysis of Directional Discrete

Cosine Transform in H.264April 16, 2012

Tools introduced in FRExts and their classification under the new high profiles

Page 14: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Overview of D-DCTOverview of D-DCT

April 16, 2012Implementation and Analysis of Directional Discrete

Cosine Transform in H.264 14

Page 15: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

OutlineOutline

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 15

Problem:- To replace the DCT-like transform for intra prediction residuals in AVC and the associated zigzag scan pattern-Solution: Directional Discrete Cosine Transform (DDCT)

DDCT:- The transforms- Properties

Implementation - Transform- Quantization-Scanning pattern

Complexity- Computation - Memory

Page 16: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Conventional DCT [3]Conventional DCT [3]

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 16

- The 2-D discrete cosine transform (DCT) of a square or a rectangular block shape is used for almost all block-based transform schemes for image and video coding.

- Implemented separately through two 1-D transforms, one along the vertical direction and another along the horizontal direction.- The conventional DCT seems to be the best choice for image blocks in which vertical and/or horizontal edges are dominating.

+

(M 1D-DCT s of Length N)

Page 17: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 17

Forward 2D DCT (NXM) [3]

Inverse 2D DCT (NXM) [3]

x(n,m) = Samples in the 2D data domain.

XC2 (k, l) = Coefficients in the 2D-DCT domain

Limitations of conventional DCT

• It is not very efficient when the conventional DCT is applied to an image block in which other directional edges dominate.• When the first 1-D DCT (vertical or horizontal) is applied, the nonzero coefficients are not well aligned across different columns (or rows). Consequently, the second 1-D DCT may produce more nonzero coefficients

Page 18: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Modes Of DDCT [4]Modes Of DDCT [4]

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 18

Mode 0 – VerticalMode 1 – HorizontalMode 2 – DC (Ignored for DDCT modes)Mode 3 – Diagonal down leftMode 4 – Diagonal down right

Mode 5 – Vertical rightMode 6 – Horizontal downMode 7 – Vertical leftMode 8 – Horizontal up

Page 19: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Mode 3 DDCT- Diagonal Down LeftMode 3 DDCT- Diagonal Down Left

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 19

Step 1: (X00, X01,…… ,X32, X33)- Pixels in the 2-D spatial domain.

Step 2: 1D- DCT is performed for the 4X4block in diagonal down-left position with lengths L=1, 2, 3, 4, 3, 2, 1.

(A,B,C,……O,P)- coefficients in the DCT domain.

Step 3: The coefficients of step2 after 1D DCT are arranged vertically as shown in the figure.Apply Horizontal 1D- DCT for lengths L=7, 5, 3 and 1 and arranged in the same pattern

STEP 1 STEP 2 STEP 3

PIXELS Coefficients

Coefficients

Page 20: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 20

STEP 4 STEP 5

Step 4: Apply Horizontal 1D- DCT for lengths L=7, 5, 3 and 1, the coefficients are arranged in the same pattern as shown in the figure step 4.

Step 5: After Step 4, move all 2D (4X4) Directional DCT coefficients to the left.Implement quantization followed by 2D VLC for compression/coding along zig-zag scan.This scanning helps to increase the runlength of zero (transform) coefficients leading to

reduced bit rate in the 2D-VLC coding (similar to JPEG [12]).

CoefficientsCoefficients

Page 21: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Mode 4 DDCT- Diagonal Down Mode 4 DDCT- Diagonal Down RightRight

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 21

STEP 1 STEP 2 STEP 3

Step 1: X00, X01, ….., X33 are the pixels in the 2D spatial domain.

Step 2: 1D DCT is performed for the 4X4 block in diagonal down-right position with lengths L= 1, 2, 3, 4, 3, 2 and 1.

Step 3: The coefficients of step 2 after 1 D DCT are arranged vertically in the same pattern as shown in step 3. Then apply horizontal 1 D DCT for lengths L = 7, 5, 3 and 1 and arrange in the same pattern.

CoefficientsCoefficientsPIXELS

Page 22: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Implementation and Analysis of Directional Discrete Cosine Transform in H.264 22

STEP 4 STEP 5

Step 4: Apply horizontal 1 D DCT for lengths L= 7, 5, 3 and 1. The coefficients are arranged the same pattern as shown in step4.

Step 5: After step 4, move all 2D (4X4) directional DCT coefficients to the left. Implement quantization followed by 2D VLC for compression/coding zigzag scan.

April 16, 2012

Coefficients

Page 23: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Mode 5 DDCT- Vertical RightMode 5 DDCT- Vertical Right

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 23

STEP 1 STEP 2 STEP 3

Step 1: X00, X01, ….., X33 are the pixels in the 2D spatial domain.

Step 2: 1D DCT is performed for the 4X4 block in vertical-right position with lengths L= 2,4,4,4,2.

Step 3: The coefficients of step 2 after 1 D DCT are arranged vertically in the same pattern as shown in step 3. Then apply horizontal 1 D DCT for lengths L = 5, 5, 3 and 3and arrange in the same pattern.

CoefficientsCoefficients

PIXELS

Page 24: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 24

STEP 4 STEP 5

Step 4: Apply horizontal 1 D DCT for lengths L= 5, 5, 3 and 3. The coefficients are arranged the same pattern as shown in step 4.

Step 5: After step 4, move all 2D (4X4) Directional DCT coefficients to the left. Implement quantization followed by 2D VLC for compression/coding zigzag scan as shown in step 5. This scanning helps to increase the run-length of zero (transform) coefficients leading to reduce bit rate in 2D-VLC coding (similar to JPEG).

Coefficients

Page 25: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Mode 6 DDCT- Horizontal downMode 6 DDCT- Horizontal down

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 25

STEP 1 STEP 2 STEP 3

Step 1: X00, X01, ….., X33 are the pixels in the 2D spatial domain.

Step 2: 1D DCT is performed for the 4X4 block in Horizontal down position with lengths L= 2, 4, 4, 4 and 2.

Step 3: The coefficients of step 2 after 1 D DCT are arranged vertically in the same pattern as shown in step 3. Then apply horizontal 1 D DCT for lengths L = 5, 5, 3 and 3 and arrange in the

same pattern.

Coefficients CoefficientsPIXELS

Page 26: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 26

STEP 4 STEP 5

Step 4: Apply horizontal 1 D DCT for lengths L= 5, 5, 3 and 3. The coefficients are arranged the same pattern as shown in step 4.

Step 5: After step 4, move all 2D (4X4) directional DCT coefficients to the left. Implement quantization followed by 2D VLC for compression/coding zigzag scan as shown in step 5. This

scanning helps to increase the run-length of zero (transform) coefficients leading to reduce bit rate in 2D-VLC coding (similar to JPEG).

Coefficients

Page 27: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Mode 7 DDCT- Vertical LeftMode 7 DDCT- Vertical Left

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 27

STEP 1 STEP 2 STEP 3

Step 1 X00, X01, ….., X33 are the pixels in the 2D spatial domain.

Step 2: 1D DCT is performed for the 4X4 block in Vertical left position with lengths L= 2,4,4,4 and 2 as shown in step 3.

Step 3: The coefficients of step 2 after 1 D DCT are arranged vertically in the same pattern as shown in step 3. Then apply horizontal 1 D DCT for lengths L = 5, 5, 3 and 3 and arrange in the

same pattern.

Coefficients

CoefficientsPIXELS

Page 28: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 28

STEP 4 STEP 5

Step 4: Apply horizontal 1 D DCT for lengths L= 5, 5, 3 and 3. The coefficients are arranged the same pattern.

Step 5: After step 4, move all 2D (4X4) Directional DCT coefficients to the left. Implement quantization followed by 2D VLC for compression/coding zigzag scan as shown in step 3.

Coefficients

Page 29: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Mode 8 DDCT- Horizontal UpMode 8 DDCT- Horizontal Up

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 29

STEP 1 STEP 2 STEP 3

Step 1: X00, X01, ….., X33 are the pixels in the 2D spatial domain.

Step 2: 1D DCT is performed for the 4X4 block in Horizontal Up position with lengths L= 2, 4, 4, 4 and 2 as shown step 2.

Step 3: The coefficients of step 2 after 1 D DCT are arranged vertically in the same pattern as shown in step 3. Then apply horizontal 1 D DCT for lengths L = 5, 5, 3 and 3 and arrange in the same pattern.

Coefficients CoefficientsPIXELS

Page 30: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 30

STEP 4 STEP 5

Step 4: Apply horizontal 1 D DCT for lengths L= 5, 5, 3 and 3. The coefficients are arranged the same pattern as shown in step 4.

Step 5: After step 4, move all 2D (4X4) directional DCT coefficients to the left. Implement quantization followed by 2D VLC for compression/coding zigzag scan as shown in step 5. This scanning helps to increase the run-length of zero (transform)

coefficients leading to reduce bit rate in 2D-VLC coding (similar to JPEG [4]).

Coefficients

Page 31: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Obtaining Mode 3 from Mode 4Obtaining Mode 3 from Mode 4

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 31

Rotate pixels by –pi/2 ( counterclock wise by 90°) to get Mode 4

Pixels

Pixels

Pixels

Page 32: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

STEP BY STEP MODE CHANGESTEP BY STEP MODE CHANGE

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 32

MODE 3

-π/2=

MODE 4Pixels Pixels Coefficients

Step 1: Rotate the pixels by –π/2

Step 2: Perform 1-D DCT with Length = 1, 2, 3, 4, 3, 2, 1

Step 3: We get the coefficients of mode 3 (Diagonal Down Left) from mode 4 (Diagonal Down Right) as shown in figure 3.

Page 33: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Obtaining Mode 6 from Mode 5Obtaining Mode 6 from Mode 5

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 33

Rotate pixels by reflecting across the diagonal axis to get Mode 6

Pixels

Pixels

Pixels

Page 34: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Obtaining Mode 7 from Mode 5Obtaining Mode 7 from Mode 5

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 34

Rotate pixels by reflecting across the horizontal axis to get Mode 7

Pixels

Pixels

Pixels

Page 35: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Obtaining Mode 8 from Mode 5Obtaining Mode 8 from Mode 5

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 35

Rotate pixels by pi/2 (clockwise by 90°) to get Mode 8

Pixels

Pixels

Pixels

Page 36: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Eigen or Basis ImagesEigen or Basis Images

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 36

Mapping of a 2D data array into a 2D DCT domain implies decomposing the 2D data array into the basis images of the DCT.

Computation of basis images for diagonal down left (a) The original 4X4 block with diagonal down left computation (b) The 1 D DCT of coefficients for lengths 7, 5, 3 and 1 for basis image (0, 0) (c) The 1 D DCT of coefficients for lengths 7, 5, 3 and 1 for basis image (0,1) (d) The 1 D DCT of coefficients for lengths 7, 5, 3 and 1 for basis image (3,3)

Page 37: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Mode 3 Diagonal Down Left Eigen Image (1,1) matrix Mode 3 Diagonal Down Left Eigen Image (1,1) matrix Computation for 4X4 blockComputation for 4X4 block

Step 1: Horizontal 1D-DCT for length =7, 5, 3, 1

Step 2: Coefficients of 1D-DCT for length =7, 5, 3, 1

Step 3: Put back the coefficients in the block form

Step 4: 1 D DCT of diagonal down left with lengths = 1, 2, 3, 4, 3, 2 and 1

Step 5: Putting back the coefficients of step 4 1 D-DCT we get (1,1) basis image for 4X4 block

Page 38: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 38

MODE 3 - Diangonal down left basis images for 4X4 block of an image

Page 39: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 39

MODE 3 - Diangonal down left basis images for 8X8 block of an image

Page 40: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 40

MODE 0 or 1 – Vertical or Horizontal basis images for 8X8 block of an image

Page 41: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 41

MODE 5 – Vertical right basis images for 8X8 block of an image

Page 42: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Computation of DDCT for an imageComputation of DDCT for an image

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 42

Page 43: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Image Quality MeasuresImage Quality Measures

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 43

Page 44: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 44

•Criteria to evaluate compression quality

•Two types of quality measures Objective quality measure- PSNR, MSE Structural quality measure- SSIM [29]

• SSIM emphasizes that the human visual system is highly adapted to extract structural information from visual scenes. Therefore, structural similarity measurement should provide a good approximation to perceptual image quality.

•MSE and PSNR for a NxM pixel image are defined as

M

m

N

n

nmynmxNM

MSE1 1

2,,*

1

MSE

LPSNR

2

10log10

where x is the original image and y is the reconstructed image. M and N are the width and height of an image and ‘L’ is the maximum pixel value in the NxM pixel image.

Image Quality Measures

Page 45: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 45

• The SSIM index is defined as a product of luminance (l), contrast (c) and structural (s) comparison functions.

where , α>0, β>0 and γ >0 are parameters used to adjust the relative importance of the three components

where μ is the mean intensity, and σ is the standard deviation as a round estimate of the signal contrast. C1 and C2 are constants.

Page 46: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

H.264 JM18.0 [24] settingsH.264 JM18.0 [24] settings

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 46

Page 47: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 47

4.4.2 Encoder Configuration in JM 18.0

• FramesToBeEncoded = 1 #Number of Frames to be coded• ProfileIDC = 66 # Profile IDC (66 = baseline, 77 = main, 88 = extended; FREXT Profiles: 100 = High, 110= High 10, 122= High 4:2:2, 244 = High 4:4:4, 44= CAVLC 4:4:4 Intra, 118 = Multiview High Profile,128 = Stereo High Profile)• IntraProfile = 1 # Activate Intra Profile for FRExt (0: false, 1: true) #(e.g. ProfileIDC = 110, IntraProfile = 1 => High 10 Intra Profile)• Transform8X8Mode = 0 # (0: only 4X4 transform, 1: allow using 8X8 transform additionally, 2: only 8X8 transform•Transform 16X16Mode=0 #(0: no 16X16 mode, 1: allow 16X16 mode)• Input YUV file: foreman_qcif.yuv• Output H.264 bitstream: test.264• Output YUV file: test_rec.yuv• YUV format: YUV 4:2:0• Frames to be encoded: 1• Frequency used for encoded bitstream: 30.00 fps• DistortionSSIM = 1 # Compute SSIM distortion (0: disable/default, 1: enabled)

Page 48: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Experimental Results and Experimental Results and GraphsGraphs

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 48

Page 49: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

QCIF and CIFQCIF and CIF

49 Implementation and Analysis of Directional

Discrete Cosine Transform in H.264April 16, 2012

• CIF (Common Intermediate Format), is a format used to standardize the horizontal and vertical resolutions in pixels for sequences in video signals, commonly used in video teleconferencing systems.

•The CIF "image sizes" were specifically chosen to be multiples of macroblocks (i.e. 16 × 16 pixels) due to the way that discrete cosine transform based video compression/decompression is handled. So, by example, a CIF-size image (352 × 288) corresponds to 22 × 18 macroblocks

•QCIF means "Quarter CIF". To have one fourth of the area as "quarter" implies the height and width of the frame are halved. QCIF-size image is 176 x 144.

Page 50: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 50

Bit Rate (kbps) QP (I frame)

PSNR in dB

MSESSIM

5590.56 0 79.159 0.00079 1

5232.48 4 68.825 0.00852 1

3891.84 8 57.204 0.12378 0.9995

2168.16 16 48.86 0.84553 0.9965

1088.88 24 41.803 4.29364 0.9846

745.68 28 38.892 8.39157 0.9735

331.92 36 33.173 31.32035 0.9342

152.64 44 27.981 103.5018 0.8388

72 51 23.335 301.693 0.6703

Image metrics for Foreman QCIF sequence in integer DCT implementation in H.264

Page 51: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 51

Bit Rate(kbps)PSNR in

dBMSE SSIM

5486.9690.357 0.00006 1

5201.9682.436 0.00068 1

3882.5369.689 0.0014 1

2264.6360.147 0.00842 0.9996

1153.8452.976 0.55385 0.9972

686.5441.876 2.43788 0.9925

302.5338.653 10.2537 0.9801

142.5334.642 50.4376 0.9208

6730.764 110.268 0.8674

Image metrics for Foreman QCIF sequence in DDCT implementation in H.264

Page 52: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 52

QP (I frame) Encoding Time of Int-DCT (sec)Encoding Time of DDCT (sec)

0 10.87618.96

4 10.03218.096

8 9.18317.264

16 7.29215.367

24 5.66612.5437

28 4.96811.0642

36 4.06710.853

44 3.4849.638

51 3.0738.428

Encoding Time of I frame for Foreman QCIF sequence in DDCT and Int-DCT implementation in H.264

Page 53: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 53

PSNR v/s bit rate for DDCT and integer DCT for foreman QCIF sequence

Page 54: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 54

MSE v/s bit rate for DDCT and integer DCT for foreman QCIF sequence

Page 55: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 55

SSIM v/s bit rate for DDCT and integer DCT for foreman QCIF sequence

Page 56: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 56

Encoding time v/s quantization parameter for DDCT and integer DCT for Foreman QCIF sequence

Page 57: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Test sequence used for simulationTest sequence used for simulation

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 57

Bit Rate:72kbits/frame Bit Rate:67kbits /frame

Page 58: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Properties of DDCTProperties of DDCT

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 58

• Adaptivity: Unlike AVC in which the same DCT-like transform is applied to the intra prediction errors for all intra prediction modes of the same block size (4x4, 8x8, or 16x16), DDCT assigns a different transform and scanning pattern to each intra prediction mode. These transforms and scanning patterns are designed taking into account the intra prediction direction.

• Directionality: Since the intra prediction mode is known, the DDCT is designed with the knowledge of the intra prediction direction. By first applying the transform along the prediction direction, DDCT has the potential to minimize the artifacts around the object boundaries.

• Symmetry: Although there are 22 DDCTs for 22 intra prediction modes (9 modes for 4x4, 9 modes for 8x8, and 4 modes for 16x16), these transforms can be derived, using simple operators such as rotation and/ or reflection, from only 7 different core modes.

Page 59: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

ConclusionsConclusions

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 59

Page 60: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 60

Directional DCT has a better coding gain when compared to integer DCT

• PSNR value for DDCT is more when compared to Integer DCT.

• MSE value of DDCT is less compared to integer DCT for the same bit rates.

• SSIM graph shows that the value obtained for different bit rates is almost 1 for DDCT when compared to Integer DCT.

• Foreman frame of QCIF format gives a better quality image obtained from DDCT with respect to the output obtained from Integer DCT.

Drawback of DDCT

• Encoding time for DDCT is more when compared to integer DCT.

Page 61: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Future WorkFuture Work

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 61

Page 62: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

Future workFuture work

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 62

• DDCT can be extended to other video standards that use integer DCT in the transform domain.

• It can be extended for the entire video – inter frame prediction.

•It can be extended to other profiles in H.264 like main and extended profiles.

•Only 8 modes are described in this research. These can be extended to other directional modes. Payoff between increasing complexity and improved visual quality can be investigated.

Page 63: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

ReferencesReferences

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 63

Page 64: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 64

1. I. E.G. Richardson, “H.264 and MPEG-4 video compression: video coding for next-generation multimedia”, Wiley, 2003.

2. N. Ahmed, T. Natarajan, and K. R. Rao, “Discrete cosine transform,” IEEE Trans. Comput., vol. C-23, pp. 90-93, Jan. 1974.

3. K. R. Rao and P. Yip, “Discrete cosine transform: Algorithms, advantages, applications,” Boca Raton FL: Academic Press, 1990.

4. B. Zeng and J. Fu, “Directional discrete cosine transforms - A new framework for image coding”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 18, no. 3, pp. 305-313, Mar. 2008.

5. B. Zeng and J. Fu, “A compensation techniques in directional DCT’s”, IEEE International Symposium on Circuits and Systems, pp. 521- 524, June, 2007.

6. E. Lallana and M. Uy, “The Information Age,” UNDP-APDIP, 2003.7. Open source article, “Digital Revolution,” Wikipedia Foundation,

http://en.wikipedia.org/wiki/Digital_Revolution8. K. Sayood, "Introduction to data compression,” 3rd Edition, Morgan Kaufmann Publisher Inc., 2006. 9. R. Schafer and T. Sikora, "Digital video coding standards and their role in video communications,"

Proceedings of the IEEE, Vol. 83, pp. 907-923, Jan. 1995.10. Information technology-generic coding of moving pictures and associated audio information: ISO/IEC

13818-2 (MPEG-2) Std.11. Advanced video coding for generic audiovisual services, ITU-T Rec. H.264 / ISO / IEC 14496-10, Nov.

2009.12. I. Ahmad et al, “Video transcoding: An overview of various techniques and research issues”, IEEE

Trans. on Multimedia, vol. 7, pp. 793-804, Oct. 2005.13. Open source article, “H.264/MPEG-4 AVC,” Wikipedia Foundation,

http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC

Page 65: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 65

14. S. Kwon, A. Tamhankar and K.R. Rao, ”Overview of H.264 / MPEG-4 Part 10”, J. Visual Communication and Image Representation, vol. 17, pp.186-216, April 2006.

15. T. Wiegand and G. J. Sullivan, “The H.264 video coding standard”, IEEE Signal Processing Magazine, vol. 24, pp. 148-153, March 2007.

16. A. Puri et al, “Video coding using the H.264/ MPEG-4 AVC compression standard”, Signal Processing: Image Communication, vol. 19, pp: 793 – 849, Oct. 2004.

17. G. Sullivan, P. Topiwala and A. Luthra, “The H.264/AVC advanced video coding standard: Overview and introduction to the fidelity range extensions”, SPIE conference on Applications of Digital Image Processing XXVII, vol. 5558, pp. 53-74, Aug. 2004.

18. K. R. Rao and P. C. Yip, “The transform and data compression handbook”, Boca Raton,FL: CRC press, 2001.

19. T. Wiegand et al, “Overview of the H.264/AVC video coding standard”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, pp. 560-576, Jul. 2003.

20. T. Wiegand and G. J. Sullivan “The picturephone is here: Really” IEEE spectrum, vol.48, pp.50-54, Sept.2011.

21. I. Richardson, “The H.264 advanced video compression standard”, Wiley, 2nd edition, 2010.22. Intra prediction modes in H.264. Website: http://www.vcodex.com/files/h264_intrapred.pdf23. F. Kamisli and J. S. Lim, “Video compression with 1-d directional transforms in H.264/AVC”, IEEE

ICASSP, pp. 738-741, Mar. 2010.24. H.264/AVC reference software. Website: http://iphome.hhi.de/suehring/tml/download 25. Intra coding with directional DCT and directional DWT, Document: JCTVC-B107_r126. Directional Discrete Transform JCTV Website:

http://wftp3.itu.int/av-arch/jctvc-site/2010_07_B_Geneva/JCTVC-B107.zip27. B.Chen, H.Wang and L.Cheng, “Fast directional discrete cosine transform for image

compression”, Opt. Eng. vol. 49, issue 2, article 020101, Feb. 2010.

Page 66: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 66

28. C. Deng et al, “Performance analysis, parameter selection and extensions to H.264/AVC FRExt for high resolution video coding”, J. Vis. Commun. Image R., vol. 22 (In Press), Available on line, Feb. 2011.

29. Z.Wang et al, “Image quality assessment: From error visibility to structural similarity”, IEEE Trans. on Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004.

30. W.Zhao, J.Fan and A.Davari, “H.264-based wireless surveillance sensors in application to target identification and tracking”, i-manager’s Journal on Software Engineering, vol.4, no. 2, Oct. 2009.

31. Website: http://web.eng.fiu.edu/fanj/pdf/J5_i-manager09h264_camera.pdf32. W.Zhao et al, “H.264-based architecture of digital surveillance network in application to computer

visualization”, i-manager’s Journal on Software Engineering, vol.4, no. 4, Apr. 2010.33. Directional Discrete Cosine Transform theory Website: http://web.eng.fiu.edu/fanj/pdf/J8_i-

mgr10architecture_camera.pdf34. D. Marpe, T. Wiegand and G. J. Sullivan, “The H.264/MPEG-4 AVC standard and its applications”, IEEE

Communications Magazine, vol. 44, pp. 134-143, Aug. 2006.35. Website: http://iphome.hhi.de/wiegand/assets/pdfs/h264-AVC-Standard.pdf36. F. Kamisli and J. S. Lim, “Transforms for motion compensation residual”, IEEE ICASSP, pp.789-792,

Apr. 2009.37. Z.Wang, E.P.Simoncelli and A.C.Bovik, “Multi-scale structural similarity for image quality assessment”,

Thirty-Seventh Asilomar Conference on Signals, Systems and Computers, vol. 2, Nov. 2003.38. C.L.Chang and B.Girod, “Direction-adaptive partitioned block transform for image coding”, 15 th IEEE

International Conference on Image Processing, pp. 145-148, Oct. 2008.39. H.Xu, J.Xu and F.Xu, “Lifting-based directional DCT-like transform for image coding”, IEEE Trans. on

Circuits and Systems for Video Technology, vol. 17, issue 10, pp. 1325-1335, Oct. 2007.40. J.Xu, B.Zeng and F.Wu, “An overview of directional transforms in image coding”, Proceedings of 2010

IEEE International Symposium on Circuits and Systems, pp. 3036-3039, Aug. 2010.

Page 67: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 67

41. MPEG-1 basics Website: http://en.wikipedia.org/wiki/Mpeg-142. MPEG-2 basics Website: http://en.wikipedia.org/wiki/Mpeg-243. MPEG-4 basics Website: http://en.wikipedia.org/wiki/Mpeg-444. H.261 basics Website: http://en.wikipedia.org/wiki/H.26145. H.262 basics Website: http://en.wikipedia.org/wiki/H.26246. H.263 basics Website: http://en.wikipedia.org/wiki/H.26347. DFT basics Website: http://en.wikipedia.org/wiki/Discrete_Fourier_transform48. K. R. Rao and J. J. Hwang, “Techniques and standards for image/video/audio coding”,

Prentice Hall, 1996.

Page 68: Implementation and Analysis of Directional Discrete Cosine Transform in H.264 for Baseline Profile Shreyanka Subbarayappa Electrical Engineering Graduate

QUESTIONS ?QUESTIONS ?

April 16, 2012Implementation and Analysis of Directional

Discrete Cosine Transform in H.264 68