Video Coding - National Chiao Tung Universitycmliu/Courses/Compression/...Video Coding C.M. Liu...

Video Coding

C.M. Liu

Perceptual Signal Processing Lab

College of Computer Science

National Chiao-Tung University

Office: EC538

(03)5731877

cmliu@cs.nctu.edu.tw

http://www.csie.nctu.edu.tw/~cmliu/Courses/Compression/

1. Color Fundamentals

Sir Isaac Newton in 1666

A glass prism

Six Broad Regions

Violet, Blue, Green, Yellow, Orange, and Red.

Light Properties

Visible light is composed of a relatively narrow band (400nm - 700 nm) of

frequencies in the electromagnetic spectrum.

Achromatic light has only the attribute: intensity or amount.

Three basic quantities used to describe the quality of a chromatic light source:

radiance, luminance, and brightness.

Radiance

The total amount of energy that flows from the light source.

It is measured in watts(W).

Luminance

A measure of the amount of energy an observer perceives from a light source.

It is measured in lumens(lm)

Brightness

A subjective descriptor that is practically impossible to measure.

It embodies the achromatic notion of intensity and is one of the key factors in describing color sensation.

Human Eye

Cones (6-7 millions in an eye)

65% of all cones are sensitive to red light.

33% are sensitive to green light.

2% are sensitive to blue light.

Primary Colors of Light

Primary Colors

Red, Green, Blue.

Secondary Colors

Cyran = G+B

Magnenta = B+R

Yellow = R+G

W=R+G+B.

Primary Colors of Pigments Primary Colors C, M, Y C absorbs R M absorbs G Y absorbs B

Secondary Colors C+M absorbs R and G C+M = B

M+Y absorbs G and B M+Y = R

Y+C absorbs B and R Y+C = G

C+M+Y = K (black)

Color Specified by brightness and chromaticity (hue and saturation). Chromaticity can be regarded as brightness-normalized color. Hue is an attribute associated with the dominant wavelength in a mixture

of light waves. Saturation referrs to the relative purity or the amount of white light

mixed with a hue. The degree of saturation is inversely proportional to the amount of white

light added.

Tristimulus Values (R, G, B) The amounts of red, green, blue required to form a specific color.

Trichromatic Coefficients x = R/(R+G+B) y = G/(R+G+B) z = B/(R+G+B) Note that x+y+z = 1, thus trichromatic coefficients can be represented

only by x and y (2D space)

2. Color Fundamentals-- CIE Chromaticity Diagram

Pure color is at the boundary of the tongue

Suppose that we have two colors (two points within the tongue)

Any mid-point on the line can be obtained by mixing the two colors

Color Gamut

Any point within the corresponding triangle can be obtained by mixing the three colors

All colors cannot be reproduced using only three primary colors.

The boundary of the color is irregular because of a combination of additive and subtractive mixing.

The rugged shape represent the colors that can be represented by a typical printing devices based on CMY space.

3. Color Models

Definition

A color model is a specification of a coordinate system and a subspace within that system where each color is represented by a single point.

Purpose

Facilitate the specification of colors in some standard way.

RGB (red, green, blue)

Display devices

CMY (cyan, magenta, yellow) and CMYK(+black)

Printing devices

HSI (hue, saturation, intensity)

Intuitive description of colors.

Hue (red, green, … violet)

Saturation

Red: high saturation

Pink: less saturation

3. Color Models-- RGB

Full Color (True color) Each of RGB components are represented by 8 bits

RGB pixel is said to have a depth of 24 bits

There are 224 (=16,777,216) colors

Safe RGM Color

Have a subset of colors that are likely to be reproduced faithfully,

reasonably independently of viewer hardware capabilities.

63 = 216 safe colors

6 reproduction levels for each R, G, B

0, 51, 102, 153, 204, or 255.

3. Color Models– RGB

3. Color Models– RGB Full Color

3. Color Models– RGB Safe Color

3. Color Models-- CMY

CMY and CMYK Models

Color printers and copiers, require CMY data input or perform an

RGB to CMY conversion internally.

C+M+Y = K

Black is used most frequently for typical printing Thus,

black ink is added, yielding the CMYK color model

3. Color Models-- HSI

HSI Model

Intensity axis : the main diagonal from black to white

A plane perpendicular to this axis contains the colors with the same

intensity

Saturation: the distance from the intensity axis

Hue: the angle on the plane with respect to the red color

3. Color Models-- HSI

3. Color Models– HSI

Intensity axis : the main diagonal from black to white

A plane perpendicular to this axis contains the colors with the same intensity

Saturation: the distance from the intensity axis

Hue: the angle on the plane with respect to the red color

3. Color Models– RGB to HSI

Saturation

Intensity

3. Color Models– HSI to RGB

)60cos(

SectorRGCase

)60cos(

240120

SectorGBCase

)60cos(

360240

SectorBRCase

Converting colors from HSI to RGB

There are three sectors of interest, corresponding to the

120° intervals in the separation of primaries.

3. Color Models– HSI

RGB with HSI representation

Discontinuity.

3. Color Models– HIS

Manipulation Example

Analog Video25

Video: A sequence of images played back fast enough to reproduce to the illusion of motion.

Early movies:

16 frames per second (fps)

Updated to 24 fps

Double-/triple-blade shutter artificial 48/72 fps

European standards PAL/SECAM

50 Hz electricity 25 fps

US/Japan/etc

60 Hz electricity 30 fps

29.97 fps (1953 w/ color TV)

Television, a medium. So called because it is nether

rare nor well done.

--Anonymous

The CRT27

Interlacing28

Odd lines

Retrace

(horizontal blanking)

Even lines

The Color CRT29

The NTSC Standard30

525 scan lines 482/483/487(?) visible

Aspect ration—4:3 (by T. Edison in 1930s) 4/3 x 483 ~ 644 pixels

Image is actually continuous horizontally

Other aspect ratios: PAL/SECAM: 1.33

16/35mm film: 1.33

HDTV: 1.78

Widescreen film: 1.85

70mm film: 2.10

Cinemascope film: 2.35

pel aspect ratio Picture element is not a dot—more like a rectangle

Resolution v.s. Aspect Ratio31

Various Resolution32

TV Formats33

Video Parameters34

Progressive Scan, Frame Rate.

The RGB-to-YUV Conversion35

Y = 0.299R + 0.587G + 0.115B

U = -0.147R - 0.289G - 0.436B = 0.492 (B-Y)

V = 0.615R - 0.515G - 0.100B = 0.877 (R-Y)

Y = 0.299R + 0.587G + 0.115B

I = 0.596R - 0.274G - 0.322B = -sin33°U + cos33°V

Q = 0.211R - 0.523G - 0.311B = cos33°U + sin33°V

CCIR 601/ITU-R BT.601-236

Standard sampling rates

Multiples of 3.725 MHz

Sampling patterns Y:Cb:Cr

4:4:4—all components samples @13.5MHz

Typical 4:2:2

Luminance sampled @13.5 MHz

Chrominance components sampled @6.75 MHz

RGB to CCIR 60137

After RGB-to-YCbCr conversion

Y normalized as Ys [0, 1]

Cb Cbs [ -½ , ½ ]

Cr Crs [ -½ , ½ ]

8-bit integer conversion

Y = 219Ys + 16, Y [16, 235]

U = 224Cbs + 128, U [16, 240]

V = 224Crs + 128, V [16, 240]

Common Interchange Format (CIF)38

Teleconferencing standard

PAL/NTSC-based: YUV/30 fps

Multiples of 16 x 16 SQCIF: 128 x 96

QCIF: 176 x 144

CIF: 352 x 288

4CIF: 704 x 576

16CIF: 1408 x 1152

Pixel aspect ratio: 1.222:1

SIF: Source Input Format

MPEG-1‘s parlance for CIF

625-line (PAL) & 525-line (NTSC) version

Motion Compensation39

The use of previous frames as prediction of current

I.e. exploitation of temporal redundancy

Rationale:

Most of the time, frame-to-frame changes will be ‗small‘

Identify ‗objects‘ that have moved and include a motion

compensation vector

Motion Compensation Example40

Frame #1

Frame #2

Frames 1 & 2 overlaid

motion vectors

Motion Compensation Example41

Block-based Motion Compensation42

‗Pixel-splitting‘ Motion Estimation43

Observation

Best fit may not be pixel aligned

―Double‖ the image size

I.e., introduce intermediate pixels with interpolated values

5.02,5.02

DBvCAv

DChBAh

Motion Estimation Considerations44

Observations:

Smaller block more possibilities to explore

Larger block higher chance of not finding a match

Note: Numerous methods exist for balancing prediction

accuracy (compression) & computation time

Motion Estimation Example45

Subpixel Eestimation48

MPEG-1/2

MPEG-1 (ISO/IEC 11172) completed in 1991 digital storage media at bit rates up to about 1.5Mbps

remove intra and inter-frame redundancy with block-based DCT and

motion compensation (I, P and B-frames)

progressive pictures only, optimized for SIF (352x240) resolution

fixed 4:2:0 color format

MPEG-2 (ISO/IEC 13818) completed in 1994 extensions that allow for greater input format flexibility, higher data

rates and better error resilience

field/frame prediction modes for interlace format support

field/frame DCT coding syntax

downloadable quantization matrix

scalability extensions (spatial, temporal, SNR)

display syntax (e.g., 3:2 pull-down, pan-and-scan, color formats)

MPEG3, 4, 7

MPEG-3

– Original intended for HDTV coding, dropped when MPEG-2 application domain was extended to HDTV

MPEG-4

– Originally intended for very low bit rate audio/visual coding

– It may be extended for both low and high bit rate application

– Object-oriented coding algorithm

MPEG-7

– There is no reason to pick up the series number 7 instead of 5 or 6 or other

– Intend to set a standard of “Multimedia content description interface” that will specify a standardized description of various types of multimedia information.

Bitrates and Resolutions

Standard TV

Over 1080P

64K 1M 1.5M 15M 300M Over 600M

MPEG-4

MPEG-4Studio Profile

MPEG-2

MPEG-24:2:2

Profile

Introduction

Video Formats

Frame Reorder

Data Hierarchy

Syntax

Compression Ratio

MPEG1-- Introduction

Backgrounds

ISO/IEC Draft Standard CD 11172, Dec., 1991

Compression and Decompression of Video & Audio Signals

Synchronization of Audio and Video

Lossy Coding Techniques

Features

A Toolkit

Supports intra and interframe modes.

Only progressive-format data is supported.

The algorithm specifies the bit stream syntax and semantics and a method for decoding it.

MPEG-1 Video54

Overall structure very similar to H.261

… with some non-trivial differences

Focus on stored as opposed to live video

Random access

In H.261 potentially all frames after the first may depend on previous one

MPEG-1 provides random access by requiring periodic independently-encoded frames /I-frames/

Distance b/w I-frames is a trade-off b/w convenience & compression

P-frames—predictively coded

B-frames—bidirectionally predictively coded

MPEG1-- Introduction (c.1)

Features

The algorithm does not specify preprocessing of the video, encoding steps(e.g. motion estimation), postprocessing.

The algorithm does not specify parameters such as coded bit rate. lines per picture(<4096), pels per line (<4096), picture rate(24, 15, or 30), and pel aspect ratio(14 choices) .

A special subset of the parameter space

pels/line <= 720

lines <= 576

macroblocks per picture <= 396

macroblocks per sec. <= 396x25

picture rate <=30

bit rate <=1.86 Mbits/sec.

4:2:0 Format

Crominance components is 1/2 resolution of Y

components ( in both directions)

The MPEG1 format

4:2:0 macroblocks

MPEG1-- Video Formats

x x x x x x

Y Cb Cr

MPEG1-- Video Formats(c.1)

4:1:1 Format

Same bits/pel area as 4:2:0 but vertical resolution is

higher

Used in DVIx x x x x x

x x x x x x

x x x x x

x x x x x x

x x x x

x x x x x x

4:2:2 Format

4:2:2 Macroblock

x x x x x x

x x x x x

x x x x x x

x x x x

x x x x x x

Y Cb Cr

4:4:4 Macroblock

Y Cb Cr

MPEG1-- Frame Reorder

Encoder Input

GOP1 is a closed GOP while GOP2 a open GOP.

Encoder Output

Decoder Ouput

1(I) 2(B) 3(B) 4(P) 5(B) 6(B) 7(P) 8(B) 9(B) 10(I) 11(B) 12(B) 13(P)

1(I) 4(P) 2(B) 3(B) 7(P) 5(B) 6(P) 10(I) 8(B) 9(B) 13(P) 11(B) 12(B)

GOP1 GOP2

MPEG1-- Data Hierarchy

Video Sequence

The highest syntctic structure of the coded video bitstream.

Sequence header, sequence extension.

Group of Pictures(GOP)

PictureSlice Macroblock Block

MPEG1-- Data Hierarchy (GOP)

I Pictures

Not dependent on another pictures

P Pictures

Predicted from I or P pictures

1 2 3 4 5 6 7 8 1I B B B P B B B I

Forward Prediction

Bidirectional Prediction

B Pictures Predicted from nearby I

and/or P pictures

MPEG-1 Frame Types63

I : All information for frame present.

P: Predictively encoded from previous I or P.

B: Predictively encoded from previous I or P

and next I or P.

I P IP P PB B B B B B B B B B

MPEG-1 Display vs. Bitstream Order

Problem: B-frames depend on future—how could they be

decoded?

Solution? Reorder frames!

I P IP P PB B B B B B B B B B

1 4 167 10 132 3 5 6 8 9 11 12 14 15

1 2 145 8 113 4 6 7 9 10 12 13 15 16

MPEG1-- Functional Blocks

DCT-Video Input Output

Buffer

ME & MC

Changable per Macroblocks

Control Signals• Macroblock type

• Coded Block Pattern

• Quantizer Scale Factor

MPEG1-- Motion Compensation & Estimation

Objects

Reduce the temporal redundancy

Motion Compensation

Process of compensating for the displacement of moving objects from one frame

to another.

Motion Estimation

The process for finding corresponding pixels in the frame; this process is referred

to as motion estimation.

t t+1 t+2

Macroblocks(16x16) Search Area

Motion/No motion: check if a motion vector transmitted or is it assumed to be zero.

Intra/Non Intra: check the variance of the estimation errors using the vector in step 1.

Coded/not coded: check if the residual is large enough to be coded using DCT

Quant/No Quant: check if the quantizer scale is satisfactory or should be changed.

MPEG1-- Encoded Tree for Macroblocks in P-picture

Macroblocks

Motion

No Motion

Non Intra

Coded No MQuant

Not Coded

No QuantQuant

No QuantQuantCoded

Not Coded

Non Intraskipped MB

MPEG-1 Rate Control68

Sequence level: within a GOP (Group of Pictures)

B-frames are easiest to eliminate

Frame level:

Quantization step adjustment

Dropping of higher order coefficients

Constrained Parameter Bitstream (CPB)

Horizontal size 768 pixels

Vertical size 576 pixels

396 macroblocks/frame @ 25 fps (352 x 288 pixels)

330 macroblocks/frame @ 30 fps (352 x 240 pixels)

Rate is 1-1.5 Mbit/s

CPB is understood to be the MPEG-1 typical setup

MPEG1-- Tree Decision for Macroblocks in P-

picture

VAR: variance of original

VAROR: variance of reconstructed error

xBD motionvector

yBD motionvector

2.7y=x/1.1

Motion Compensation

Motion

Compensation

Non Intra

MPEG1-- Macroblock Types in B-picture

Macroblock Types

Intracoded

Forwrd predictive coded

Backward predictive coded

Bidirectional predictive coded

Bidirection prediction

Yields accurate prediction in the case of cover/uncovered images.

Two pictures are needed to decode a B picture.

Significant compresion relative to unidirectional prediction (7 kbits/picture versus 100 kbits/picture).

MPEG1-- Encoded Tree for Macroblocks in B-picture

Macroblocks

Forward

No MQuant

Not Coded

MQuant

CodedNo MQuant

Not Coded

MQuant

Not Coded

No MQuant

MQuant

No MQuant

MQuant

Backward

Forward &

Backward

MPEG-1 Bitstream Syntax72

Macroblock and Slice73

MPEG-1 Bitstream Syntax (2)74

extension.header 0000 01B5

GOP.start 0000 01B8

picture.start 0000 0100

reserved 0000 01B0

reserved 0000 01B1

reserved 0000 01B6

sequence.end 0000 01B7

sequence.error 0000 01B4

sequence.header 0000 01B3

slice.start.1 0000 0101

slice.start 0000 01AF

user.data.start 0000 01B2

MPEG1-- Discrete Cosine Transform

Objects

Orthogonal transform

Filter-bank-oriented

With the frequency domain interpretation

A fast algorithm and a close approximation to the optimal for a large class of

images

Transform a block in spatial domain into another domain suitable for removing

spatial and psychovisual redundancy

DCT u vC u C v

I j k j u k v

where C x for x and C x for x

( , )( ) ( )

( , ) cos[( ) / ]cos[( ) / ]

( ) , ( )

2 1 16 2 1 16

20 1 0

MPEG1-- Quantization

Concepts

The combination with run-length coding contribute

to most of the compression

Visual quality achievement by adaptive

quantization

Coarser quantizer for higher frequencies

Application specification quantization matrix

8 16 19 22 26 27 29 34

16 16 22 24 27 29 34 37

19 22 26 27 29 34 34 38

22 22 26 27 29 34 37 40

22 26 27 29 32 35 40 48

26 27 29 32 35 40 48 58

26 27 29 34 38 46 56 69

27 29 35 38 46 56 69 83

For Intra blocks (both luminance and chrominace)

Nonintrablocks all scales are 16

(( , )

( , ) * _) ( , )

( ( , ) int

W v u quantizer scalek QF v u

where k for rablock

k sign I u v for non rablock

Quantizer77

MPEG1-- VLC & Runlength Coding

Reduce the coding redundancies

Runlength coding in Zig-Zag Scanning

Variable length coding for the value and runs

3 0 0 0 0 0 0 0

2 -2 0 0 0 0 0 0

4 0 20 0 0 0 0 0

0 0 0 0 0 0 0 0

run-length level run value

MPEG1-- Syntax

Sequence Layer

Sequence_header_code

Horizontal_size

Vertical_size

Pel_aspect_ratio

Picture_Rate

Bit_Rate

Quantizer specification

User Data

MPEG1-- Syntax (c.1)

Group of Picture Layer

Group start code

Time code

Open/closed GOP

User data

Picture Layer

Picture header

temporal_reference

picture_coding_typw (I, P, B, DC only)

MPEG1-- Syntax (c.2)

Slice (resynchronization)

Slice_start_code

Macroblock Layer(motion compensation)

Macroblock header

Macroblock_address_increment

Macroblock_type

Quantizer scale

Motin_vector

Block Layer-Block Data (DCT Unit)

DCT data

MPEG1-- Compression Ratios at Each Stage

Single Frame at 640x480x24 bpp 910KB

Preprocess (filter to reduce noise) ??

YUV (4:2:2) Conversion (from RGB) 460 KB

Scaling to CIF 115 KB

DCT 115 KB

Quantization 115 KB

Run-Length + Huffman Coding 24 KB

Intraframe Compression 5 KB

23:1 Compression ratio or a 184:1 compression ratio

MPEG2-- MPEG Working Group

Formed in 1988 to establish standards for coding of moving

pictures and associated audio for various applications such

as storage media, distribution and communication.

ITU-T SGXV

The Experts Group for ATM Video Coding was formed in

1990 to develop video coding standards appropriate for B-

ISDN using ATM transport.

MPEG2 Draft

Draft was prepared by MPEG and ITU-T SG15

MPEG2 Standards

Eight Parts

13818-1 System

13818-2 Video

13818-3 Audio

13818-4 Conformance Testing

13818-5 Simulation Software

13818-6 Digital Storage Media Command and Control (DSM-CC, July, 1996)

13818-7 Nonbackwards Compatible Audio (April 1997)

13818-8 10 bit Video

13818-9 Real-Time Interface (July 1996)

Profiles

4:2:2 Profile ( Jan, 1996)

Multiview Profile (Nov., 1996)

MPEG2 Applications

BSS Broadcasting Satellite Service (to the home)

CATV Cable TV Distribution on Optical Networks, Copper, etc.

CDAD Cable Digital Audio Distribution

DAB Digital Audio Broadcasting (terrestrial and satellite broadcasting)

DTTB Digital Terrestrial Television Broadcasting

EC Electronic Cinema

ENG Electronic News Gathering(including SNG, Satellite News Gathering)

FSS Fixed Satellite Service (e.g. to head end)

MPEG2 Applications (c.1)

HTT Home Television Theatre

IPC International Communications (videoconferencing, videophone, etc)

ISM Interactive Storage Media (optical disks, etc)

MMM Multimedia Mailing

NCA News and Current Affairs

NDB Networked Database Services (via ATM, etc.)

RVS Remote Video Surveillance

SSM Serial Storage Media (digital VTR, etc.)

MPEG2-- System

Video audio synchronization

Multiplexing multiple programs

Transporting over communication channels

Multi-media on CD-ROM

Broadcasting

Digital Storage Media Command and Control (DSM)

Protocal

MPEG2-- Audio

Three Layers Coding (I, II, III)

3/2 Stereo (3 front/ 2 surround) plus Low

Frequency Enhancement (LFE) Channels

Downwards and Backwards Compatibility

Multi-Lingual Capability

Multi-Channel Audio Coding

MPEG2-- Video

Scalable and Nonscalable Syntax

Profiles and Levels

Progressive and Interlaced Sequences

Frame and Field Picture Processing

Error Concealment

MPEG2-- Video

MPEG-2 Enhancements

– Basic coding mode is interframe DCT with I, P, B pictures

– New field/frame prediction modes for interlace support

– Quantization/coding extensions to MPEG-1 syntax for improved quality Improved quatization with greater range/adaptive

Mew Intra-frame VLC‘s

– Scalability extensions for hierarchical service, robustness, etc. Spatial scalability modes for compatibility

Temporal scalability

SNR scalability

DATA partitioning (frequency scalability)

– New system layer for multiplexing, transport, etc.

MPEG2-- Video

MPEG-2 Prediction Modes

– MB syntax extended to include a number of alternative

prediction modes for better compression of interlaced video

– Frame-based prediction (identical to that of MPEG-1)

1 or 2 vector

MPEG2-- Video

– Field-based prediction

Each field of a MB is predicted separately in this mode

– Adaptive field/frame selection based on better match (should say better compression performance)

2 or 4 vectors 2 vectors

MPEG2-- Video

– Special prediction mode - dual prime

Basically a set of field motion vectors with a scaling to near

or far field, plus a transmitted delta

Reference Prediction

MPEG2-- Video

Field/frame DCT coding syntax

Field DCT Coding Luminance Macroblock Frame DCT Coding

Note: Chrominance blocks in 4:2:0 mode are always DCT coded in Frame order

MPEG2-- Video

Alternative Zig-Zag scan

8x8 block of quantized DCT coefficients

Normal Zig-Zag scan.

Mandatory in MPEG-1

Option in MPEG-2

Alternative Zig-Zag scan

Not used in MPEG-1

Option in MPEG-2

For Frame DCT

coding of interlaced

video, more energy

exists here, so run

length coding is more

efficient.

MPEG2-- Profile and Levels

Concepts

MPEG2 is a generic standard and it is not practical to implement the full specification at the early stages of its adoption.

A limited number of subsets have been defined by means of "profile" and "level".

Profile

A subset of the bitstream syntax. Within this subset, it is still possible to have a large variation in encoders and decoders on values taken by parameters in the bitstream.

Levels

Levels are defined within each profile to deal with the variation in a profile.

A level within a profile is a defined set of constraints imposed on parameters in the bitstream.

profilesyntax

profile

Levels

(1440)

Simple (4:2:0) Main (4:2:0)Main+ (4:2:0)

(scalable)

(scalable)Max. Resolution

Y samples/sec

Min. Resolution

Y samples/sec

# of layers

Bit-Rates (Mbps)

Max. Resolution

Y samples/sec

Min. Resolution

Y samples/sec

# of layers

Bit-Rates (Mbps)

Max. Resolution

Y samples/sec

Min. Resolution

Y samples/sec

# of layers

Bit-Rates (Mbps)

Max. Resolution

Y samples/sec

Min. Resolution

Y samples/sec

# of layers

Bit-Rates (Mbps)

720/576/30

1920/1152/60

62.7 M

1440/1152/60

47.0 M

720/576/30

352/288/30

3.05 M

1440/1152/60

47.0 M

720/576/30

10.4 M

60(a), 40 (mid+b), 15(b)

720/576/30

10.4 M

15(a), 10(b)

352/288/30

3.05 M

4(a), 3(b)

1920/1152/60

62.7/83.6 M

960/576/30

14.8/19.7 M

100(a), 40(m+b), 15(b)

1440/1152/60

720/576/30

11.1/14.8 M

80(a), 60(m+b), + 20(b)

720/576/30

11.1/14.8 M

352/288/30

3.05 M

20(a), 15(m+b), 4(b)

MPEG2-- Profile and Level (c.1)

MPEG2-- Compatibility between Different Profiles/Levels

NP@HL x

NP@H-14 x x

NP@ML x x x

M+@H-14 x x x

M+@ML x x x x x

M+@LL x x x x x x

MP@HL x x

MP@H-14 x x x x x

MP@ML x x x x x x x x x

MP@LL x x x x x x x x x x

SP@ML x x x x x x x x x

MPEG2-- Scalable Extensions

Motivation

Support applications such as video on ATM,

interworking of video standards, HDTV with embedded

TV, etc.

Four Modes of Scalability

Data Partitioning

SNR Scalability

Spatial Scalability

Temporal Scalability

HDTV Compression100

―Grand Alliance‖

FCC-encouraged partnership to define HDTV standard

MPEG-2 compression

HDTV == MP@HL

H.263101

Based on H.261

Focus on non-interlaced video

GOBs/slices

Strip of pixels w/ multiple of 16

Bottom strip may have fewer

GOB macroblocks

Main upgrades

Works with P & I frames

Motion compensation [-16, 15.5]

Prediction == median of motion vectors of neighbors

Half-pixel motion compensation

H.263: Bitstream Structure102

H.263 Optional Modes103

Unrestricted motion vector [-31.5, 31.5] useful for higher resolutions

Motion vector can point outside picture

Syntax-based arithmetic coding Var-length codes replaced w/ AC, m = 16

Specifies various CC tables:

MVector, intra-DC, intra-/inter-AC coefficients

Advanced prediction Four luminance vectors (vs. one for baseline)

Overlapped Block Motion Compensation (OMBC)

Weighted sum of predictions

PB-frames P + B picture/frame

H.263+ Modes104

Advanced intra coding

Prediction-based encoding for coefficients

Deblocking filter

Smoothing of block boundaries for better prediction

Reference picture selection

Selection of reference frame other than the preceding

Temporal, SNR, & spatial scalability

Similar to MPEG-2

Temporal achieved through separate B frames

SNR through layering

Spatial through upsampling

H.263+ Modes (2)105

Reference picture resampling

Resizing/warping of reference picture to obtain better prediction

Reduced resolution update

For highly active scenes

Macroblock is assumed twice as high/wide

Alternate VLC

Enables the use of intra frame codes for inter coding

Helps during high-activity periods

Modified quantization

Split luminance/chrominance quantization

Escape sequences for overload situations

Enhanced reference picture selection

H.264/MPEG-4 Part 10106

Same baseline macroblock structure, plus Submacroblocks:

8x4, 4x8, 4x4, 8x8, 16x8, 8x16

Motion compensation Variable granularity motion tracking at various

levels of detail

Quarter-pixel accuracy

Block-edge filters

Up to 32 possible reference pictures

B pictures—up to two motion vectors (as before)

Pskip—only motion vector is transmitted

H.264 Transform107

New 4x4 DCT/DWHT combination

Simpler implementation

Better for small stationary

Less noise & noise propagation

Not normalized--compensated through scaling during quantization

H.264 Intra Prediction108

H.261-263—no de-correlation for I frames

H.264—prediction for intra-coding (9 modes):

Components in an MPEG-4 Terminal

Hierarchical, Interactive,

Audiovisual Scene

Elementary

Streams

Demultiplex

...Primitive

AV Objects

Decompression Composition and

Rendering

...Upstream Data

(User Events, Class Request, ...)

Composition

Information

Scene Description

(Script or Classes)

Basics of MPEG-4

A scene is constructed of multiple independent objects

Audio or visual, natural or synthetic

Objects can be encoded separately with scene description information

This allows to create the combination of different object types, e.g., animation with natural

video, 3D mesh, Web papers, ...

Objects are composited in a scene at the decoder side:

MPEG-4 has standardized a binary format for scene description, referred to as BIFS,

which is based on VRML

This allows to multiplex and synchronize the data associated with objects, so

that they can transported over network providing a QoS appropriate for

the nature of the specific objects

And interactivity with audiovisual scene generated at the receiver‘s side

Parts of the StandardPart 1: Systems Part 6: DMIF

Part 2: Visual Part 7: Optimized encoder tools

Part 3: Audio Part 8: MPEG-4 on IP

Part 4: Conformance Part 9: Reference hardware

Part 5: Software framework Part 10: Advanced Video Coding

Video Object and Video Object Plane

(Background)

VO2 (Moving

Object)

VO1 (Stationary

Object)

VOP: instance of a video object at a

given time

Syntax Hierarchy

VOL0 VOL1

GOV0 GOV1

VOP0 .....VOPn VOPn+1 .....VOPm

Video Object Sequence

Visual Object

Video Object Layer

Group of Video Object Plane

Video Object Plane

Encoder/Decoder Structure

InputVOP

Definition

Coding

MUX Bitstream

OutputComposition

Decoding

DEMUXBitstream

VOP Encoder

Coding

Motion

Estimation

Motion

Compensation

Texture

Coding

Previous Reconstructed

Buffer

Motion

Coding

VOP Formation

Control MB

Tightest Rectangle

Extended

Bounding

Intelligently generated VOP

: control point

Object

Code the boundary of each

block with shape coding

Binary Shape Coding

Context-based arithmetic encoder (CAE)

Basic idea

operates on the macroblock level

compute a context from neighboring pixels, 8 or 9 bit integer

based on context, use LUT to get probability (pixel is either 0 or 1)

obtain sequence of probabilities that drive an arithmetic encoder

C9 C8 C7

C5 C4 C3C6 C2

C0 xC1

C6 C5C7

C3 C2 C1

Previous Current

Lossy Shape Coding

CAE is able a achieve a lossless representation

For rate reduction, MPEG-4 allows the encoder to sub-sample the blocks by

a factor of 2 or 4 - lossy shape coding

distortion is the difference btwn the original and up-sampled block

must also transmit the conversion ratio

downsampling

upsampling

conversion error

P-VOP Motion

Estimation/Compensation

Basic techniques

motion estimation (ME) modified for arbitrarily shaped VOP

full and half pixel motion vectors, Intra/Inter decisions

Padding Process

pixels outside of VOP boundary must be padded before ME

padded pixels are not included in matching process

Advanced prediction

16x16, 8x8, and field predictions

16x16 mode

(1 MV)8x8 mode

(4 MV’s)

Differential Coding of MV‘s for P-

MV2MV3

MV3MV2

MV1 MVMV1

MV2MV3

MV3MV2

For 16x16 mode For 8x8 mode

MVDx = MVx - Px

Px = median(MV1x, MV2x, MV3x)

Texture Coding

Basic techniques

DCT with motion compensation as in MPEG-2

VLC tables for DC coefficients

run-length coding and VLC tables for AC coefficients

New methods

Intra DC/AC prediction for I- and P-VOPs

texture coding for arbitrarily shape VOPs

low-pass extrapolation (LPE) technique

shape-adaptive DCT (SA-DCT)

supports both H.263 and MPEG quantization methods

Adaptive DC Prediction

Choose best DC predictor based on gradients of the DC values

if (|QDCA - QDCB| < |QDCB - QDCC|)QDCX’ = QDCC

QDCX’ = QDCA

Obtain differential DC value from this best predictor

X MacroblockY

Adaptive AC Prediction

Either coefficients from

the first row or the first

column of a previous

coded block are used to

predict the co-sited

coefficients of the current

The best direction is

chosen from the direction

of the DC prediction

Macroblock

Q-Step Scaling for AC Prediction

To compensate for differences in the quantization of previous horizontaly or

vertically adjacent blocks used in AC prediction of the current block, scaling

of prediction coefficients becomes necessary

For example, if block A was chosen to predict block X

Note, complexity increase due to division and storage of previous row of

AC coefficients

i A Ai X

QAC QPQAC

Low-Pass Extrapolation Padding125

3 types of MB‘s in a VOP with arbitrary shape completely located inside VOP (no special treatment)

completely located outside VOP (skipped, 1 bit)

blocks that lie on the boundary (need padding before DCT)

LPE padding for intra blocks only Step 1: assign pixels outside the VOP boundary the mean value of pixels inside

the VOP

Step 2: beginning from upper-left, process outside pixels row by row, taking the

average of 4 pixel values

This process is intended to fill in the undefined pixel

values, while not adding significant energy to HF

Shape-Adaptive DCT (SA-DCT)

The SA-DCT algorithm is based on predefined orthonormal sets of DCT basis functions

Apply 1D DCT vertically and horizontally according to the number of active pixels in the row and column of the block

Final number of the SA-DCT coefficients is identical to the number of active pixels of image

Zigzag scan is modified so that non-active coeffs are neglected

Column

Active image pixels Coefficients of Column DCTs SA-DCT result

Error Resilience: Resynchronization

Enable resynchronization between the decoder and bitstream after an error has been detected

Packet approach: provide periodic resync markers based on the number of bits within a packet, not the

number of MB‘s in a packet

header information is contained at the start of a packet so that decoding can be restarted

all predictively coded info must be contained in one packet to prevent error propagation

Interval synchronization to avoid start code emulation: start codes appear only at legal fixed interval locations

Error Resilience: Data Recovery

After synchronization has been reestablished, data recovery attempts to

recover data that would be lost

Reversible Variable Length Codes (RVLC)

Huffman codes are designed to be read in both directions

loss of coding efficiency, but substantial increase in error resilience

Resync

Marker

Macroblock

_number

_scale

HEC Motion & Header

Information

Motion

Marker

Texture

Information

Resync

Marker

Texture

Header

Forward Backward

Errors

Decode Decode

Error Concealment

Assuming resynchronization can localize errors, attempt to conceal errors by using available info

Data Partitioning: separate motion and texture bits

place resync marker in between

if texture is lost, use motion to conceal the error

Outside of the standard, error concealment can be done in a number of other ways for example, if motion vector is lost, try to predict it from ones that have already

been decoded

Sprite Coding

A sprite is an image composed of pixels that are visible throughout an entire video segment

e.g., sprite contains all the background pixels in a panning sequence

Initial sprite coded with I-VOP techniques, then updated

Sprites can be used to reconstruct and predict VOPs

Need to estimate warping parameters that define the relation between the sprite and pixels in a VOP (global motion parameters)

Good coding efficiency for scenes with global motion

Object-Based Scalable Encoding

Spatial Scalability

Temporal Scalability

Enhancement layer

Base layer

frame number

Base Layer

0 6 12

Enhancement

frame number0 6 122 4 8 10

Wavelet-based Texture Coding

Still Texture Object

Decomposition of image using DWT

high coding efficiency

excellent for spatial and SNR scalability

Quantization of wavelet coefficients

LL band is coded using DPCM

higher order band are coded using zero-tree

Entropy coding using adaptive arithmetic encoder

QUANTZeroTree

ScanningAC

QUANT AC

Bitstream

PredictionLow-Low

2D Mesh and Face Animation

ES0IRISD0

Mesh Objects

tessellation of 2D planar region into polygon patches

vertices of mesh are referred to as node points

node points are warped from one frame to the next; motion information is coded

• Face Objects

– shape, texture and expressions

of the face are controlled by

Facial Animation Parameters

– FAPs can also be used for

accurate speech articulation,

where visemes are used to

code lip configurations and

mood of the speaker

Amendments to Visual Part

2000 Edition issued in Jan‘01 (version1 + version2) included in the 2000 edition are:

all tools/techniques discussed so far

additional coding efficiency tools (global motion, 1/4-pel motion, SA-DCT)

increased flexibility in object-based scalable coding

improved error robustness; NEWPRED tool that switches ref frames

dynamic resolution conversion

Amendment 1 tools to support Studio Profile; spec has been frozen Jan‘01

Amendment 2 tools to support Streaming Video Profile; spec has been frozen Jan‘01

major addition: Fine Granularity Scalability (FGS) where enhancement layer is

bit-plane coded

Amendment 1 - Studio Profile

Objectives Object-based techniques for video creation

Higher coding efficiency for studio storage

Applications Professional broadcast, Studio and post production, Inter-studio transmission

Requirements Formats - 4:2:2, 4:4:4 (YUV and RGB), progressive and interlaced

Resolutions - up to 2048 by 2048 pixels per VOP

Bit-rates - up to 1.2 Gbps bitrate with up to 12 bits pixel depth

Lossless coding capability

Support for Binary Shape, Grayscale Shape for Alpha transparency, depth, displacement, Sprites

New Tools in Studio Profile

High efficient VLC for high bitrate

Grouping of DCT coefficients based on their values

Recursive selection of VLC table for groups of coefficients as function of

previously coded group

Coded data = group indicator + fixed length code determining the actual value

within the group

Flexible access by special slice structure

New tools for lossless coding

Profile Definitions

Simple Studio Profile

To be applied for image acquisition and editing

Only Intra coding for independent processing of frames

Lossless transcoding from MPEG-2 4:2:2 Profile

Support for Arbitrary Shape (binary or grayscale)

Core Studio Profile

To be applied for inter-studio transmission

Inter (P-VOP) coding for more efficient compression

Support for Sprites

Amendment 2 - Streaming Video

Profiles

Encoder DecoderChannel

Traditional Model of a Communication System

Internet Streaming Applications

Encoder

Server DecoderChannel

Server

Channel

Decoder

Basic assumptions of traditional model:

Encoder knows channel capacity

Decoder is able to process all received bits

New Objective for Video Coding

Channel Bandwidth

Received

Quality

Traditional

Source

Coding

Objective

Moderate

HighLow

Traditional

Distortion-Rate

Fine Granularity Scalability

Motion compensated DCT coding in base layer to reach lower bound of

bitrate range

Bitplane coding of DCT coefficients in enhancement layer to cover bitrate

Enhancement layer bitstream may be truncated into any number of bits per

Decoder may ignore some enhancement bits

Reconstructed video quality proportional to number of decoded bits

Basic Encoder Structure

Motion

Compensation

Motion

Estimation

Memory

VLCInput Video

Base Layer

Bitstream

Bit-plane

Maximum

Bit-plane

VLC Enhancement

Bitstream

Enhancement Layer Encoding

Clipping

Basic Decoder Structure

VLD Q-1 IDCT

Motion

Compensation

Memory

Bit-plane

VLDIDCT

Enhancement Layer Decoding

Base Layer

Bitstream

Enhancement

Bitstream

Base Layer Video

(optional output)

Enhancement VideoClipping

Clipping

Bit-plane

Basic Bitplane Coding Technique

Bit-Plane

A block of 8x8 DCT coefficient differences

Zigzag ordering of a block of 8x8 DCT

coefficient differences

Bit-Plane

A block of 8x8 DCT coefficient differences after zigzag ordering

10 0 0 0 0 0 0

18 zeros12 zeros

20 zeros

Bit-Plane

(RUN, EOP) symbols for a block

of 8x8 DCT coefficient differences

after zigzag ordering

(0, 1)

(28, 1)

(6, 0)

(0, 0) (0, 0) (26, 1)

(2, 0) (31, 1)

Application: Varying BW Environment

Source

ConsumerConsumer

SourceConsumer

Bandwidth

User Variation

Temporal

Variation

User Variation: bandwidth varies from user to user

Temporal Variation: bandwidth varies with time

Multiplexer CBR Channel

FGS Enhancement Layer Encoder

VBR Base Layer Encoder

Data Server

FGS Enhancement Layer Encoder

VBR Base Layer Encoder

Video 1

Video N

Base 1

Base N

Application: Statistical Multiplexing

MPEG-4 System Layer Model

SL SL SL

TransMux Layer

FlexMux

TransMux Streams

FlexMux Channel

TransMux Channel FlexMux Streams DMIF Network Interface

DMIF Application Interface

Elementary Stream Interface

SL-Packetized Streams

Elementary Streams

FlexMux

Sync Layer

DMIF Layer

SL SLSL

FlexMux

MuxFile Broad-

Inter-

active (not specified in MPEG-4)

Profile Definitions of Version 1

Simple Profile

Bssic tools of I/P VOP AC/DC Prediction and 4 MV unrestricted

Short header and Error Resilience tools

Core Profile

Simple + Binary Shape, Quantization Method 1/2 and B-VOP

Main Profile

Core + Grey Shape, Interlace and Sprite

Simple Scalable Profile

Simple + Spatial and temperal scalability and B-VOP

N-Bit Profile

Core + N-Bit

Animated 2D Mesh

Core + Scalable Still Texture, 2D dynamic Mesh

Basic Animated Texture

Banary Shape, Scalable Still Texture and 2D Dynamic Mesh

Still Scalable Texture - Scalable Still Texture

Simple Face - Face Animation Parameters

Profile Definitions of Version 2 Advanced Real Time Simple Profile

Simple +

Advanced error resilience with back channel,

improved temporal scalability with low buffering delay

Core Scalable Profile

Simple scalable +

Core +

SNR, Spatila/Temporal Scalability for Region or Object of interest

Advanced Coding Efficiency Profile

Tools for improving coding efficiency for both rectangular and arbitrary shaped objects

For applications such as mobile broadcast reception

Advanced Scalable Texture Profile

Tools for decoding arbitrary shaped texture and still image including scalable shape coding

Advanced Core Profile

Core Profile +

Tools for decoding arbitrary shaped video objects and arbitrary shaped scalable still image

Simple Face and Body Animation Profile

Simple face animation + body animation

Profile Definitions in subsequent version

Advanced Simple Profile

Simple Profile +

Several tools to make it more efficient:

B-frames

1/4 pel motuon compensation

Extra quantization tables

Global Motion Compensation

Fine Granularity Scalable Profile

Use Advanced Simple Profile as base layer

Fine granularity scalability (FGS)

Fine granularity scalability - temporal (FGST)

Simple Studio Profile

I-frames only

Arbitrary shape

Multiple alpha channels

Up to 2 Gbps

Core Studio Profile

Simple Studio Profile + P-frames

MPEG-4 Video Profiles @ Levels

Spatial

Temporal

Scalability

Arbitrary

Rectangular

Scalability

Quality

Temporal

Scalability

Additional

Higher

Resilience

Simple

Scalable

Advanced

Simple

Advanced

Coding

Efficiency

Granularity

Scalable

Advanced

Realtime

Simple

Studio

Additional

Profiles are used to limit the set of tools in a decoding device

Levels are used to place limits on complexity

Visual Object Types/Tools 0f V.2

Visua Tools Visual Object Types

Advanced Real

Time Simple

Advanced Coding

Efficiency

Advanced Scalable

Texture

Core Scalable Simple FBA

•I/P-Vop

•AC/DC Prediction

•4-MV, Unrestricted MV

Error Resilience

•Slice resynchronization

•Data partitioning

•Reversible VLC

Short Header X X X

B-VOP X X

P-VOP with OBMC (Texture) X X

Method 1/Method 2

Quantization

P-VOP based Temporal

Scalability

•Rectangular

•Arbitrary Shape

Binary Shape X X

Grey Shape X

Interlace X

Sprite

Visual Object Types/Tools 0f V.2

(Cont‘d)Visua Tools Visual Object Types

Advanced Real

Time Simple

Advanced Coding

Efficiency

Advanced

Scalable Texture

Core Scalable Simple FBA

Temporal Scalability (Rectangular) X

Spatial Scalability (Rectangular) X

Scalable Still Texture X

2D Dynamic Mesh with uniform

topology

2D Dynamic Mesh with Delaunay

topology

Facial Animation Parameters X

Body Animation Parameters X

Dynamic Resolution Conversion X

NEWPRED X

Global Motion Compensation X

¼ Pel motion Compensationn X

SA-DCT X

Error Resilience for Visual Texture

Coding

Wavelet Tiling X

Scalable Shape Coding for Still Texture X

Object Based Spatial Scalability X

Visual Profiles of V.1

Object Types Simpl

Simple

Scalable

Core Core

Scalable

Advance

d Real

Simple

Advance

d Coding

Efficienc

Advanced

Scalable

Texture

Simple

Profiles

V2-1 Advanced

Real Time

Simple

V2-2 Core Scalable X X X X

V2-3 Advanced

Coding

Efficiency

V2-4 Advanced

V2-5 Advanced

Scalable

Texture

V2-6 Simple FBA X

What‘s New in MPEG-4 Visual

New Video Codec: ―MPEG-4 Advanced Video Coding‖

It is developed by JVT (Joint Video Team) of ITU-T and MPEG

Major task is the coding performance improvement

It is based on H.26L

It will be Part 10 of MPEG-4 at MPEG

It probably will be H.264 at ITU-T

Animation Framework eXtension - AFX

High level description of anumation

Enhanced rendering

Compact representations

Low bit rate animations

Scalability based on terminal capabilities

Interactivity at user level, scene level and client-server session level

Compression of representations for static and dynamic tools

3D Video Coding

Interframe Wavelet Coding

MPEG-4 Advanced Video Coding (1)

Summary of Fairfax meeting

A total of 160 proposals have been submitted to JVT

Working draft WD.1 has been created

Two profiles: baseline profile and main profile have been decided

Baseline profile to be royalty free, main decoding features include:

I, P pictures

In loop deblocking filters

Interlace support (Level dependent, Level 2.1 or above)

1/4 pel motion prediction

Tree-structured motion segmentation down to 4x4

VLC-based entropy coding

Flexible Macroblock ordering

Main profile

Including all features in baseline profile and adds

B-pictures,

CABAC (Content Adaptive Binary Arithmetic Coding)

Adaptive Block-size Transforms

1/8-sample motion compensation

Notes on Profiles and Levels

Motion vector range will be limited

A limit is imposed on extreme aspect ratios

Number of reference pictures increases with picture size, never exceeding 15

Exact values of motion vector range limit

Smaller than 8x8 bi-predictive motions in B-pictures for Main profile

Adaptive B-picture interpolation in Main profile

Unfulfilled requirements

No 4:2:2 source format support (pending further study)

Mixing Intra and Inter coding type within macroblocks

Data partitioning

SP & SI ("switching" pictures)

Level's summarized with typical format as follows:

Level 1 = QCIF @ 15 (Intermediate levels 1.1 = CIF @ 7.5, 1.2 = CIF @ 15)

Level 2 = CIF @ 30, (Intermediate levels 2.1 = HHR and 2.2)

Level 3 = SDTV (Intermediate levels 3.1, 3.2)

Level 4 = HDTV

Level 5 = SHDTV (1920x1088 @ 60p)

Technical summary

Order of bitstream within MB: total seven modles

0 0 1 0 1

16x16 16x8 8x16 8x8

8x8 8x4 4x8

MB-Modes

8x8-Modes

CBPY 8x8 block order

Luma residual coding 4x4 block order

2x2 DC

Chroma residual coding 4x4 block order

Motion compensation

Motion vector data for 1-16 blocks are transmitted

Motion vector prediction

Median prediction is used except for 16x8 or 8x16 blocks

• The prediction of E is formed as median of A, B, and C

Directional segmentation prediction

• Vector block size 8x16

Left block: A is used if it has same reference picture as E, otherwise "median prediction" is used

Right block: C is used if it has same reference picture as E, otherwise "median prediction" is used

• Vector block size 16x8:

Upper block: B is used as prediction if it has same reference picture as E, otherwise "median

prediction" is used

Lower block: A is used as prediction if it has the same reference picture as E, otherwise "median

prediction" is used

16x88x16

Reference pictures

Default reference field number assignment when the current picture is first field coded

Default reference field number assignment when the current picture is second field coded

current field0 12 34 5

Ref. Frame (field) Buf.

Ref. Field No.

......

f1 f2f1 f2f1 f2f1 f2f1 f2f1 f2 f1 f2

6 78 910 11

current field0 12 34 5

Ref. Frame (field) Buf.

Ref. Field No.

......

f1 f2f1 f2f1 f2f1 f2f1 f2f1 f2 f1 f2

6 78 910 11

Intra prediction: two intra prediction modes

Intra prediction modes for 4x4 of luma

Mode 0: DC prediction

Mode 1: Vertical Prediction

Mode 2: Horizontal prediction

Mode 3: Diagonal Down/Right prediction

Mode 4: Diagonal Down/Left prediction

Mode 5: Vertical-Left prediction

Mode 6: Vertical-Right prediction

Mode 7: Horizontal-Up prediction

Mode 8: Horizontal-Down prediction

Intra prediction for 16x16 mode for luma

Mode 0: Vertical

Mode 1: Horizontal

Mode 2: DC prediction

Mode 3: Plane prediction

Adaptive Block size Transforms (ABT)

Use of ABT to increase coding efficiency

ABT is synchronized with Motion Compensation

for frame motion compensation, ABT applied to frame MBs

for field motion compensation, ABT applied to field MBs

ABT transform coefficient decoding

Progressive scan

Interlaced scan

4x8 8x8

1 3 9 13

2 6 10 14

4 7 11 15

5 8 12 16

1 5 13 21

2 6 14 22

3 7 15 23

4 12 20 28

8 16 24 29

9 17 25 30

10 18 26 31

11 19 27 32

1 3 7 11 15 19 23 27

2 6 10 14 18 22 26 30

4 8 12 16 20 24 28 31

5 9 13 17 21 25 29 32

1 4 9 16 23 31 39 53

2 5 15 22 30 38 46 54

3 8 17 24 32 40 47 59

6 10 21 29 37 45 52 60

7 14 25 33 41 48 55 61

11 18 26 34 42 49 56 62

12 19 27 35 43 50 57 63

13 20 28 36 44 51 58 64

4x44x8

8x48x8

Content-based Adaptive Binary Arithmetic Coding (CABAC)

Context modeling: provides estimates of conditional probabilities of the coding symbols, utilizing

suitable context models, given inter-symbol redundancy can be exploited by switching between

different probability models according to already coded symbols

Arithmetic codes: permit non-integer number of bits to be assigned to each symbol of the alphabet, this

is extremely beneficial for symbol probabilities much greater than 0.5, which often occur with efficient

context modeling. This is extremely beneficial for symbol probabilities much greater than 0.5, which

often occur with efficient context modeling. In this case, a variable length code has to spend at least one

bit in contrast to arithmetic codes, which may use a fraction of one bit

Adaptive arithmetic: codes permit the entropy coder to adapt itself to non-stationary symbol statistics,

For instance, the statistics of motion vector magnitudes vary over space and time as well as for different

sequences and bit-rates. Hence, an adaptive model taking into account the cumulative probabilities of

already coded motion vectors leads to a better fit of the arithmetic codes to the current symbol statistics

Other techniques for image quality and encoding performance improvement

In loop deblocking filter

a conditional filtering is applied to boundaries of the 4x4 blocks of a reconstructed MB

in the first step, 16 pel of the 4 vertical edges (horizontal filtering) of the 4x4 raster are filtered

after that, 4 horizontal edges (vertical filtering) follow.

Encode optimization

Using R-D optimizations

Finding optimum prediction mode

the best reference frame

the best motion vectors

fractional pel accuracy

Macroblock level optimum mode decision

decision between intra and inter

adaptive block size

Conclusions

MPEG-4 visual standards overviewed

MPEG-4 is the first standard which can be used for object-based coding

Simple profile of MPEG-4 has been used for wireless and internet video transmission

MPEG-4 is expected to be major coding scheme for multimedia applications

Several parts of MPEG-4 designed for special applications such as FGS for video streaming, AVC for increasing coding efficiency (may be used for HD DVD with red laser)

For Further Information

MPEG-4 Industry Forum

http://www.m4if.org/

MPEG Home Page

http://mpeg.nist.gov/

IEEE Trans. CSVT special issues:

Feb ‗97: on MPEG-4

Nov ‗98: on representation/coding of images/ video (part I)

Feb ‗99: on representation/coding of images/ video (part II)

Dec ‗99: on object-based coding

Mar ‗01: on streaming video

Video Coding - National Chiao Tung Universitycmliu/Courses/Compression/...Video Coding C.M. Liu...

Documents

VLSI and SOC Systems - National Chiao Tung University

Ad Hoc Networks - National Chiao Tung University

National Chiao Tung University Chun-Jen Tsai 6/8/2012people.cs.nctu.edu.tw/~cjtsai/courses/ics/classnotes/ics12_11... · National Chiao Tung University Chun-Jen Tsai 6/8/2012. 2/24

Security - National Chiao Tung University

Chapter 6 Internet QoS - National Chiao Tung University

Marketing Research Course National Chiao Tung University

Implementation of DSP IC - National Chiao Tung University

Chapter 5 Topological Graph Theory - National Chiao Tung

Duct Design - National Chiao Tung University

Chap 28 半導體製程 - National Chiao Tung University

Yi-Bing Lin, National Chiao Tung University Abstract · 2014-12-09 · OA8GM for the GSM Network Yi-Bing Lin, National Chiao Tung University Abstract This article provides a road

ěǚfhněǚ ǛJ - National Chiao Tung University

Chapter 2 Link Layer - National Chiao Tung University

Web - National Chiao Tung University

National Chiao Tung University Office Of International Affairs

Prof. Ming-Jer Chen Department of Electronics Engineering National Chiao-Tung University

National Chiao Tung University Chun-Jen Tsai 05/4/2012people.cs.nctu.edu.tw/.../ics12_06_Programming_Languages.pdf · National Chiao Tung University Chun-Jen Tsai 05/4/2012. ... Lexical

LDAP - National Chiao Tung University

cognitive radio channels IEEE - National Chiao Tung University

Prof. Ming- Jer Chen Department of Electronics Engineering National Chiao -Tung University