Fine Granularity Fine Granularity Scalability in MPEG-Scalability in MPEG-

4 Video4 Video

by Weiping Liby Weiping LiPresentation by Warren CheungPresentation by Warren Cheung

Fine Granularity ScalabilityFine Granularity Scalability

● Amendment to MPEG-4Amendment to MPEG-4● Bitplane coding of the DCT Bitplane coding of the DCT

coefficientscoefficients

MotivationMotivation

● Video optimised for bit rate rangeVideo optimised for bit rate range

1.1. Limited Channel CapacityLimited Channel Capacity

2.2. Partial DecodingPartial Decoding

Nonscalable EncodingNonscalable Encoding

● Fixed bit rateFixed bit rate● Bandwidth efficiency depends on:Bandwidth efficiency depends on:

1.1. Channel bit rateChannel bit rate

2.2. Encoded bit rateEncoded bit rate

Techniques in MPEGTechniques in MPEG

● MPEG-2, MPEG-4 MPEG-2, MPEG-4 ● incorporate layered scalability incorporate layered scalability

techniquestechniques1.1. SNR scalabilitySNR scalability

2.2. Temporal scalabilityTemporal scalability

3.3. Spatial scalabilitySpatial scalability

SNR scalabilitySNR scalability

● Signal-to-noise ratio (SNR)Signal-to-noise ratio (SNR)● Encode two layers at same frame Encode two layers at same frame

rate and spatial resolutionrate and spatial resolution● Use different quantization accuracyUse different quantization accuracy

SNR scalability(2)SNR scalability(2)

● Two layers: base layer, enhancement Two layers: base layer, enhancement layerlayer

● Base layer decodes DCT coefficients Base layer decodes DCT coefficients as beforeas before

● Can decode and add enhancement Can decode and add enhancement layer DCT coefficients layer DCT coefficients

● Drift can occurDrift can occur

DriftDrift

● Drift occurs in the Drift occurs in the base layerbase layer if if● Enhancement layer information used in Enhancement layer information used in

motion prediction loop motion prediction loop andand● Enhancement layer is not decoded by the Enhancement layer is not decoded by the

decoderdecoder

1.1. Drift occurs in the Drift occurs in the enhancement layer enhancement layer ifif

1.1. Enhancement layer information not used in Enhancement layer information not used in motion prediction loop motion prediction loop andand

2.2. Enhancement layer is not decodedEnhancement layer is not decoded

Temporal ScalabilityTemporal Scalability

● Two layers at different frame ratesTwo layers at different frame rates● Base layer with only P-type Base layer with only P-type

predictionprediction● Enhancement layer adds Enhancement layer adds

1.1. Extra P and B frames based on base Extra P and B frames based on base layerlayer

2.2. Extra P frames based on enhancement Extra P frames based on enhancement layerlayer

Spatial ScalabilitySpatial Scalability

● Two layers same frame rate but Two layers same frame rate but different spatial resolutiondifferent spatial resolution

● Base layer has lower resolutionBase layer has lower resolution● Enhancement layer upsamples the Enhancement layer upsamples the

base layer and for predictionbase layer and for prediction● No drift occurs if enhancement layer No drift occurs if enhancement layer

is not decoded (but unavailable for is not decoded (but unavailable for prediction of base layer)prediction of base layer)

PerformancePerformance

● Enhancement layer must be Enhancement layer must be completely decoded, otherwise it is completely decoded, otherwise it is uselessuseless

● Jagged “stair” performance curveJagged “stair” performance curve● Want to be able to truncate the Want to be able to truncate the

enhancement bitstream and still enhancement bitstream and still provide partial enhancementprovide partial enhancement

FGS streamingFGS streaming

Bit-Plane CodingBit-Plane Coding

● Two Layers: base and enhancementTwo Layers: base and enhancement● DCT coefficients normally encoded DCT coefficients normally encoded

using run-level encodingusing run-level encoding● Use Bit-Plance Coding for Use Bit-Plance Coding for

enhancement layerenhancement layer● Reorganise the bitstream such that Reorganise the bitstream such that

even if it is truncated, we get a even if it is truncated, we get a usable set of DCT coefficientsusable set of DCT coefficients

Sending NumbersSending Numbers

● Given a set of numbersGiven a set of numbers● Which bits are the most significant?Which bits are the most significant?

Bit-Plane Coding ExampleBit-Plane Coding Example

● Suppose we wish to send:Suppose we wish to send:● 10,0,6,010,0,6,0

1.1. 10 is the largest number and 10 is the largest number and requires 4 bits so we have 4 bit-requires 4 bits so we have 4 bit-planesplanes

1.1. 1, 0, 0, 0 (MSB)1, 0, 0, 0 (MSB)

2.2. 0, 0, 1, 00, 0, 1, 0

3.3. 1, 0, 1, 01, 0, 1, 0

4.4. 0, 0, 0, 00, 0, 0, 0

Bit-Plane Coding of Bit-Plane Coding of DCT CoefficientDCT Coefficient

● We apply Bit-Plane coding for each We apply Bit-Plane coding for each 8x8 block of DCT coefficients8x8 block of DCT coefficients

● Bit-Planes are encoded using (RUN, Bit-Planes are encoded using (RUN, EOP) symbolsEOP) symbols

1.1. RUN: Number of zeroes before next oneRUN: Number of zeroes before next one

2.2. EOP: Is this the last one in the planeEOP: Is this the last one in the plane

Additional DetailsAdditional Details

● Each of the YUV color components Each of the YUV color components may have a different number of bit-may have a different number of bit-planes in the same frameplanes in the same frame

1.1. Y = luminance (black and white)Y = luminance (black and white)

2.2. U, V = chrominance (color difference)U, V = chrominance (color difference)

1.1. MSB (most significant bit plane) is MSB (most significant bit plane) is the first plane that is not all zeroesthe first plane that is not all zeroes

Advanced FeaturesAdvanced Features

● Frequency Weighting: Low frequency Frequency Weighting: Low frequency DCT coefficients are more important DCT coefficients are more important than high-frequency coefficients and than high-frequency coefficients and should be transmitted firstshould be transmitted first● Shift the bits for DCT coefficients up/downShift the bits for DCT coefficients up/down

1.1. Parts of the picture may be more Parts of the picture may be more significantsignificant

1.1. Shift the bit-planes for these elements upShift the bit-planes for these elements up

Additional Features(2)Additional Features(2)

● Error Resilience: Error Resilience: ● Resynchronization markersResynchronization markers● bit-plane start codebit-plane start code

1.1. Temporal ScalabilityTemporal Scalability1.1. Add additional frames that can be Add additional frames that can be

decodedto improve bitratedecodedto improve bitrate

2.2. Predictions not allowed in the additional Predictions not allowed in the additional frames, but quantization can be scaled frames, but quantization can be scaled and bit-plane coding can also be usedand bit-plane coding can also be used

ReferencesReferences

● Scalability Extensions (brief Scalability Extensions (brief overview)overview)● http://www.cs.ucl.ac.uk/staff/jon/http://www.cs.ucl.ac.uk/staff/jon/

mmbook/book/node133.htmlmmbook/book/node133.html● FGS and applicationsFGS and applications

● http://www.eesi.tue.nl/VCA/projects/http://www.eesi.tue.nl/VCA/projects/fine_granular_scalability/fine_granular_scalability/

Questions and CommentsQuestions and Comments

Advantages/Disadvantages of FGS Advantages/Disadvantages of FGS (and the other techniques)(and the other techniques)

Applications and Related IssuesApplications and Related Issues

Future Directions for improvementFuture Directions for improvement Wavelet based methodsWavelet based methods

Discussion TopicsDiscussion Topics● OverheadOverhead

● Noticeable degradation of image qualityNoticeable degradation of image quality● Quantization is only a coarse way to Quantization is only a coarse way to

manipulate the image quality (powers of two)manipulate the image quality (powers of two)● Many people share similar connectionsMany people share similar connections

● Vary due to path through network, physical Vary due to path through network, physical locationlocation

● Wireless connectionWireless connection● Other people on the networkOther people on the network● Unreliable bandwidthUnreliable bandwidth

Discussion(2)Discussion(2)● New MPEG standardNew MPEG standard

● Soliciting solutions to scalability issuesSoliciting solutions to scalability issues● Future work may be based on wavelatesFuture work may be based on wavelates

● No current mainstream video format No current mainstream video format currently uses waveletscurrently uses wavelets

● Used in static image format JPEG-2000Used in static image format JPEG-2000