MPEG-4 streams

Comp-ress

Comp-ress

Comp-ress

Comp-ress

DE LIVERY

INTERFACE

(DAI)Comp-ress

Scene Des.

decomp-ress

decomp-ress

decomp-ress

decomp-ress

DE LIVERY

INTERFACE

(DAI)

decomp-ress

COMPOSITOR

Obejct-des

Synchronization of elementary streams with the scene

Synchronization of multiple elementary streams

Based on two well known concepts

_______ references – Convey the speed of the encoder clock

_________ stamps – Convey the time at which an event should happen

Time stamps and clock references are – defined in the _____________ model – conveyed on the ______ layer

System Decoder Model

Object-based compression and delivery

Comp-ress

Comp-ress

Comp-ress

Comp-ress

DE LIVERY

INTERFACE

(DAI)

Comp-ressScene Des.

decomp-ress

decomp-ress

decomp-ress

decomp-ress

DE LIVERY

INTERFACE

(DAI)

decomp-ress

COMPOSITOR

System Decoder Model

decoderCompositionMemory CM1

DecoderBuffer DB1D

E LIVERY

INTERFACE

(DAI)

COMPOSITOR

decoderCompositionMemory CM1

DecoderBuffer DB1

decoderCompositionMemory CM1

DecoderBuffer DB1

decoderCompositionMemory CMn

DecoderBuffer DBn

System Decoder Model

Ideal model of the decoder behavior – Instantaneous ______________

Incorporates the__________ model – Decoding & composition time

Manage decoder buffer resources – Useful for the _______ – Ignores delivery ______

Designed for a rate-controlled “______” scenario – Applicable also to flow-controlled “pull” scenario

Defines composition memory (CM) behavior – A random access memory to the current composition unit – CM resource management not implemented

So, how does synchronization work?

Synchronization of elementary streamswith time events in the scene description

How are time events handled in the scene description?

How is this related to time in the elementary streams?

Which time base is valid for the scene description?

Time line (“object time base”) for the scene

Scene description stream with time stamped BIFS access units Object descriptor stream with pointers to all other streams

Video stream with (decoding & composition) time stamps

Audio stream with (decoding & composition) time stamps

Alternate time line for audio and video

Where are time stamps carried?

The synchronization layer

Synchronization layer (short: sync layer or SL) SL packet = one packet of data consists of header and __________

Indicates boundaries of __________ units – AccessUnitStartFlag, AccessUnitEndFlag, AULength

Provides consistency checking for ____ packets Carries object _______ reference (OCR) stamps Carries decoding and composition time stamps

(DTS, CTS)

Elementary Stream Interface

Compression Layer

SL-Packetized stream

Sync Layer

Elementary stream Access Unit AU Access Unit AU

Elementary Stream Interface

Elementary stream Access Unit AU Access Unit AU

Elementary stream Access Unit Access Unit

Elementary stream AU AUAUAU AUAU

Elementary Stream Interface

Elementary stream Access Unit Access Unit

ENCODER

This guy knows about• Access unit _________ • Random access points • Desired __________ time • The _________ time

Elementary Stream Interface

Access Unit Access Unit

ENCODER

Interface may be hidden in the Encoder

Access Unit

How is the sync layer designed?

The sync layer design

SL Packet SL Packet SL Packet

Access units are conveyed in SL packets

The sync layer design

SL P SL PacketSL Packet SL PacketSL P SL P

Access units may use more than one SL packet

The sync layer design

SL packets have a header to encode the information conveyed through the ESI

SL packets that don’t start an AU have a smaller header

SL P SL PacketSL Packet SL PacketSL P SL P

How is the sync layer designed?

As flexible as possible to be suitable for – a wide range of ______________ – a wide range of different media __________s

Time stamps have – variable length – variable _______________

Same for clock reference (OCR) values – OCR may come via another stream

Alternative to time stamps exists for lower bitrate Indication of ______ time and ________ of units (accessUnitDuration, compositionUnitDuration)

How is the sync layer flexibility signaled?

SLConfigDescriptor within ES_Descriptor Determines all variable parameters Allows to select from predefined _________s Must be known to parse an SL-packetized

stream

SLConfigDescriptor syntax example

class SLConfigDescriptor {

uint (8) predefined;

if (predefined==0) {

bit(1) useAccessUnitStartFlag;

bit(1) useAccessUnitEndFlag;

bit(1) useRandomAccessPointFlag;

bit(1) usePaddingFlag;

bit(1) useTimeStampsFlag;

uint(32) timeStampResolution;

uint(32) OCRResolution;

uint(6) timeStampLength;

uint(6) OCRLength;

if (!useTimeStamps) {

................

SDL-Syntax Description

Language

How to ship the SL packets?

Wrap SL packets in a suitable layer!

SL Packet SL Packet SL Packet

SL Packet SL Packet SL Packet

SL Packet SL Packet SL Packet

e.g. FlexMux

e.g. RTP/UDP/IP

e.g. MPEG-2 TS

MPEG-4 stream delivery (a sneak preview to DMIF ;-)

MPEG-4 delivery framework

The delivery layer

The “delivery layer” is an abstraction to refer generically to – any ________ multiplex (a. k. a. file format) – any transmission __________ – and the associated control functionality

The DMIF (Delivery Multimedia Integration Framework) part of MPEG-4 describes the _________ to any multiplex (DAI - DMIF Application Interface)

Adaptations of MPEG-4 to, e.g., RTP, MPEG-2, H.223 are being defined by the responsible group (with MPEG help)

Adaptation of the sync layer syntax

“Wrapping” of SL packets was the original design goal

MPEG-2 TS adaptation (Amendment 7 to MPEG-2 Systems)and FlexMux encapsulation really work like that

RTP encapsulation still in development – “Payload formats” specify the adaptation – They may use “Header compression”, i.e., – Syntax elements of SL are replaced by their RTP counterparts

e.g., composition time stamp --> RTP time stamp The MPEG-4 file format stores elementary streams

– conversion to SL-packetized streams during playout (if needed)

Multiplex of elementary streams

Multiplex of elementary streams

Not a core MPEG task

Just respond to specific needs for MPEG-4 content ______________

– Low delay – Low overhead – Low complexity

This prompted the design of the “___________” tool

One ___________ file format desirable – This lead to the design of the MPEG-4 file format

The MPEG-4 content access procedure

MPEG-4 content access procedure

Locate an MPEG-4 content item (e.g. by URL) and connect to it

How? – Via the _______________________– Reminder: This abstraction is being

mapped to actual signaling protocols (RTSP, MPEG-2 PSI, etc.)

Retrieve the _____________________ This Object Descriptor points to _____ +

_____________– Open these streams via DAI

The Scene Description points to other streams through __________ Descriptors

– Open the required streams via DAI Start playing!

A second attempt – more graphics