Multimedia SynchronizationI. Fatimah Alzahrani

Definitions•Multimedia System :A system or application that supports the

integrated processing of several media types.

•Multimedia Synchronization : Synchronization in multimedia system

refers to the temporal relations between media objects in a multimedia system.

Data StreamsData Stream is any sequence of individual

packets transmitted in a time-dependent fashion

– Packets can carry information of either continuous or discrete media.

Transmission modes :Asynchronous.Synchronous.

Asynchronous • In asynchronous transmission, the transmitter

transmits data bytes at any instant of time.• Only one byte is sent at a time. There is ideal

time between two data bytes.• Transmitter and Receiver operate at different

clock frequencies.• To help receiver ‘start’ and ‘stop’ bits are used

along with data in middle.• Ideal time between byte is not constant. They are

also known as gaps.• In asynchronous transmission timing of signal is

not important.

Synchronous • In synchronous transmission, data transmission

is carried out under the control of a common master clock.

• Bytes are transmitted as a block in a continuous stream of bits.

• Transmitter and Receiver operate at synchronised clock frequencies.

• No ‘start’ and ‘stop’ bits are used.• No need of ideal time between data bytes.• In synchronous transmission timing of signal is

important.

•Two types of media data :1.Continuous : Time-Dependent Ex :- Real-Service (Live Video).

2.Discrete : Time-IndependentEx :- Text or Graphic.

•Three types of relationships :1. Content.2.Spatial.3.Temporal.

Content Relationships•Content Relation: defines dependency of

media objects for some data.

Example: dependency between a filled spreadsheet and graphics that represent data listed in spreadsheet.

Spatial Relationship• Spatial relation is represented through layout relation and defines space used for presentation of a media object on an output device

at a certain point of time in a multimedia document.

• Example: desktop publishing Layout frames are placed on an output device and a content is

assigned to this frame Positioning of layout frames:

Fixed to a position of a document Fixed to a position on a page Relative to the position of other frame

• Concept of frames is used for positioning of time-dependent objects Example: in window-based systems, layout frames correspond to

windows and video can be positioned in a window

Temporal Relationship•Temporal relation defines temporal

dependencies between media objects.Example: lip synchronization

•Time-dependent objects – represent a media stream because there exists temporal relations between consecutive units of the stream

Cont. Temporal Relations•Temporal synchronization is supported by

many system components:OS (CPU scheduling).Communication systems (traffic shaping,

networkscheduling.)

•1.Intra-Media Synchronization:Time Relation Between Various Presentation Units

of One Time-Dependent Media Object.

Ex : Time relation between the single frames of a video sequence. For a video witha rate of 25 frames per second each of the frames has to be displayed for 40 ms .

Frames of a video sequence that shows a jumping ball :

2.Inter-Media Synchronization : The Synchronization Between Media Objects .

Ex : Time Relations of A Multimedia Synchronization That Starts With Audio/Video Sequence , Followed By Several Pictures and An Animation That is Commented By An Audio Sequence .

Logical Data Unit (LDU) Hierarchy :• LDU is the unit of a media stream. Different

streams have different LDU size.

•Closed LDU▫Predictable duration▫Ex: Stored media objects

•Open LDU▫Duration not known before the

presentation starts▫Ex: Input from live source

Logic Data Unit•Physical units as LDU or blocks of

samples as LDU

Logic Data Unit •User-defined vs capturing defined.

Live Synchronization•Goal is to exactly reproduce at a

presentation the temporal relations as they existed during the capturing process

•Need to capture temporal relation information during the capturing

•Live Sync is needed in conversational services Video Conferencing, Video Phone.

Lip Synchronization•Lip synchronization

▫Coupling between audio and video.

Reference Models for Multimedia Synchronization

• Focus on the following four-layer model Specification Layer - contains applications and tools for

creating synchronization specification.It can be classified into :

(time) Interval-based specification Axes-based specification Control flow-based specification (given synchronization points) Event-based specification .

Object Layer - presentation object synch Stream Layer - inter-stream synch of media streams Media Layer - intra-stream synch of continious media

Synchronization Specification Methods•Interval-based Specification

presentation duration of an object is considered as interval

Examples of operations: A before(0) B, A overlap B, A starts B, A equals B, A during B, A

while(0,0) B

Advantage: easy to handle open LDUs ad therefore user interactions

Disadvantage: complex specifications

Audio1 while (0,0) Video Audio1 before RecordedInteraction RecordedInteraction before (0) B1 P1 before (0) P2 P2 before (0) P3 P3 before (0) Interaction P3 before (0) Animation Animation while (2,5) Audio2 Interaction before (0) P4.

Synchronization Specification Methods•Control Flow-based Specification –

Hierarchical Approach.Basic hierarchical specification: 1. serial

synchronization, 2. parallel synchronization of actions.

Advantage: easy to understand, natural support of hierarchy and integration of interactive objects is easy

Disadvantage: Additional description of skews and QoS is necessary, we must add presentation durations

Synchronization in Distributed Environment(1)

• Information of synchronization must be transmitted with audioand video streams, so that the receiver side can synchronize the

streams

• Delivery of complete sync information can be done before thestart of presentation

This is used in synthetic synchronization Advantage: simple implementation Disadvantage: presentation delay

• Deliver of complete sync information can be using out-of-bandcommunication via a separate sync channel

This is used in live synchronization Advantage: no additional presentation delays Disadvantage: additional channel is needed; additional errors can occur

Synchronization in Distributed Environment(2)

• Delivery of complete synchronization information can be done using in-band communication via multiplexed data streams, i.e., synchronization information is in headers of the multimedia PDU Advantage: related sync information is

delivered together with media units Disadvantage: difficult to use for multiple

sources

Synchronization in Distributed Environment(3)(Location of Synchronization Operations)

• It is possible to synchronize media objects by recording objects together and leave them together as one object, i.e.,

combine objects into a new media object during creation; Synchronization operation happens then at the recording site.

• Synchronization operation can be placed at the sink. In this case the demand on bandwidth is larger because additional

sync information must be transported.

• Synchronization operation can be placed at the source. In this case the demand on bandwidth is smaller because the streams are multiplexed according to synchronization requirements.

Synchronization in Distributed Environment(4)(Clock Synchronization)

• Consider synchronization accuracy between clocks at source and destination.

• Global time-based synchronization needs clockSynchronization.

• In order to re-synchronize, we can allocate buffers at the sink and start transmission of audio and video in advance, or use NTP (Network Time Protocol) to bound the maximum clock offset.

Audio1 while (0,0) Video Audio1 before RecordedInteraction RecordedInteraction before (0) B1 P1 before (0) P2 P2 before (0) P3 P3 before (0) Interaction P3 before (0) Animation Animation while (2,5) Audio2 Interaction before (0) P4.

Multimedia Applications•Multimedia plays major role in following

areas•Instruction•Business

Advertisements Training materials Presentations Customer support services

•Entertainment Interactive Games

Multimedia Applications•Enabling Technology

Accessibility to web based materials.Teaching-learning disabled children &

adults.

•Fine Arts & HumanitiesMuseum tours.Art exhibitions. Presentations of literature.