72
10C-sync.fm 1 15.February.01 Scope Contents http://www.kom.e-technik.tu-darmstadt.de http://www.tk.informatik.tu-darmstadt.de © R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing. Ralf Steinmetz Prof. Dr. Max Mühlhäuser MM: TU Darmstadt - Darmstadt University of Technology, Dept. of of Computer Science TK - Telecooperation, Tel.+49 6151 16-3709, Alexanderstr. 6, D-64283 Darmstadt, Germany, [email protected] Fax. +49 6151 16-3052 RS: TU Darmstadt - Darmstadt University of Technology, Dept. of Electrical Engineering and Information Technology, Dept. of Computer Science KOM - Industrial Process and System Communications, Tel.+49 6151 166151, Merckstr. 25, D-64283 Darmstadt, Germany, [email protected] Fax. +49 6151 166152 GMD -German National Research Center for Information Technology httc - Hessian Telemedia Technology Competence-Center e.V

Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

  • Upload
    doannhu

  • View
    221

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

u-darmstadt.de Fax. +49 6151 16-3052

, Dept. of Computer Science51 166151,M.tu-darmstadt.de Fax. +49 6151 166152

ogy

10C-sync.fm 1 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser Multimedia-Systems:

Synchronization

Prof. Dr.-Ing. Ralf SteinmetzProf. Dr. Max MühlhäuserMM: TU Darmstadt - Darmstadt University of Technology,

Dept. of of Computer ScienceTK - Telecooperation, Tel.+49 6151 16-3709,Alexanderstr. 6, D-64283 Darmstadt, Germany, [email protected]

RS: TU Darmstadt - Darmstadt University of Technology,Dept. of Electrical Engineering and Information Technology

KOM - Industrial Process and System Communications, Tel.+49 61Merckstr. 25, D-64283 Darmstadt, Germany, Ralf.Steinmetz@KOGMD -German National Research Center for Information Technolhttc - Hessian Telemedia Technology Competence-Center e.V

Page 2: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

User Interfaces

.Synchro-nization

GroupCommuni-

cations

Programming

ms Communications

ice Networks

ession

Video Audio

10C-sync.fm 2 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Scope

Usa

ge Applications

Learning & Teaching Design

Ser

vice

s ContentProcess-

ing

Docu-ments

Security ..S

yste

ms Databases

Media-Server Operating Syste

Opt. Memories Quality of Serv

Bas

ics Computer

Archi-tectures

Compr

Image &Graphics

Animation

Page 3: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

zation?

zation

nt

10C-sync.fm 3 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Contents

1. Motivation - What is (Multimedia) Synchroni

2. Synchronization and Multimedia

3. Reference Model for Multimedia - Synchroni

4. Synchronization in a Distributed Environme

5. Synchronization Techniques

6. Case Studies

Page 4: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

Synchronization?

ion 4

ntent

10C-sync.fm 4 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

1. Motivation - What is (Multimedia)

We concentrate on:• Temporal relationship

• New flavor in the context of multimedia• Essential for communications

Applicat

Application 2

Application 3

Application 1

Space

Time

Co

Page 5: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

heetame data

d update

e related views.

the use of

10C-sync.fm 5 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Content Relations

Dependence of media objects on data

Examples:• A graphic that visualizes data from a spreads• Two graphics showing different views of the s

Explicit definition of dependences for automate• Only the data are edited• All views of the data are

• generated automatically• cannot be edited directly

• An update of the data triggers an update of th

Implementation of content relations is based on• common data structures or• common object interfaces

Page 6: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

tation of a media object

entation

gned to this frament may be

cts:d in a windowon a stereo output device

vatars are positioned inith talking avatar!)

10C-sync.fm 6 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Spatial Relations

Usually known as layout relationships• Define the space which is used for the presen

• on an output device• at a certain point of time in a multimedia pres

Typically expressed in layout frames:• A layout frame is placed and a content is assi• The positioning of a layout frame in a docume

• fixed to a position in a document• fixed to a position on a page• relative to the positioning of other frames

Spatial relations for time-dependent media obje• A frame or a group of frames may be presente• Note: an audio presentation can be positioned

(imagine, e.g., a virtual meeting room where aa VRML space: audio should be positioned w

Page 7: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

existed before)

tandardization

st correspond to the temporal

10C-sync.fm 7 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Temporal Relations

Importance for time-dependent media objects• new in context of Multimedia (content, spatial• essential for communications, scheduling• non-trivial yet "general" issue --> subject of s• --> to be emphasized in this chapter

Example for temporal relations:• Temporal relation between

• video object• audio object

recorded during a concertAt presentation time:

• temporal relation of the two media objects murelation at the recording moment

Relations:• at the same time• independently• in sequel

Page 8: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

e relation betweenndent media object

ideo sequence.d each of the frames has to

ws a jumping ball

t

10C-sync.fm 8 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Intra-object Synchronization

Intra-object Synchronization

Intraobject synchronization defines the timvarious presentation units of one time-depe

Example:• Time relation between the single frames of a v

For a video with a rate of 25 frames per seconbe displayed for 40 ms

Frames of a video sequence that sho

40ms

Page 9: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

nchronization between

n

dio sequence

o2

tion

t

10C-sync.fm 9 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Inter-object Synchronization

Inter-object Synchronization

Inter-object synchronization defines the symedia objects

Example:• Time relations of a multimedia synchronizatio

that starts with an audio/video sequence,followed by several picturesand an animation that is commented by an au

Audio1

Video

P1 P2 P3

Audi

Anima

Page 10: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ence of information

bjects

phony

10C-sync.fm 10 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Logical Data Units

Logical Data Unit

An information unit - usually part of a sequunits - of a time-dependent media object

Granularity levels of LDUs• Application dependent• Imply a hierarchical decomposition of media o• Example:

The LDU hierarchy of a Sym

Symphony

Movements

Notes

Samples

Page 11: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

f continuous media like

ation execution

or a microphone, or

10C-sync.fm 11 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs

Closed LDUs• have a predictable duration

Example:• LDUs that are parts of stored media objects o

• audio and video or• stored media objects with a fixed duration.

Open LDUs• duration is not predictable before the present

Example:• Open LDUs typically represent

• input from a life source, for example a camera• media objects that include a user interaction

Page 12: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ce as LDUs

s except lowest)

equence as LDUs

Us, open block-LDU

t

Pic.n

t

Audio n

cy (e.g., 1/8000 s)

(e.g., = 0.064 s)

t

10C-sync.fm 12 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs (cont.)

Physical units as LDUs:

Physical units of a video sequen

Blocks of samples as LDUs: (all hierarchy level

Blocked physical units of an audio s

note: e.g., life-stream video has closed phys-LD

Pic. 1 Pic. 2 Pic. 3 Pic. 4

1/30 s

Audio 1 Audio 2 Audio 3 Audio 4

physical frame duration = 1/sample frequen

duration of a Logical Data Unit of 512 Bytes

Page 13: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

n

generated media objects.

ser selected size

rded user interactions.

em with varying durations

Frame n

e (= e.g.1/15 s)t

tion 4

t

10C-sync.fm 13 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs (cont.)

User-defined vs. capturing-defined LDU duratio

Fixed vs. varying LDU duration

• e.g., fixed&user-defined: typical for computer

LDUs of an animation sequence of u

• e.g., fixed&capturing-defined: typical for reco

LDUs of an user interaction in a window syst

Frame1 Frame 2 Frame 3 Frame 4

duration of Logical Data Units of selected siz

Action 1 Action 2 AcA.3

LDUs of a recorded user interaction

Page 14: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

rationknown duration

er interaction

DUs

l processingt

t

10C-sync.fm 14 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs (cont.)

Note: open LDUs always with unpredictable du• Typically a user interaction with in advance un

An open LDU representing an us

Note: Timers represented as stream of empty L

LDUs of a timer

LDU duration not known before the actua

User selection

LDUs of 10 ms duration

Page 15: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ed by

User

nimation,Timer

Userteraction

10C-sync.fm 15 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs - Overview

• open vs. closed• open: always unknown duration• closed: fixed or variable duration

• fixed vs. variable duration• user-defined vs. capturing-related• physical --> blocked

Duration defin

Capturing

Fixed Duration of LDU Audio, Video A

Variable or unknownDuration of LDU

Recordedinteraction in

Page 16: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

eo streams withs

zation

ow

Frame n

Audio n

ms advance

e 3 Slide 4

10C-sync.fm 16 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Examples

Lip synchronization• demands for a tight coupling of audio and vid• a limited skew between the two media stream

LDU view of lip synchroni

Slide show with audio comment:

LDU view of a slide sh

Frame 1 Frame 2 Frame 3 Frame 4

Audio 1 Audio 2

Audio max. 80 ms advance Video max. 80

Audio sequence

Slide 1 Slide 2 Slid

Page 17: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

udio1 and Video) is follo-ion (RI), a slidetion) which is partiallyo2). Starting the anima-n is presented to thection, a final picture

ple

Interaction

Audio2

Animation

P4

t

10C-sync.fm 17 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Application ExampleExample used further in the course:

A lip synchronized audio video sequence (Awed by a replay of a recorded user interactsequence (P1 - P3) and an animation (Animacommented using an audio sequence (Audition presentation, a multiple choice questiouser (Interaction). If the user has made a sele(P4) is shown.

Synchronization exam

Audio1

Video

RI

P1 P2 P3

Page 18: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ser Interface

Sync. QoSProvision

10C-sync.fm 18 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

2. Synchronization and Multimedia

Typical Requirements:• hard vs. soft

Strong Interaction with QoS• acceptable, too bad ...

Variety of Mechanisms• Live Synchronisation• Live Synchronisation with Storage• Synthetical Synchronisation

MMOperatingSystemEnvironment

ProgrammingEnvironment

Application

MM U

MM

MM Sync. QoSRequirements

Page 19: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ed by Skew (QoS)?????????

r Interface

Sync.QosProvision

10C-sync.fm 19 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Why Multimedia-Synchronization?

Media Influenc

Video

Animation

?

AudioImageText

Audio

AnimationAudioImageText

Pointer

MMOperatingSystemEnvironment

ProgrammingEnvironment

Application

MM Use

MM

MMSync.

QoSRequirements

Page 20: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

iments

ation requirements for:

equipment

10C-sync.fm 20 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Multimedia Synchronization - Exper

Example:• at IBM ENC Heidelberg to quantify synchroniz

• Audio/video synchronization• Audio/pointer synchronization

Selection of material:• Duration

• 30s in experiments• 5s would have been sufficient

• Reuse of same material for all tests

Introduction of artificial skew:• By media composition with professional video• With frame based granularity

Test conditions:• Huge set of test candidates

• Professional: cutter at TV studios• Casual: every day “user”

• Awareness of the synchronization issues• Set of tests with different skews lasted 45 min

Page 21: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

any artifact or

ords. (-)

f or behind the

or.o watch all your(.)

10C-sync.fm 21 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Feedback - Questionnaire

1 While watching this video clip, did you detectstrange effect?

If so, please try to describe it in a few w

....

2 Are you able to identify if audio was ahead omoving pictures? (.)

a) Yes, I identify audio to be played aheadof video

b) Yes, I identify audio to be played behindvideo

c)No, I notice that audio is out of sync with

respect to video but, I am not sure ifaudio is played ahead of or behind video.

3 You noticed a synchronization errHow would you qualify this error if you have t

TV programs with such an error?

a) I would not mind, the error is acceptable

b) I dislike it, the error is annoying

c) I am not sure if I would accept such anerror or if I would really dislike it

Page 22: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

encing Factors

(hammer and nails)

iew)

10C-sync.fm 22 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Lip Synchronization: Major Influ

Video:• Content

• Continuous (talking head) vs. discrete events• Background (no distraction)

• Resolution and quality• View mode (head view, shoulder view, body v

Audio:• Content• Background noise or music• Language and articulation

Page 23: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

tection

ahead of video

10C-sync.fm 23 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Lip Synchronization: Level of De

Areas:• In sync QoS: +/- 80 ms• Transient• Out of sync

audio behind video audio

Page 24: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

noyance

o ahead of video

10C-sync.fm 24 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Lip Synch.: Level of Accuracy/An

Some observations:• Asymmetry• Additional tests with long movie

• +/- 80 ms: no distraction• -240 ms, +160 ms: disturbing

audio behind video audi

Page 25: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

etch”)n map”)

10C-sync.fm 25 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Pointer Synchronization

Fundamental CSCW shared workspace issue

Analysis of CSCW scenarios:• Discrete pointer movement (e.g. “technical sk• Continuous pointer movements (e.g. “route o

Most challenging probes:• Short audio• Fast pointer movement

Page 26: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

etection

ahead of pointer

10C-sync.fm 26 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Pointer Synchronization: Level of D

Observations:• Difficult to detect “out of sync”:

• i.e., other magnitude than lip sync• Asymmetry:

• According to every day experience

audio behind pointer audio

Page 27: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

nnoyance

ahead of pointer

10C-sync.fm 27 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Pointer Synchronization: Level of A

Areas:• In sync: QoS -500 ms, +750 ms• Transient• Out of sync

audio behind pointer audio

Page 28: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

n QoS+/- 120 ms

+/- 80 ms

+/- 240 ms

+/-500 ms

+/- 240 ms

+/-500 ms

+/- 80 ms

+/- 11 µs

with various +/- 120 ms

d music) +/- 500 ms

otes) +/- 5 ms

) +/- 500 ms

+/- 240 ms

-500ms+750 ms

10C-sync.fm 28 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization QoS: Skew Values

Media Mode, Applicatio

Video

Animation correlated

Audio lip synchronization

Imageoverlay

non-overlay

Textoverlay

non-overlay

Audio

Animation event correlation (e.g., dancing)

Audio

tightly coupled (stereo)

loosely coupled (dialogue modeparticipants)

loosely coupled (e.g., backgroun

Imagetightly coupled (e.g., music with n

loosely coupled (e.g., slide show

Text Anmerkungen zu Text

Pointer Audio Related to the Itlem

Page 29: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

Synchronization

isms

ynchronization systems

10C-sync.fm 29 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

3. Reference Model for Multimedia -

A reference model is needed

• to understand the various requirements formultimedia synchronization

• to identify and structure runtime mechanisms

• to identify interfaces between runtime mechan

• to compare system solutions for multimedia s

Page 30: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

t

Timecompo-sition

Deriva-

Inter-pretation

tion

10C-sync.fm 30 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Existing Synchronization Models

Classification methods:• Physical, system, and human level• Intra-stream (fine-grain) synchronization and

inter-stream (coarse-grain) synchronization• Live and synthetical synchronization• Mapping

Uninterpreted

Media Objects

Bytestream

DerivedMedia Objects

MultimediaObject

Page 31: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

chniques

on

10C-sync.fm 31 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of Synchronization Te

Inter-object synchronization techniques:• used to control jitter between media streams• classification according to the type and locati

of the synchronization control

Distributed synchronization control:• distributed control based on protocols• distribution based on servers• distribution on nodes without server structure

Local synchronization control:• control on several layers• use of local servers

Page 32: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

Ab

stractio

n

low

highlayer

10C-sync.fm 32 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Four Layer Model

Media layer

Stream layer

Object layer

Multimedia application

Specification

Page 33: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

cluded objects

ct

ject synchronizations

ject synchronizations

ynchronization

10C-sync.fm 33 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Specification

Synchronization specification• describes all temporal dependencies of the in

in a multimedia object• is part of the description of a multimedia obje• should comprise:

• Intra-object synchronization specificationsfor the media objects of the presentation

• Quality of service descriptions for the intra-ob• Inter-object synchronization specifications

between the media objects of the presentation• Quality of service descriptions for the inter-ob

• may comprise:• alternative presentation forms

• must be explicitly described in a synthetical s• is implicitly defined in a live synchronization

Page 34: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

Tasksto edit

to format

map the quality of service values tothe object layer

Plan and coordinate presentationexecution

tiate presentation of time-dependentmedia objects

itiate presentation of time-indepen-dent media objects

source Reservation and schedulingof LDU processing

File and device access

10C-sync.fm 34 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Layered Model - Summary

Layer Interface AbstractionSpecification The tools performing the tasks of this

layer have interfaces, the layer itself hasno upper interface

to

Object Objects that hide types of enclosedmedia

Synchronization Specification

Media-oriented quality of service (Interms like allowed skew, allowed jitter)

Ini

In

Stream Streams and groups of streams

Guarantees for intra-streamsynchronization

Guarantees for inter-stream synchroni-zation of streams in a group

Re

Media Device independent access to LDUs.

Guarantees for single LDU processing

Page 35: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

Environment

ation

10C-sync.fm 35 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

4. Synchronization in a Distributed

More complex than in a local environment:

• Distributed storage of synchronization inform

• Distributed storage of media objects

• Communication delays and jitter

• Multi-party communication patterns

Page 36: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

fication

ation

Sink

ation

Sink

10C-sync.fm 36 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Transport of Synchronization Speci

Delivery of the complete synchronization informbefore the start of the presentation

Use of an additional synchronization channel:

Use of a multiplexed channel:

Source

Video

Audio

Synchronization

Source

Video

Audio

Synchroniz

multiplexed

Page 37: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ions

ink

Sinkidth

and

10C-sync.fm 37 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Location of Synchronization Operat

Source SBandwidthdemand

Synchronization at the sink node

Synchronization at the source node

Source Bandwdem

Page 38: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ing of the presentation:

Ta=Tav - Nla - SaTv=Tav - Nlb - Sv.skew

Sink AVTav

10C-sync.fm 38 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Clock Synchronization

Problem: Skew between the clocks

• Clock skew has to be considered in the plann• Tav: Start of presentation• Nla: Known net delay• Sa,Sv as Skews• Start time of audio transmission of Source A:• Start time of video transmission of Source V:

• Restricted buffer size demands for restricted

Source A

Source V

Audio

Video

Ta, Sa

Tv, Sv

Nla

Nlv

Page 39: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

Sinks

10C-sync.fm 39 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Multiple Communication Relations

1:1

n:1

1:n

n:m

Sources

Sources : Sinks

Page 40: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

o the network

10C-sync.fm 40 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Multi Step Synchronization

• Synchronization during object acquirement

• Synchronization of retrieval

• Synchronization during delivery of the LDUs t

• Synchronization during the transport

• Synchronization at the sink

• Synchronization within the output device

Page 41: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

en two objects

A B

AB

AB

A meets B

A during B

A finishes B

10C-sync.fm 41 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

5. Synchronization Techniques

Specification of great importance

• Simple specification method:

Types of temporal relations betwe

B

AB

A

BA

AB

A before B

A overlaps B

A starts B

A equals B

Page 42: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ts

nance of specificationsnit in the specificationbe easily describable

ia objectsribableation

10C-sync.fm 42 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Specification Methods - Requiremen

Complex specification comprise:• multiple object synchronization• user interactions

Requirements for a specification method:• Support of object consistency and for mainte• Media objects should be kept as one logical u• All types of synchronization relations should• Support for the integration

of time-dependent and time-independent med• Quality of service requirements must be desc• Support of hierarchical levels of a synchroniz

Page 43: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

either a single-media object

ronization operators

Video

onous audio/videoesentation

10C-sync.fm 43 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Basic Hierarchical Specification

Two main synchronization operations:• Serial synchronization of actions• Parallel synchronization of actions

Action:• an atomic action handles the presentation of

or a user input or a delay.• a compound action is a combination of synch

and atomic actions.

Serial and parallel presentations

Pic. 1 Pic. 2 Pic. 3

Slide sequence

Audio

Lip synchrpr

Page 44: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

synchronization.

(RI = Recorded Interaction, Pic.teraction)

Pic.3

Ani.3 UI Pic.4

10C-sync.fm 44 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Basic Hierarchical Specification

Delay as a possible action allows for• delays in serial presentations and• delayed presentations of objects in a parallel

Application example:

Hierarchical specification of the application example= Picture, Aud. = Audio, Ani. = Animation, UI = User In

Audio1 Video

RI Pic.1 Pic.2

Ani.1 Ani.2 Aud.2

Page 45: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

not be described

of skewessary

ependent mediamust be added

ia objects foroses

r media object

A3

B3

C3

10C-sync.fm 45 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Basic Hierarchical Specification

Not all relations can be described.

Example of a synchronization that can

Advantages Disadvantages

Easy tounderstand

Additional descriptionquality of service nec

Natural supportof

hierarchies

For the presentation of time-indobjects presentation durations

Integration ofinteractive

objects is easy

Need for the splitting of medsynchronization purp

No adequate abstractions focontents

A1 A2

B1 B2

C1 C2

Page 46: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

rios can not be

10C-sync.fm 46 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Some synchronization scenadescribed

Advantages Disadvantages

Page 47: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

Timer

xis

the application example

teraction

io2

ation P4

?t

10C-sync.fm 47 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on a Global

All single-media objects are attached to a time a

Application example

Time axis synchronization specification of

Audio1

Video

Record. UI In

P1 P2 P3

Aud

Anim

Page 48: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

Timer

vantages

nknown durationgrated, extensionsel are required

f service has to beirectly by using the

axis or additionalvice specificationsto be given

d environment thebal unique time hasbe kept

10C-sync.fm 48 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on a Global

Advantages Disad

Easy to understandObjects of u

cannot be inteto the mod

Support of hierarchies easy torealize

Skew quality ospecified indcommon timequality of ser

have

Easy to maintain because of themutual independence of objects

In a distributenotion of a glo

to

Good abstraction for mediacontents

Integration of time-independentobjects is easy

Page 49: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

xes

ement unitsate space

xes

at

10C-sync.fm 49 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on Virtual A

Specification using• coordinate systems with user defined measur• several virtual axes to create a virtual coordin

Example:

Music notes as example for virtual axes

At runtime the virtual axes are mapped to real a

Virtual axis with measurement unit be

Virtual axes with measurement unit pitch

Page 50: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

xes

ple

teraction

io2

ation

P4

t

t

10C-sync.fm 50 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on Virtual A

Application example:.

Virtual time axis specification of the application exam

Audio1

Video

Record. UI

In

P1 P2 P3

Aud

Anim

Time axis

Time axis

Interaction axis

Page 51: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

xes

vantages

ty of service onlyor by additionaltions definable

on may becomeby many axes

xes at runtime maynd time consuming

10C-sync.fm 51 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on Virtual A

Advantages Disad

Easy to understandSkew quali

indirectlyspecifica

Often specification according tothe problem space possible

Specificaticomplex

Good possibility for buildinghierarchies

Mapping of abe complex a

Easy maintainable, becauseobjects are kept as units andmutual independent objects

Good abstraction for mediacontent

Easy integration of timeindependent media objects

Interactive objects can beincluded using specialized axes

Page 52: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

e Points

rded as sequences of

ation of a media objectent media objects

it is reachedlicit reference to time.

10C-sync.fm 52 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization based on Referenc

Synchronization via reference points:• Time-dependent single-media objects are rega

LDUs• Reference points:

• the start time and the stop time of the present• the start-times of the subunits of time-depend

• Synchronization point:• A set of connected reference points• Defines the synchronization between objects• LDUs have to be started resp. stopped when

• Description of temporal relations without exp

Page 53: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

e Points

ation model

eference point model

3 Slide 4

10C-sync.fm 53 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization based on Referenc

Lip synchronization example

Lip synchronization in the reference point synchroniz

Slide show example.

Example of a slide show with audio sequence in the r

Video

Audio

Slide 1 Slide 2 Slide

Audio

Page 54: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

e Points

e-independent media objectssynchronization specification

Interaction

dio

mation

P4

10C-sync.fm 54 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization based on Referenc

Application example

Example of the integration of time-dependent and timas well as closed and open LDUs in a reference point

Audio

Video

Record. UI

P1 P2 P3 Au

Ani

Page 55: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

e Points

vantages

to change becausect description ofndencies

omplex hierarchies

10C-sync.fm 55 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization based on Referenc.

Advantages Disad

Easy to understandMore difficult

of the diredepe

Natural description oftemporal relations Sometimes c

Easy integration of open LDUs

Integrated description of skewquality of service

Easy integration of time-independent objects

Time axis are special case of thereference point synchronization

model

Page 56: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ns

hat reaches a specific LDU

10C-sync.fm 56 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Event-based Synchronization

Event-based synchronization presentation actioare initiated by the synchronization events

Typical presentation actions are:• to start a presentation• to stop a presentation• to prepare a presentation

Events• may be external,

e.g. generated by a timer, or• internal to the presentation

generated by a time-dependent media object t

Page 57: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

r1.dy ...

p

rt

10C-sync.fm 57 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Event-based Synchronization

Application example:

ActionEvent

Start Audio.stop Timerea

Audio1 start

Video start

Pic.1 start sto

Timer1 start(3)

Pic.2 sta

...

Page 58: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

vantages

sy to handle

specification

to maintain

f time-independentng additional timers

riptions of quality ofnecessary

e of hierarchies

10C-sync.fm 58 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Event-based Synchronization

Advantages Disad

Easy integration of interactiveobjects Not ea

Easy extensible by new events Complex

Flexible because any eventcan be specified Hard

Integration oobjects by usi

Separate descservice

Difficult us

Page 59: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

on blocking token.n is removed and to each

that is assigned to its new

ion

F4

A1

ms 11 ms

33ms

F5

10C-sync.fm 59 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Petri net-based synchronization

The rules for a timed petri net are:• A transition fires if all input places contain a n• If a transition fires, on each input place a toke

output place a token is added• After firing, a token is blocked for the duration

place

Lip synchronization example.

Petri net based synchronization of a lip synchronizat

F2 F3

A1

11 ms 11 ms 11 ms

33ms

11

F1

Page 60: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

F3

11 ms D

0 ms

10C-sync.fm 60 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Petri-Net based synchronization

Example for a hierarchy:

F1 F2

A1

11 ms

33ms

11 msD

0 ms

S

33 ms

Page 61: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

A3nimationubnet 3

10C-sync.fm 61 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Petri-Net based synchronization

Application example:

Petri net specification of the application example

S

60 s

Audio-Videosubnet

R

20 s

Record.UI

P

15 s

Picturesubnet

PI

A1

Au

A2Animation-subnet 1

As

Animation subnet 2and Audio subnet

P4

User interaction Closing picture

Page 62: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

vantages

sy to handle

specification

f media objects

bstraction for thehe media objects

10C-sync.fm 62 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Petri-Net based synchronization.

Advantages Disad

Hierarchies can be created Not ea

Easy integration of time-independent objects Complex

Easy integration of interactiveobjects Splitting o

Integrated skew quality of service Insufficient acontent of t

Page 63: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ariots

y the timing operations

10C-sync.fm 63 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Scripts

Script• Textual description of a synchronization scen• Elements of scripts are activities and subscrip• Three main operations in a script are

• serial presentation• parallel presentation• repeated presentation of a media object

• Often full programming languages extended b

Page 64: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

;

.rec”);eg”);eg”);eg”);eg”);;on.au”);on.ani”);

;

ple {(

}

10C-sync.fm 64 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Scriptsactivity DigAudioAudio(“video.au”)activity SMPVideo(“video.smp”);activity XRecorderRecorder(“windowactivity PicturePicture1(“bild1.jpactivity PicturePicture2(“bild2.jpactivity PicturePicture3(“bild3.jpactivity PicturePicture4(“bild4.jpactivity StartIntercationSelectionactivity DigAudioAniAudio(“animatiactivity RTAnimaAnimation(“animatiscript Picture_sequence 3Pictures=

Picture1.Duration(5) >>Picture2.Duration(5) >>Picture3.Duration(5);

script Lipsynch AV = Audio & Videoscript AniComment AA = Animation &AniAudio.Translate(2);script Multimedia Application_exam

AV >> Record. UI >> 3Pictures >>( (Selection >> Picture4) & AA )

Page 65: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

vantages

sy to handle

specification

age of commonnecessary

structs for skewervice necessary

10C-sync.fm 65 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Scripts

Advantages Disad

Easy hierarchies Not ea

Logical structure of objectscan be kept Complex

Easy integration of time-independent objects

Implicit ustimers

Easy integration of interactiveobjects

Special conquality of s

Skew quality of service canbe specified

Easy extensible by newsynchronization constructs

Flexible because programmable

Page 66: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

reated further here:

vironment)

AND HYPERMEDIA OBJECTS

tion objects to bed services

at are part of the

p a set of MHEG objectsctions to be performed

oral and conditional

10C-sync.fm 66 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

6. Case Studies

1. Some "historical" or academic systems not t• HyTime• FireFly• MODE (Multimedia Objects in a Distributed En• Multimedia Tele-Orchestra• ACME (Abstractions for Continuous MEdia)• Apple QuickTime

2. MHEG: CODED REPRESENTATION OF MULTIMEDIA

• representation and encoding of• final form multimedia and hypermedia informa

interchanged within or across applications an

MHEG objects• Content objects encapsulate media objects th

presentation• Composite objects act as a ‘container’ to grou• Action objects are used to exchange sets of a

between objects• Link objects are used to specify spatial, temp

relations between MHEG objects

Page 67: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

tent objectsneric units of time=altitude) of finite length

inate space

are sent to an object

ables the object

10C-sync.fm 67 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization in MHEG

Generic space:• provides a virtual coordinate system• used to specify the layout and relation of con

• one time axis of infinite length measured in ge• three spatial axes (X=latitude, Y=longtitude, Z

in an interval of [-32768,+32767]• MHEG maps from the virtual to the real coord

Presentation of content objects:• Based on the exchange of action objects that• Examples for actions are:

• prepare for setting the object in a state that ento present itself

• run to start the presentation• stop to stop the presentation

Page 68: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

elayed sequential actions.

...

10C-sync.fm 68 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

MHEG action lists

Action list:• Combination of actions to action lists• Parallel action lists are executed in parallel• Each list is composed of a delay followed by d

Links:• used to create event-based synchronizations• can trigger events

delay delay

action action

Parallel lists of actions

Del

ayed

sequ

entia

l

actio

ns

Page 69: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

Gen

eric

Pre

sent

atio

nS

ervi

ces AV-Sub-

UserInterface

system

Services

10C-sync.fm 69 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

MHEG Engine

Application

MHEG Engine

Object Manager

Link Processor Interpreter

Operating System

Page 70: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

ultimedia object

Engine

User Interface

bsystem

Service

10C-sync.fm 70 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

MHEG Relation to Reference Model

Media layer

Stream layer

Object layer

Specification layer

Object layer interface

Stream layer interface

Media layer interface

MHEG coded m

MHEGInterpreter

AV Su

Object ManagerLink Processor

Page 71: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

guage

)

ia Synchronization

ay (cf. Flash, Shockwave)

ontents not accessible

b?

10C-sync.fm 71 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

SMIL

SMIL: Synchronized Multimedia Integration Lan

part of W3C standards, part of XML&friends

treated in lectures about Web Engineering

based on "Amsterdam Model" (CWI Amsterdam

major contribution of recent years

goal: integrate Hypertext concept and Multimed

remember: multimedia possible on the Web tod

but:• requires plug-in• MM presentation is "one node" in hypertext, c• no links into MM presentation, no links out• same text used within flash and as part of We

--> duplicate --> two different elements!

SMIL introduces MM synchronization in XML• similar to hierarchical specification• plus support for spatial arrangements

Page 72: Synchronizationcs778/ralf/10c-sync.pdf · ©R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing.Ralf Steinmetz ... KOM - Industrial Process and System

MM node B

TML

t

MM node B

TML

t

ave been, e.g.:

" duration="0.5" target=2>e" in <img> tag)

left=... top.. height.. width../>

10C-sync.fm 72 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

SMIL

Example for temporal spec:<par>

<img src="logo.rp" region="logo"/><text src="info.rt" region="info"/><video src="pk.rm" region="video"/>

</par>

<seq><audio src="music1.ra"/><audio src="music2.ra"/>

</seq>

link

in H

MM node A

single mediumwithin A

t

link

in H

MM node A

single mediumwithin A

t

prior definitions, might h

<fadein start="0.2(target: -> "handl

<region id="logo"