Chapter 2: Basics€¦ · Chapter 3.4: Synchronization Page 6 NTP Hierarchy 1 2 3 2 33 Primary servers are connected to UCT sources Secondary servers are synchronised to primary servers

Lehrstuhl für Informatik 4

Kommunikation und verteilte Systeme

Page 1Chapter 3.4: Synchronization

Chapter 2: Basics

Chapter 3: Multimedia Systems – Communication Aspects and Services• Multimedia Applications

and Communication• “Multimedia” Transfer

and Control Protocols

• Quality of Service and Resource Management

• Synchronization

• Multimedia Operating Systems

Chapter 4: Multimedia Systems – Storage Aspects

Chapter 5: Multimedia Usage

3.4: Synchronization

• Computer Clocks• Network Time Protocol

• One-way Delay Measurements

• Synchronization of Media Streams




Why Synchronization?

Necessity:

• Synchronization of different media streams, e.g. in videoconferencing: audio and video are transmitted as two independent streams: synchronization has to take place when streams are played

• Furthermore: distributed operations must be coordinated for in-time data delivery and buffer management

• Most media synchronization schemes demand for knowledge about temporal relations between source and sink(s) → accurate global time scale required

Example: Audio/Video presentation with multiple sources

• Presentation at sink must start at TC

• Sources A and B must start at TC - OAC - DAC and TC - OBC - DBC

• Problem: OAC, OBC, DAC, DBC unknown

Source A:Audio

Source B:Video

Clock offsetA ↔ C: OAC

Clock offsetB ↔ C: OBC

Sink C

Networkdelay DAC

Networkdelay DBC




Computer Clocks

Problem of synchronization

• Universal Coordinated Time (UCT): International time and frequency standard, based on synchronized Caesium clocks (accuracy about 10-12 sec)

• Investigation of 20.000 clocks of Internet hosts: 50% show offset > 2 minutes (with respect to UCT), 10 % show offset > 4 hours, some several weeks!

Reasons

• Clock adjustment usually via wrist watch of operator• Differences in the oscillator frequencies (clock drift) → clocks tend to wander from the exact time

• Drift in parts per million (ppm) sometimes as high as a few hundred ppm (or some seconds per day)

• Further problems: Clock resolution (as bad as 10 ms), non real-time operating systems (e.g., UNIX)

Oscillator(e.g., quarz)

Prescaler(e.g. 100 Hz)

Clock counter

SystemProcessor

Ticks / tick adjust




Computer Clocks / Hardware Oscillators

• Atomic clocks are far too expensive for common use• Alternative: transmitting time information from hosts with atomic clocks

→ Software-based clock synchronization methods needed

Typical stability and drift values:

Oscillator type Stability (per day) DriftCaesium beam 3⋅10-7 ppm 3 ⋅10-6 ppm (per year)Rubidium cell 5⋅10-6 ppm 3⋅10-5 ppm (per year)Quarz (digital) 0.05 ppm 0.001 ppm (per day)Quarz (analog) 0.5 ppm 0.003 ppm (per day)Quarz (standard) up to 1 ppm up to several 100 ppm (per day)

i.e. up to ≈ 1 ms / day




Clock Synchronization in the Internet

Network Time Protocol (NTP) (RFC 1119):

••

Atomic clock

GPS

Primary servers

Backup path

Client

Stratum-4

SynchronizedLAN clusterSecondary

servers

Stratum-1

Stratum-2 Stratum-3

• Exchange of timestamps between time servers and clients via UDP• Virtual hierarchy of synchronized Internet hosts




NTP Hierarchy

1

2

3

2

3 3

Primary servers are connected to UCT sources

Secondary servers are synchronised to primary servers (Synchronization subnet )

Note: this is only an example, there can be more than three layers

More accurate time

Lowest level servers in users’computers, synchronised to secondary servers

• The synchronisation subnet can reconfigure if failures occur, e.g.– a primary that loses its UCT source can become a secondary

– a secondary that loses its primary can use another primary

• Modes of synchronisation: Multicast, Procedure call, Symmetric – depending on the needed accuracy




NTP Algorithm (Symmetric)

Client Time server

Delay: 33 ms

Delay: 17 msti,1 = 17.00.00.017ti,2 = 17.00.00.018

ti,0 = 15.00.00.000

ti,3 = 15.00.00.051

tsrc tdest

( ) ( ) ( )i ,1 i ,0 i ,3 i ,2est ii i

t t t t 2.00.00.017 1.59.59.967: 1.59.59.992

2 2 2εθ θ

− − − − −= = = = +

6i i

17 332; 10

2θ ε −−= = ⋅

Client sends requests, time server answers with current time ti,2

Client computes estimated time offset as:

where εi = difference between packet delays on forward and return pathθi = true offset of the two clocks




NTP Algorithm

A B

ti,0

ti,3

i ,0 i ,3t t

2

+

ti,1

ti,2

i ,1 i ,2t t

2

+

message 1

message 2

time time

θiest = estimation of

offset between A and B




( ) ( )

( ) ( ) ( )

i ,1 i ,3 i ,0 i ,2esti

c c S Ri ,1 i ,3 i ,0 i ,2 i i

S Ri i

i

t t t t

2

t t t t

2

2

θ

ε ε

ε εθ

− − −=

− − − + −=

−= +

NTP Algorithm

Impact of different propagation delays:

• Let tci,1 = time of receiver when message 1 was sent

→ ti,1 = tci,1 + εSi (where εS

i = propagation delay from sender to receiver)• Let tci,3 = time of receiver when message 2 was sent

→ ti,1 = tci,3 + εRi (where εR

i = propagation delay from receiver to sender)

→




Applicability of NTP

Problems

• Estimated time depends on packet delays, which are mostly non-symmetric → Accuracy drops in case of asymmetric routing and high network load

• Typical precision in the order of a few tens of ms (even for time servers!)

• Errors propagate through the synchronization network• Slow clock adaptation by tick adjustment of client clock counter→ Oscillations with amplitudes as high as 10 ms

Still: Sufficient accuracy for most multimedia applications ...... but not for delay measurements:

• Mean one-way delays in national WANs (e.g. GWIN) usually well below 20ms → NTP’s “long-term” accuracy not sufficient here

→ New measurement approach needed




One-Way Delay Measurements

• Approach:

� Delay measurement to first synchronize oscillator frequencies� Correction of timestamps by measured drift

� Subsequent estimation of clock offset• Drift influence on delays: 1000 probe packets, 10 s interdeparture delay

0 1000 2000 3000 4000 5000

0

100

200

-100

-200

cisco-rz.rz.rwth-aachen.de

Time [seconds]

Mea

sure

d dr

ift [m

s]

0 1000 2000 3000 4000 5000

40

60

80

20

0

rcs2.urz.tu-dresden.de

Time [seconds]

Mea

sure

d de

lays

[ms]

Round-Trip

Forward path

Return path





• Drift estimation:

� Measured latencies show clear linear behavior, errors due to delay variances� Running minimum approach to eliminate delay fluctuations

� Linear regression on first derivate yields estimator for relative drift• Running minima: minimum calculated for a window size of 50 packets (500 s)

0 1000 2000 3000 4000 5000

0

100

200

-100

-200


Time [seconds]

Mea

sure

d dr

ifts

[ms]

0 1000 2000 3000 4000 5000

30

40

50

20

10


Time [seconds]

Mea

sure

d de

lays

[ms]

0

Round-Trip

Forward path

Return path





Results of drift estimation:• Aachen: Forward path: -33.93 ppm

Return path: +34.15 ppm• Dresden: Forward path: +2.001 ppm

Return path: -2.001 ppm

• Adaptation of sender or receiver timestamps possible• Correction of timestamps by mean fdrift of the two estimators, e.g. for sender:

where t0 indicates the start of the sampling period

• Goodness of estimation indicators:� Equal drift amount on both paths� First derivate of running minima is a constant function

• Several enhancements of this basic algorithm are possible

( ) ( )new newi ,o i ,0 drift i ,0 0 i ,3 i ,3 drift i ,3 0t t f t t and t t f t t= + ⋅ − = + ⋅ −

adaptation of sender times ti,0 and ti,3 by average drift




Estimation of clock offset:• Assumption: Equal minimum delays on both paths within sampling period

• More general than assuming equal delays for each measurement packet→ Shift of measured empirical distributions to common minimum:

• Clock offset: difference between sender and receiver clock for arbitrary packet j:

Offset at tj,3: θestj = tj,2 + estimated delay of packet j on return path

• Algorithm performs better than NTP• Drawback: Higher overhead due to more measurement packets

{ } { }new newi ,3 i ,2 i ,1 i ,0min t t min t t

New distribution offset2

− + −=






Results: Empirical one-way delay distributions for Aachen and Dresden

Example benefits:

• Computation of jitter distribution at the receiver (→ buffer requirements)• Detection of network bottlenecks (compare forward vs. return path to Dresden)

-5 0 5 10 15 20

0.6

0.8

1

0.4

0.2


Time [seconds]

Mea

sure

d de

lays

[ms]

00 10 20 30 40 50

0.6

0.8

1

0.4

0.2


Time [seconds]

Mea

sure

d de

lays

[ms]

060 70

Round-Trip

Forward path

Return path




Quality of Service for Media Objects

Synchronization in principle also is part of QoS, related to the playout of multimedia data:• QoS specification of single Logical Data Units (LDUs), e.g.

• QoS specification concerning two related media objects:� Acceptable skew between presentation of different objects

� Production level synchronization: QoS guaranteed prior to presentation (e.g. prefetching of video frames and playback with synchronized audio)

� Human perception of synchronization

Image Video AudioColor Depth Color Depth Sampling Method

Resolution Resolution Sampling SizeFrame rate Sample rate

Jitter JitterError rate Error rate




Quality of Service for Media Objects

Guidelines for human perception of synchronization:

Media combination Mode, Application QoS Video / Animation Correlated ± 120 msVideo / Audio Lips synchronization ± 80 ms

Video / Image Overlay ± 240 msNon-overlay ± 500 ms

Video / Text Overlay ± 240 msNon-overlay ± 500 ms

Audio / Animation Event correlation e.g., dancing ± 80 msTightly coupled (e.g., stereo) ± 10 ms

Audio / Audio Tightly coupled (e.g., stereo) ± 10 msLoosely coupled (e.g., background music) ± 500 ms

Audio / Image Tightly coupled (e.g., music with scores) ± 5 msLoosely coupled (e.g., slide show) ± 500 ms

Audio / Text Text annotation ± 240 ms




QoS for Multiple Related Media Objects

Example of skew specification:

• Video with audio parallel in English and Spanish• Requirements:

1. max skew(video ahead_of audio-english): 80 ms

2. max skew(audio-english ahead_of video): 80 ms3. max skew(video ahead_of audio-spanish): 80 ms4. max skew(audio-spanish ahead_of video): 80 ms

5. max skew(audio-english ahead_of audio-spanish): 400 ms6. max skew(audio-spanish ahead_of audio-english): 400 ms

In shortened from:

1+2: V-AE ∈ [-80:80]

3+4: V-AS ∈ [-80:80]5+6: AS-AE ∈ [-400:400]

The first requirements cannot be satisfied if AS-AE ∉ [-160:160]This results in the following modification of the requirements




1. max skew(video ahead_of audio-english): 80 ms

2. max skew(audio-english ahead_of video): 80 ms3. max skew(video ahead_of audio-spanish): 80 ms4. max skew(audio-spanish ahead_of video): 80 ms

5. max skew(audio-english ahead_of audio-spanish): 160 ms6. max skew(audio-spanish ahead_of audio-english): 160 ms

QoS for Multiple Related Media Objects

AE-V ∈ [-80:80]V-AS ∈ [-80:80]

→ AE - AS = (AE-V) + (V-AS) ∈ [-160:160]

Modified requirements:




Synchronization Specification

Describes all temporal dependencies of the included objects in a multimedia environment:

• Time relations between presentation units (e.g., spacing between succeeding video frames: 40 ms)

• QoS descriptions of media objects (e.g., maximum jitter of 125 ms for audio frames)

• Time relations between different media objects(e.g., lip synchronization in an audio/video sequence: ± 80 ms)

• QoS descriptions between different media objects(e.g., maximum jitter of audio in a slide presentation: 500 ms)

→ Synchronization specification is an essential part of the description of a multimedia object!




Synchronization Specification

Requirements• Treatment of each media object as a single logical unit• Abstraction of the media object contents• Easy description of all types of synchronization• Integration of time-dependent and time-independent media objects• Handling of user interaction• Definition of QoS requirements• Support of hierarchical levels of synchronization (for complex scenarios)

Methods• Interval-based specification• Axes-based synchronization• Control flow-based synchronization

� Hierarchical specification� Specification via reference points� Timed Petri Networks

• Event-based synchronization• Scripts




Synchronization Specification Example

α: A lip synchronized audio/video sequence is shownβ: A recorded user interactionγ: 3 picturesδ: An animation which is partially audio commentedε: An interaction, e.g. a multiple choice question is shown until the user has made a selectionχ: A final picture is presented after the user has made his choice

time

A

V

RI

P1 P2 P3

AN1 AN2 AN3

A2

Interaction

P4

α

δ

ε χγβ




Interval-based Specification

Presentation duration is represented as interval• Basic types of temporal relations between objects:

A before B A B A meets B A B

A overlaps B A

B

A during B A

B

A starts B A

B

A finishes B A

B

A equals B A

BEnhanced interval-based method:• 29 interval relations can be specified

• 10 operations defined to handle these temporal relations(before, cobegin, coend, overlaps, while, ...)





• Interval operations have one, two or three parameters• Operations with one Delay Parameter

A AA A

B B BB

δ1δ1 δ1δ1

before(δ1) beforeendof(δ1) cobegin(δ1) coend(δ1)

A AA A A

B

δ1 δ2

B

δ1δ2

B

δ1δ2

B

δ2δ1

B

δ2 δ1

while(δ1, δ2) delayed(δ1, δ2) startin(δ1, δ2) endin(δ1, δ2) cross(δ1, δ2)

• Operations with two Delay Parameters





• Operation with three Delay Parameters

A

B

δ1

δ2

δ3

overlaps(δ1, δ2, δ3)

=̂

=̂=̂

A before(3) B end of A is 3 time units before the start of B

A beforeendof(7) B A starts 7 time units before end of BB while(3,5) A A starts 3 time units after start of B and ends 5 time

units after start of B




Interval-based Specification - Example

Audio1 cobegin(0) Video

Audio1 coend(0) Video /* i.e., Lip synchronization*/

Audio1 before(0) RecordedInteraction

RecordedInteraction before(0) Picture1

Picture1 before(0) Picture2

Picture2 before(0) Picture3

Picture3 before(0) Interaction

Picture3 before(0) Animation

Animation while(2,5) Audio2

Interaction before(0) Picture4

A2

I

P4P3P2P1

RI

A1

V

Animation

AN1 AN2 AN3





Advantages:

• Easy to handle open (i.e., non-predictable) LDUs (closed LDU: predictable duration, open LDU: non-predictable duration, typical for live sources, e.g. objects which include an user interaction)→ easy to handle user interaction

• Possibility of specifying additional indeterministic temporal relations by defining intervals for durations and delays

• Disjunction of operators supported (e.g., “not parallel“)→ Flexible model that supports many run-time presentation variations

Disadvantages:• Model does not include skew specifications

• No direct specification of relations between subunits of media objects possible• Flexible specification leads to possible inconsistencies

Example: (A not in parallel with B) & (A while(2,3) User Interaction) &(User Interaction before(0) B)→ Conflict: B must be started at end of interaction, but A may still run!




Different possibilities:

1. Global timer: all objects attached to a global time axis2. Virtual time axis: coordinate systems with user defined measurement units;

possibly several time axes are used simultaneously, e.g. for musical notes

Axes-based Synchronization

Synchronization is based on global timing:• All objects are attached to a time axis representing an abstraction of real-time, which

is shared among all objects

• Removing one object does not affect other objects• Synchronization of objects only based on fixed points of time → very good for closed

LDUs, several problems with open LDUs

A2P4

RI

A1

VAnimation

P3P2P1

Interaction

?Time unknown!

unknown duration




Control Flow-based Synchronization

Basic hierarchical specification:

• Flow of concurrent presentation threads is synchronized in predefined points of the representation

• Serial and parallel synchronization of “atomic” and “compound” actions

• Atomic actions handle presentation of single-media objects, user input or delay• Compound actions are combinations of synchronization operators and atomic

actions

• Basic examples for serial and parallel presentations:

Pic. 1 Pic. 2 Pic. 3

Slide sequence

Audio Video

Lip synchronization

serial parallel





Specification example:

Advantages• Easy to understand• Natural support of hierarchies

• Integration of interactive objects

Disadvantages• Synchronization only possible at the beginning or at

the end, thus a split of “Animation“ into AN1, AN2, AN3 was necessary

• Skew QoS not definable• Presentation durations must be added

• Some synchronization scenarios cannot be described

An1 An2 An3Audio2

Rec.Interaction

Pic. 1 Pic. 2 Pic. 3

Audio1 Video User Interaction

Pic. 4

other representationof previous example





Synchronization via Reference Points:

• Time-dependent media objects are regarded as sequence of closed LDUs• Start and stop times of media objects: “Reference Points”

• Synchronization between objects by connecting reference points; a set of connected reference points is a “Synchronization Point”→ Approach specifies temporal relations between objects without explicit reference

to time• Example: Slide show with audio sequence (slides are control points)

Audio

Slide 1 Slide 2 Slide 3 Slide 4





Problems:

• Requires mechanism for detecting inconsistencies• Does not allow specification of delays

• Possible solution: Inclusion of a global timer as a reference axis

Audio 1

Rec. Interaction

Audio 2

User Interaction

Animation

P4

Video

Animation 2

Animation 1

Animation 3

P1 P2 P3

The previous example with “Synchronization by reference points”





Timed Petri Networks• Directed bipartite graph formalism for specifying state machines, dynamic extension

into the time domain

• Simple Petri Networks consist of places (circles) and transitions (bars), interconnected by directed arcs (arrows)

• Places can contain tokens (black filled circles)

• Transitions “fire“, if all input places contain tokens: Remove one token from eachinput place, add one token to each output place

• Places or transitions consume time, i.e. tokens are delayed

InputPlaceof T1

OutputPlaceof T1

Token

Transition T1

Arcs





Control Flow Specification with Timed Petri Networks:

• Places represent LDUs, synchronization is modeled by transitions• Example: Lip synchronization of audio and video streams

• Petri Networks allow all kind of synchronization specifications

• Problems:� Complex specifications

� Insufficient abstraction from media contents

V1 V2 V3 V4 V5

V1 V1

11 ms 11 ms 11 ms 11 ms 11 ms

33 ms33 ms





Our previous example with Petri Nets:

RI P1

P4

P2 P3A1/V

I

AN1

A2

AN2

AN3

5s

5s5s5s20s60s

Places can be refined to subnets of finer time granularity

Example:

α1 α2 α3 D

D

AN2

20 sec

0 sec

5 sec 5 sec 10 sec 0 sec

=̂




Event-based Synchronization

Presentations are initiated by synchronization events:• Examples: start / stop / prepare a presentation• Events that initiate representations may be external (e.g., timer-based) or internal

(e.g., media object reaching a specific LDU)• Advantage: Easy integration of interactive objects• Disadvantages:

� Difficult to handle� Complex specification

� Skew QoS cannot be defined� No hierarchies

� Maintenance very difficult

StartEvent

ObjectAudio1Stop

Timer1Ready

...

Audio1

start

start

startVideo

Pic.1 start

start(3)Timer1

Pic.2

action




Scripts

• Elements: Activities and subscripts• Often: Full programming language, extended by timing operations

• Different specification methods can be supported• Example:

� Script based on basic hierarchical method, which supports serial (“>>“-operator), parallel (“&“-operator) and repeated presentation (“n“ denotes n-fold repetition):

• 3Pictures = Picture1.Duration(5) >> Picture2.Duration(5) >> Picture3.Duration(5)• AA = Animation & Audio2.Delayed(2)

Definition of activities:activity DigAudio Audio(„video.au“);

activity SMP Video(„video.smp“);

activity Xrecorder Record(„window.rec“);

activity Picture Picture1(„picture1.jpeg“);

...

activity StartInteraction Selection;

activity RTAnima Animation(„animation.ani“);

Script:script Lipsynch AV = Audio&Video;

script Multimedia Example {

AV >>

Record >>

3Pictures >>

((UserInteraction>>Picture4)&AA)

}




Scripts

Advantages:

• Powerful and flexible programming environment• Support of hierarchies and logical objects

• Easy integration of time-independent and interactive objects

Disadvantages:

• Difficult to handle: Procedural approach instead of declarative• Complex specification

• Implicit usage of common timers necessary• Special constructs for skew QoS necessary




Conclusions

Synchronization specification methods:

• Different specification capabilities and differing user friendliness• Proposed methods are just different “views” of the same problem

• Selection of a method depends on target application and environment, there is no best or worst solution

Examples:

• Simple presentations without user interaction: Global timer seems best solution (but requires exact clock synchronization!)

• Complex structure with interaction: Reference point models more suitable

→ Different standards use different specification methods

Documents

Chapter 2: Basics€¦ · Chapter 3.4: Synchronization Page 6 NTP Hierarchy 1 2 3 2 33 Primary servers are connected to UCT sources Secondary servers are synchronised to primary servers