SELF-ORGANIZED INTER-DESTINATION MULTIMEDIA

SYNCHRONIZATION FOR ADAPTIVE MEDIA STREAMING

Benjamin Rainer, Christian Timmerer

Alpen-Adria-Universität Klagenfurt, Institute of Information Technology

TYPES OF SYNCHRONIZATION §  Intra-Stream Synchronization

§  Avoid jitter between the presentation of two consecutive media units

§  Inter-Stream Synchronization §  E.g., Audio + Video + Subtitles

Network Mul$media  Playback  

Time  (t)  

Time  (t)  

Video

Audio  

Receiver  

Network

40ms  

Time  (t)  

Receiver Video

Playback

Slide 2 of 19

MOTIVATION §  Why do we need Inter-Destination Multimedia

Synchronization? §  Friends are watching a soccer match and they communicate via text messaging/

phone/…

User  1   User  2  Goal!   Did  you  see  the  goal?  

Which  goal?  Thanks  for  the  spoiler!  

Slide 3 of 19

STATE OF THE ART §  Most IDMS solutions

§  Extend sender/receiver reports of RTCP [4] §  Assume multicast [4, 18] §  Demand a centralized instance [4, 24]

•  Master-/Slave- scheme •  Synchronization Master

§  Push-based multimedia streaming [4, 24, 9, 18]

§  Overcoming the asynchronism §  By pausing and skipping media content §  Increasing or decreasing the playback rate [17]

•  Adaptive Media Playout (AMP)

Slide 4 of 19

OUR APPROACH Content Provider

Application LayerPeer-to-Peer Overlay

Geographically Distributed

Clients

MPDMPD Provides MPDs

enriched with Session Information

MPD Server

Content Servers

§  Research Objectives §  Adapt MPEG-DASH for IDMS §  Agree on a reference in a distributed and self-organized manner §  Carry out the actual synchronization more smartly than using plain AMP

Slide 5 of 19

SESSION MANAGEMENT §  Inter-Destination Multimedia Synchronization Session Object

(ISO) §  Time bounded entity, contains a set of peers, uniquely identifiable §  (IP, port) and the type of the Network Address Translator (NAT)

•  NAT covered in the paper §  Allows a (unique) numbering of peers

§  ISO is identified by session key §  Provided by 3rd party application or the user

§  Integrated into the MPD of MPEG-DASH

§  Server imports the corresponding ISO when requested §  E.g., a peer requests the MPD with a session key

Slide 6 of 19

SYNCHRONIZATION §  Two phase synchronization using non reliable communication (UDP)

§  Coarse synchronization §  Fine synchronization

§  Coarse synchronization §  Overlay creation §  Educated guess where to start downloading

§  Fine synchronization §  Distributed algorithm – Merge and Forward §  Periodical sends constants sized message to neighbors §  Negotiate on reference playback timestamp

§  Overcome the identified asynchronism §  Dynamic Adaptive Media Playout

Slide 7 of 19

COARSE SYNCHRONIZATION §  Overlay creation

§  Ask peers in the ISO for their current Playback Timestamp (PTS) + NTP TS

•  If the response is received, the peer is added to the list of known peers

§  May lead to a not fully connected network •  E.g., due to packet loss

§  Strategies for selecting the segment to start with §  Maximum PTS from peers §  Minimum PTS from peers §  Weighted average PTS from peers

Slide 8 of 19

MERGE AND FORWARD - EXAMPLE 1  

2  

3  

Initial state: {BF, LID, HID, Cnt, ATS} 1: {{1}, 1, 1, 1, P1} 2: {{2}, 2, 2, 1, P2} 3: {{3}, 3, 3, 1, P3}

1  

2  

3  

1: {{1,2}, 1, 2, 2, (P1+P2)/2} 2.1: {{1,2}, 1, 2, 2, (P1+P2)/2} 3: {{2,3}, 2, 3, 2, (P2+P3)/2}

τ =1

BF                  …  Bloom  filter      Cnt              …  Cumula$ve  Count  H/LID      …  Lowest/Highest  Peer  ID    ATS            …  (Weighted)  Average  TS  

Slide 9 of 19

1  

2  

3  

1: {{1,2}, 1, 2, 2, (P1+P2)/2} 2: {{1,2,3}, 1, 3, 3, (P3+2*(P1+P2/)/2)/3} 3: {{2,3}, 2, 3, 2, (P2+P3)/2}

τ =1 1  

2  

3  

1: {{1,2,3}, 1, 3, 3, (P1+P2+P3)/3} 2: {{1,2,3}, 1, 3, 3, (P1+P2+P3)/3} 3: {{1,2,3}, 1, 3, 3, (P1+P2+P3)/3}

τ = 2

M&F - EVALUATION §  Simulation environment OMNeT++ with INET

framework §  Random networks (Erdős-Rényi) §  40, 60, and 80 peers §  Probabilities for creating connections between peers: 0.1

to 0.9 (uniformly distributed) §  Period of 250ms §  RTT of 300ms between peers

§  Compared to Aggregate §  Periodically sends list of known playback timestamps and

NTP timestamps [10] §  Peers extend this list and/or update their PTS and NTP TS

Slide 10 of 19

M&F - EVALUATION

§  Y-axis denotes the average traffic generated per peer in kbit §  X-axis denotes the connectivity of the overlay network Slide 11 of 19

M&F - EVALUATION

§  Y-axis denotes the time required for the synchronization process §  X-axis denotes the connectivity of the overlay network Slide 12 of 19

DYNAMIC ADAPTIVE MEDIA PLAYOUT §  Increase/decrease playback rate

§  Pausing decreases QoE [7]

§  Overcome asynchronism by increasing or decreasing the playback rate §  Select those content sections which mask the playback rate variation

§  Content features for measuring the distortion caused by AMP §  Audio: spectral energy of audio frames §  Video: motion intensity between consecutive video frames

§  Metrics for the distortion §  Difference between the impaired and the unimpaired case for both

modalities

Slide 13 of 19

DYNAMIC ADAPTIVE MEDIA PLAYOUT §  Combined metric:

§  Constrained optimization problem for finding appropriate content sections:

x1...x2...x3...ξ...L...B...bc...br...T...tmax...

X ∈ R3

(1)argminX

f (X)

(2)x2 ⋅ (x3sign(ξ ) −1) ⋅ sign(ξ ) = ξ

(3)L ≤ B− x2 ⋅ x3 + x2 ⋅bcbr

(4)x1 ≤ T(5)x2 ≤ tmax

start time

duration playback rate asynchronism

lower buffer threshold current buffer fill state

bandwidth

content bit-rate

maximum start time

maximum duration

f (X) =|| d(X) ||2, d(X)= (dv (X),da (X))T

Slide 14 of 19

DYNAMIC AMP - EVALUATION §  Subjective Quality Assessment using Crowdsourcing

§  Microworkers §  15 minutes §  $0.25 as reward §  80 participants

§  Sequences §  Babylon A.D. for training

•  {1, 0.5, 2} times the nominal playback rate §  Big Buck Bunny for the main evaluation

•  {0.5, 0.6, 0.8, 1,1.2, 1.4, 1.6, 1.8, 2} times the nominal playback rate •  Selected content sections

Slide 15 of 19

DYNAMIC AMP - EVALUATION

§  Y-axis denotes the Mean Opinion Score, x-axis denotes the average f(X) and playback rate (µ)

§  High linear correlation between distortion metric and QoE •  µ > 1: •  µ < 1:

ρ = 0.975ρ = −0.995

Slide 16 of 19

Slide 17 of 19

§  Demo video available on YouTube §  https://www.youtube.com/watch?v=2V9rO5SbI7A §  Source Code available at: https://github.com/grishnagkh/mf

DEMO

CONCLUSION §  Introduced IDMS to Adaptive Streaming

§  MPEG-DASH

§  Distributed Control Scheme that scales with the number of peers §  Can be combined with any streaming protocol §  Not coupled with the session management or the overlay creation

§  Dynamic AMP for carrying out the actual synchronization §  (General) Optimization problem that aims on finding appropriate content

sections

§  Demo video available on YouTube §  https://www.youtube.com/watch?v=2V9rO5SbI7A §  Search for: MergeAndForward §  Source Code available at: https://github.com/grishnagkh/mf

Slide 18 of 19

THANK YOU!

Q & A

REFERENCES [4] F. Boronat Segu, J. Guerri Cebollada, and J. Lloret Mauri. An RTP/RTCP based approach for multimedia group and inter-stream synchronization. MTAP, pp. 40:285-319, 2008. [9] C. Hesselman, D. Abbadessa, W. Van Der Beek, D. Gorgen, K. Shepherd, S. Smit, M. Gulbahar, I. Vaishnavi, J. Zoric, D. Lowet, R. De Groote, J. O'Connell, and O. Friedrich. Sharing enriched multimedia experiences across heterogeneous network infrastructures. IEEE Comm. Mag., pp. 48(6):54-65, 2010. [10] T. Hossfeld, M. Seufert, M. Hirth, T. Zinner, P. Tran-Gia, and R. Schatz. Quantification of YouTube QoE via Crowdsourcing. In IEEE ISM, pages 494{499, 2011. [17] M. Montagud and F. Boronat. On the Use of Adaptive Media Playout for Inter-Destination Synchronization. IEEE Communications Letters, pp. 15(8):863-865, 2011. [18] M. Montagud, F. Boronat, and H. Stokking. Design and Simulation of a Distributed Control Scheme for Inter-destination Media Synchronization. In IEEE 27th AINA, pp. 937-944, March 2013. [24] H. Stokking, M. Van Deventer, O. Niamut, F. Walraven, and R. Mekuria. IPTV inter-destination synchronization: A network-based approach. In 14th ICIN, pp. 1-6, 2010.

Slide 20 of 19