1

CPS 214Computer Networks and Distributed Systems

“Live” Video and Audio Streaming•End System Multicast•Analysis of Akamai Workload

2

Presentations• Monday, April 21

– Abhinav, Risi– Bi, Jie– Jason, Michael– Martin, Matt– Amre, Kareem

• Wednesday, April 23– Ben, Kyle– Jayan, Michael– Kshipra, Peng– Xuhan, Yang

The Feasibility of Supporting Large-Scale Live Streaming Applications with Dynamic

Application End-Points

Kay Sripanidkulchai,

Aditya Ganjam, Bruce Maggs*, and Hui Zhang

Carnegie Mellon University

* and Akamai Technologies

4

Motivation• Ubiquitous Internet broadcast

– Anyone can broadcast– Anyone can tune in

5

Overlay multicast architectures

RouterSourceApplication end-point

6

Infrastructure-based architecture[Akamai]

+ Well-provisionedRouterSourceApplication end-pointInfrastructure server

7

Application end-point architecture[End System Multicast (ESM)]

+ Instantly deployable+ Enables ubiquitous broadcast

RouterSourceApplication end-point

8

Waypoint architecture [ESM]

+ Waypoints as insurance RouterSourceApplication end-pointWaypoint

W

W

9

Sample ESM Broadcasts http://esm.cs.cmu.edu

Event Duration(hours)

Unique Hosts

Peak Size

SIGCOMM ’02 25 338 83

SIGCOMM ’03 72 705 101

SOSP’03 24 401 56

DISC’03 16 30 20

Distinguished Lectures

11 400 80

AID Meeting 14 43 14

Buggy Race 24 85 44

Slashdot 24 1609 160

Grand Challenge

6 2005 280

10

Feasibility of supporting large-scale groups with an application end-point architecture?

• Is the overlay stable enough despite dynamic participation?

• Is there enough upstream bandwidth?• Are overlay structures efficient?

11

Large-scale groups

• Challenging to address these fundamental feasibility questions– Little knowledge of what large-scale live streaming

is like

12

Chicken and egg problem

Publishers with compelling content need proof that the system works.

System has not attracted large-scale groups due to lack of compelling content.

13

The focus of this paper

• Generate new insight on the feasibility of application end-point architectures for large scale broadcast

• Our methodology to break the cycle– Analysis and simulation– Leverage an extensive set of real-world workloads

from Akamai (infrastructure-based architecture)

14

Talk outline

• Akamai live streaming workload• With an application end-point architecture

– Is the overlay stable enough despite dynamic participation?

– Is there enough upstream bandwidth?– Are overlay structures efficient?

• Summary

15

Measurements used in this study

• Akamai live streaming traces– Trace format for a request

[IP, Stream URL, Session start time, Session duration]

• Additional measurements collected– Hosts’ upstream bandwidth

An Analysis of Live Streaming Workloads on the Internet

Kunwadee Sripanidkulchai, Bruce Maggs*, Hui ZhangCarnegie Mellon University and

*Akamai Technologies

17

Akamai live streaming infrastructure

A

A

A

A

A

A

Reflectors Edge servers

Source

18

Extensive traces

~ 1,000,000 daily requests ~ 200,000 daily client IP addresses from over

200 countries~ 1,000 daily streams~ 1,000 edge servers~ Everyday, over a 3-month period

19

Largest stream

75,000 x 250 kbps = 18 Gbps!

20

Highlight of findings

• Popularity of events [Bimodal Zipf]

• Session arrivals [Exponential for short time-scales, time-of-day and time-zone-correlated behavior, LOTS of flash crowds]

• Session durations [Heavy-tailed]

• Transport protocol usage [TCP rivals UDP]

• Client lifetime• Client diversity

21

Request volume (daily)

Weekdays

Weekends

Missing logs

Num

ber

of r

eque

sts

22

Audio vs. video

Unknown 22%

Audio 71%

Video 7%

Most streams are audio.

23

Stream types• Non-stop (76%) vs. short duration (24%)

– All video streams have short duration

• Smooth arrivals (50%) vs. flash crowds (50%)– Flash crowds are common

24

Client lifetime

• Motivating questions– Should servers maintain “persistent” state about

clients (for content customization)?– Should clients maintain server history (for server

selection problems)?

• Want to know– Are new clients tuning in to an event?– What is the lifetime of a client?

25

Analysis methodology

• Windows media format• Player ID field to identify distinct users

• Birth rate = Number of new distinct users

Total number of distinct users

26

Daily new client birth rate

• New client birth rate is 10-100% across all events.• For these 2 events, birth rate is 10-30%.

Weekends

Weekdays Xmas

27

One-timers: tune in for only 1 day

In almost all events, 50% of clients are one-timers!

28

Client lifetime (excluding one-timers)

y = x

y = 3x

For most events, average client lifetime is at least 1/3 of the event duration.

29

Client lifetime

• Motivating questions– Should servers maintain “persistent” state about

clients (for content customization)? Any state should time-out quickly because most clients are one-timers.

– Should clients maintain server history (for server selection problems)? Yes, recurring clients tend to hang around for a while.

30

Where are clients from?

0.00E+00

2.00E+05

4.00E+05

6.00E+05

8.00E+05

1.00E+06

1.20E+06

1.40E+06

1.60E+06 US

CN

DE

ES

FR

GB

CA

JP

PT

CH

BE

MX

NL

SE

KR

BR

Countries

Num

ber

of I

P A

ddre

sses

Clients are from over 200 countries.Most clients are from the US and Europe.

31

Analysis methodology

• Map client IP to location using Akamai’s EdgeScape tool

• Definitions– Diversity index = Number of distinct ‘locations’ that

a stream reaches– Large streams are streams that have a peak group

size of more than 1,000 clients

32

Time zone diversity

Almost all large streams reach more than half the world.

Many small streams reach more than half the world!

33

Client diversity

• Motivating questions– Where should streaming servers be placed in the

network? Clients are tuning in from many different locations.

– How should clients be mapped to servers? For small streams which happen to have a diverse set of clients, it may be too wasteful for a CDN to map every client to the nearest server.

34

Summary

• Publishers are using the Internet to reach a wider audience than traditional radio and TV

• Interesting observations– Lots of audio traffic– Lots of flash crowds (content-driven behavior)– Lots of one-timers– Lots of diversity amongst clients, even for small

streams

35

Nice pictures…see paper for details

0

20

40

60

80

100

rtp

http

mms

rtsp

Quicktime Real Windows media

Per

cent

age

of r

eque

sts

36

Abandon all hope, ye who enter here.

37

Talk outline

• Akamai live streaming workload• With an application end-point architecture

– Is the overlay stable enough despite dynamic participation?

– Is there enough upstream bandwidth?– Are overlay structures efficient?

• Summary

38

When is a tree stable?

Not stable More stable

• Departing hosts have no descendants

• Stable nodes at the top of the tree

X

XX

Stable nodes

Less stablenodes

Interruptions

TimeAncestor leaves

39

Extreme group dynamics

45% stay less than 2 minutes!

15% stay longer than 30 minutes(heavy-tailed)

40

Stability evaluation: simulation

• Hosts construct an overlay amongst themselves using a single-tree protocol– Skeleton protocol of the one presented in the ESM

Usenix ’04 paper• Findings are applicable to many protocols

– Goal: construct a stable tree• Parent selection is key

• Group dynamics from Akamai traces (join/leave)

• Honor upstream bandwidth constraints– Assign degree based on bandwidth estimation

41

Join

IP1IP2...

Join

42

Probe and select parent

IP1

IP2

...

IP1

IP2

43

Probe and select parent

• Oracle: pick a parent who will leave after me • Random • Minimum depth (select one out of 100 random)• Longest-first (select one out of 100 random)

Parent selection algorithms

44

Parent leave

Host leaves

X

45

Parent leave

Host leaves

All descendants are disconnected

?

46

Find new parent

Host leaves

All descendants are disconnected

All descendants probe to find new parents

47

Stability metrics

• Mean interval between ancestor change

• Number of descendants of a departing host

X

Interruptions

TimeAncestor leaves

48

Stability of largest stream

Oracle: there is stability!

Min depth

Random

Longest-first

49

Min depth, 82%

Random, 72%

Longest-first, 91%

Oracle, ~100% no descendants

Is longest-first giving poor predictions?

50

Stability of 50 large-scale streams

Min depthRandom

Longest-first

OracleThere is stability! Of the practical algorithms, min depth performs the best.

Per

cent

age

of s

essi

ons

with

inte

rval

betw

een

ance

stor

cha

nge

< 5

min

utes

51

There is inherent stability

• Given future knowledge, stable trees can be constructed

• In many scenarios, practical algorithms can construct stable trees – Minimum depth is robust– Predicting stability (longest-first) is not always

robust; when wrong, the penalty is severe

• Mechanisms to cope with interrupts are useful– Multiple trees (see paper for details)

52

Talk outline

• Akamai live streaming workload• With an application end-point architecture

– Is the overlay stable enough despite dynamic participation?

– Is there enough upstream bandwidth?– Are overlay structures efficient?

• Summary

53

Is there enough upstream bandwidth to support all hosts?

What if application end-points are all DSL?

Video 300 kbps

Upstream bandwidth only 128 kbps

DSLDSL

Saturated tree

DSL

54

Metric: Resource index

• Ratio of the supply to the demand of upstream bandwidth Resource index == 1 means the system is saturated

• Resource index == 2 means the system can support two times the current members in the system

Resource Index:

(3+5)/3 = 2.7

55

Large-scale video streams

Most streams have sufficient upstream bandwidth.

A few streams are in trouble or close.

1/3 of the streams are in trouble.

56

Talk outline

• Akamai live streaming workload• With an application end-point architecture

– Is the overlay stable enough despite dynamic participation?

– Is there enough upstream bandwidth?– Are overlay structures efficient?

• Summary

57

Relative Delay Penalty (RDP)• How well does the overlay structure match the

underlying network topology?

RDP = Overlay distance

Direct unicast distance

US

USEurope

US

US Europe

50ms

50ms 50ms

20ms

Results are more promising than previous

studies using synthetic workloads and topologies.

58

Summary

• Indications of the feasibility of application end-point architectures– The overlay can be stable despite dynamic participation– There often is enough upstream bandwidth– Overlay structures can be efficient

• Our findings can be generalized to other protocols• Future work: Validate through real deployment

– On-demand use of waypoints in End System Multicast– Attract large groups

Thank you!