Ion Stoica, CTO (also at UC Berkeley and Databricks)pv2013.itec.aau.at/wp-content/uploads/2013/10/Stoica.pdf · 2018. 7. 25. · ContentPublisher Team PMC:&RealG5me! Visibility!for!CDNs!!

Improving Video Quality by Information Sharing !!Ion Stoica, CTO !(also at UC Berkeley and Databricks) !!

Across many sites

While monitoring more than 4 Billion streams per month

We’ve seen 3 patterns

21.8

6

0

3

6

9

12

15

18

21

24

15.21

9.08

0

2

4

6

8

10

12

14

16

32.39

9.74

0

5

10

15

20

25

30

35

* June 2012

26.8

15.5

0

4

8

12

16

20

24

28

263% more 233% more 67.5% more

* June 2012

73% more

1 Viewers watch more when video is not interruptedUGC TV Channel Sport Aggregator

Buffering Impacted Views (BIV): > 2% buffering or > 5s continuous bufferingnon-BIV BIV

2 Viewers watch more when video is higher definition68

31

0

10

20

30

40

50

60

70

89

46

0

15

30

45

60

75

90

32.64

5.97

0

5

10

15

20

25

30

35

4.2

2.6

0

1

2

3

4

5

* July 2012 * June 2012 Live

46% more 119% more 62% more 447% more

Pre. Channel TV Channel TV Channel News

1 Mbps

Viewers leave and do not return to sites when video fails to start 3

Viewers who experience a single video start-up failure return 54% less

Every step of the delivery chain presents unique challenges to delivering video with low interruptions and high bit-rate

Encoders CDNs ISPs Devices

Inaccurate choice of bit-rates & poor matching of screen sizes & resolution to bit-rates

Wide variation in CDN performance across various ISPs with peak load, type of traffic and time of day

Heterogeneity of screen sizes and rendering performance across PCs, mobile hand-sets, tablets, game consoles, set top boxes, connected TVs …

Great diversity in access link speeds (fiber, cable, dsl, 3G/Wi-fi, corporate networks …)

CDNs Vary in Performance over Geographies and Time

There is no single best CDN across geographies, network, and time

25%

50%

25%

CDN 1

CDN 2

CDN 3

•  Metric: buffering ratio•  One month aggregated data-set

(2011)§  Multiple Flash (RTMP) customers§  Three major CDNs

•  31,744 DMA-ASN-hour with > 100 streams from each CDN§  DMA: Designated Market Area

•  Percentage of DMA-ASN-hour partitions a CDN has lowest buffering ratio

Washington DC (Hagerstown): ASN-CXA-ALL

10% 20% 100% 30% 40% 50% 60% 70% 80% 90%

Washington, DC viewer experience differed greatly… Comcast viewers got the best

streams from CDN 1 51% of the time and only 9% from CDN 2

Washington DC (Hagerstown): VZGNI-TRANSIT (19262)

Verizon users got the best streams from CDN 1 only 17% of the time and 77% from CDN 2

There is no single best CDN in the same geographic region or over time

CDN Streaming Failures Are Common Events" % of stream failures: % of streams with video start failures" Three months dataset (May-July, 2011) for a premium customer

using Flash

CDN (relative) performance varies greatly over time

Performance Changes Minute-by-minute

"   For the same ISP, CDN performance varies minute by minute over the course of a day

CDN1 CDN2

Buffering Ra5o

Video Start Failures

Average Bitrate

September 24th, 2013 (1 minute data-‐points over 24 hours)

Different quality metrics no always correlated

Viewer Centric Position

"   True end-point sensor to see what the viewer sees and inform the source in real time

Conviva Approach to Optimize Viewer Experience

Real-time Measurement from Every Viewer

Stopped/

Exit

Player

States Joining Playing Buffering Playing

Network/

stream

connection

established

Video

buffer

filled upVideo

buffer

empty

Buffer

replenished

sufficiently

time

User

action

Events

Player

Monitoring

Video download rate,

Available bandwidth,

Dropped frames,

Frame rendering rate, etc.

JoinTime (JT) BufferingRa2o(BR) RateOfBuffering(RB)

RenderingQuality(RQ)

Viewer Centric Position

Continuous Optimization

"   True end-point sensor to see what the viewer sees and inform the source in real time

"   Adjustments to streams every second to account for local environment and Internet variables

Global View and Policy Control

"   Dynamic policy control based on real-time patterns across viewers, affiliates, and networks

Conviva Approach to Optimize Viewer Experience

IP Video Streaming Architecture

Mezzanine Crea5on / Transcoding

CMS / Origin Storage

CDNs: Origin storage HDS fragmenta5on HLS fragmenta5on

Real-‐5me Heartbeats

Real-‐5me Feedback/ac5on

HLS Fragments

HDS/HLS Fragments

Conviva Pla.orm Real-‐5me data ingest and processing Real-‐5me metric computa5on Real-‐5me global op5miza5on Big-‐data analysis infrastructure

SmoothStreaming

CDN 1

CDN 2

Business Policy Input

Pulse: Real-‐5me Visibility for publishers

Content Publisher Team

PMC: Real-‐5me Visibility for CDNs

CDN Team

Offline Analysis and Recommenda5ons

Conviva Team

Mezzanine Crea5on / Transcoding

CMS / Origin Storage

CDNs: Origin storage HDS fragmenta5on HLS fragmenta5on

Real-‐5me Heartbeats

Real-‐5me Feedback/ac5on

HLS Fragments

HDS/HLS Fragments

SmoothStreaming

CDN 1

CDN 2

Business Policy Input

Pulse: Real-‐5me Visibility for publishers

Content Publisher Team

PMC: Real-‐5me Visibility for CDNs

CDN Team

Offline Analysis and Recommenda5ons

Conviva Team

Conviva Pla.orm Real-‐5me data ingest and processing Real-‐5me metric computa5on Real-‐5me global op5miza5on Big-‐data analysis infrastructure

IP Video Streaming Architecture

Conviva Platform

Real-‐5me Alerts

Con5nuous real-‐5me measurements from every client

Real-‐5me and historical Insights

Real-‐5me global op5miza5ons

Inference & Predic5on Engine

Bit Rates

CDNs

Decision Engine

Op5mize viewer performance by selec5ng the best op5on within the set of bit rates and CDNs

Akamai

DMA

ASN

DMA

ASN DMA

ASN

Limelight

Level3 Time of day

Localize issues by region, network, CDN, and 5me

Real-‐5me Global Data Aggrega5on and

Correla5on (Streaming Map-‐reduce)

Historical Data Aggrega5on and Analysis

(Hadoop+Hive+Spark)

Global Inference, Decision & Policy

Engine

Real-‐5me Alerts

Con5nuous real-‐5me measurements from every client

Real-‐5me and historical Insights

Real-‐5me global op5miza5ons

Bit Rates

CDNs Op5mize viewer

performance by selec5ng the best op5on within the set of bit rates and CDNs

Akamai

DMA

ASN

DMA

ASN DMA

ASN

Limelight

Level3 Time of day

Localize issues by region, network, CDN, and 5me

Real-‐5me Global Data Aggrega5on and

Correla5on (Streaming Map-‐reduce)

Historical Data Aggrega5on and Analysis

(Hadoop+Hive+Spark)

Global Inference, Decision & Policy

Engine

Inference & Predic5on Engine Decision Engine

Conviva Platform

Key Idea behind Inference & Prediction Engine

Share quality information across views

Use quality information from existing views to predict§  performance of new viewers at join time§  performance of existing views if we were to switch bitrate or

CDN

Create a model consisting of a set of decision tables

Reactively adapting after failure too late!

Inference and Decision Engines

Inference & Prediction Engine

Spark Cluster

Decision Engine

Update decision tables every 1min (5-8 sec processing)

Make decisions in constant time (

Spark•  In-memory computation engine

§  APIs in Scala, Python, Java•  Unifies batch, streaming, interactive computations

§  Powerful machine learning (MLlib) and soon graph (GraphX) libraries•  Used by tens of companies including Yahoo!, Intel

Spark

Spark

Streaming

Shark SQL

GraphX

MLlib

Streaming

Batch, Interactive

Interactive ,IterativeSophisticated algos.

Use Case 1: Best Starting Bit Rate for Single CDN

Pick the best starting bit rate based on decision tables…•  Device•  Connection type (3G, 4G,

wifi)•  Geo (DMA)•  ASN•  Protocol•  Player version•  Etc.

Desktop rate

ASN

rate

ASN

rate

ASN

iPad

iPhone

Quality (e.g., buf ra5o) of desktop users in ASN[1] x rate[1]

Use Case 1: Best Starting Bit Rate for Single CDN

Pick the best starting bit rate based on decision tables…•  Device•  Connection type (3G, 4G,


Desktop rate

ASN

rate

ASN

rate

ASN

iPad

iPhone

For an iPad in ASN[1] select highest bit rate providing good quality (i.e., rate[3])

Use Case 2: Best Starting CDN for Multi-CDNs

Pick the best CDN based on decision tables…•  Device•  Connection type (3G, 4G,


Desktop DMA

ASN

DMA

ASN

DMA

ASN

iPad

iPhone

Desktop DMA

ASN

DMA

ASN

DMA AS

N

iPad

iPhone

CDN2 CDN1

Use Case 2: Best Starting CDN for Multi-CDN

Pick the best CDN based on decision tables…•  Device•  Connection type (3G, 4G,


Desktop DMA

ASN

DMA

ASN

DMA

ASN

iPad

iPhone

Desktop DMA

ASN

DMA

ASN

DMA AS

N

iPad

iPhone

CDN2 CDN1

CDN1 >> CDN2 for ASN[2]xDMA[2]

Assign viewers in ASN[2]xDMA[2]

to CDN1

Use Case 3: CDN Switch on Quality DegradationDesktop DMA

ASN

Desktop DMA

ASN

CDN2 CDN1

0

10

20

30

40

0

1000

2000

3000

Buffer Length Trend

Available Bandwidth Trend

Detect and track buffer drop and bandwidth trends and take ac5on to prevent buffering

Client centric predic=ve algorithms track recent trends in available bandwidth, buffer length, rendering performance … and predict quality problems

Global Inference & Predic=on algorithms track global trends by geography, network (ASN), CDN… iden5fy anomalies and predict quality of views

Use Case 4: Asset Publishing and Caching IssuesCDN1 DMA

Asset

DMA

Asset

DMA

Asset

CDN2

CDN3

Global inference algorithms track individual asset failures by device geography, network (ASN), CDN… and iden5fy any regional anomalies

0 10 20 30 40 50 60

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Asset

% Video

Start Failures

Use Case 5: ISP Saturation

CDN1 DMA

ASN

DMA

ASN

DMA

ASN

CDN2

CDN3

This ASN/DMA is saturated on all three CDNs è Don’t switch CDN. Reduce bit rates and maintain

0

20

40

60

80

100

0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000

Band

width Fluctua5o

n

Peak Concurrent Viewers Network Satura5on Point

Use Case 5: ISP Saturation

CDN1 DMA

ASN

DMA

ASN

DMA

ASN

CDN2

CDN3

This ASN/DMA is saturated on all three CDNs è Don’t switch CDN. Reduce bit rates and maintain

0

10

20

30

40

0 5000 10000 15000 20000 25000 30000 35000

0

20

40

60

80

100

0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000

Band

width Fluctua5o

n Ba

ndwidth Fluctua5o

n

Peak Concurrent Viewers

Peak Concurrent Viewers

Network Satura5on Point

Unsaturated Network

Challenges" What happens if a partition doesn’t has enough data?

1)  Spatial aggregation

§  Which partition use for prediction?

?

CDN1 (buffering raBo)

DMA

ASN

Challenges" What happens if a partition doesn’t has enough data?

1)  Spatial aggregation

§  Which partition use for prediction?

2)  Temporal aggregation (increase window)

§  What window size?

?


DMA

ASN


DMA

ASN

Example 1: CDN and Bit Rate Selection

Example 1: Impact on CDN Usage"   ASN 7922 (Comcast) – for all of Aug 19th

Metric Buffering Ra=o Avg. Bitrate Video failures

CDN1 1.26 % 2270 K 9.3%

CDN2 1.08 % 2668 K 8.12%

CDN3 1.47 % 2451 9.9%

Metric Buffering Ra=o Avg. Bitrate Video failures

CDN1 2.12 % 1832 K 10.7%

CDN2 2.46 % 1874 K 11.0%

CDN3 2.09 % 1830 K 9.6%

"   ASN 20057 (AT&T Wireless) for all of Aug 19th

Precision picks CDN2 64% of the 5me

Precision picks CDN1 & CDN3 74% of the 5me

Example 1: Aggregate over 5 Days

Metric Non-‐Precision Precision Precision Improvement

Buffering Ra5o 2.3 % 1.5 % 33%

Average Bitrate 1692 K 2287 K 35%

Failures and Exits 11.5% 10.6 % 8% Buffering Impacted Views

13.4% 9.4 % 30%

Example 2: Preserving Quality in Presence of Failures!"   Precision ensures that audience quality is NOT impacted by CDN failures"   Content brands and audience are protected "   Content owners can be more aggressive in using capacity from CDN vendors

With Precision Video, CDN problem has no effect on viewers

Without Precision Video, CDN problem has big effect on viewers

Summary" Key transition of main-stream video to the Internet

" Video quality presents opportunity and challenge" Premium video on big screens à zero tolerance for poor quality

" Ability to infer and predict viewer quality key to maximize

quality perceived by users" Reacting after the fact too late!

Conviva Precision Protects Brands, Audience & CDN!"   Precision ensures that audience quality is NOT impacted by CDN failures"   Content brands and audience are protected "   Content owners can be more aggressive in using capacity from CDN vendors

With Precision Video, CDN problem has no effect on viewers

Without Precision Video, CDN problem has big effect on viewers

Other VoD Entertainment Competitors

Unint

erru

pted

View

s (%

)

Average Bitrate (kbps)

Conviva Precision: Simultaneously Increasing Resolution & Reducing Buffering

60

65

70

75

80

85

90

95

100

500 1000 1500 2000 2500

Highest Quality/Most Engaged Audience

(before Precision)

(aker Precision)

With Conviva Precision, Viewers Watch More, Come Back More Often

Before AfterConviva Conviva

Event Info EventDates Feb Mar

Views 41,652 49,607Uniques 9,930 11,448Viewed Minutes 855,002 1,160,667Minutes per View 20.5 23.4Minutes per Unique 86.1 101.4

Quality

Audience

Improvement

19%15%36%14%18%

Increased average bitrate !from 1.7 mbps to 2.1 mbps

Reduced views !impacted by buffering !from 16.13% to 5.56%

Raised engagement by 36%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

1600170018001900200021002200

Views Impacted by Buffering

Average Bit Rate

Before AfterConviva Conviva

Event Info EventDates Feb Mar

Views 41,652 49,607Uniques 9,930 11,448Viewed Minutes 855,002 1,160,667Minutes per View 20.5 23.4Minutes per Unique 86.1 101.4

Quality

Audience

Improvement

19%15%36%14%18%

First Full Month Results

Lift with Precision

The Truth"   Video delivery over the internet is hard

§  CDN variability makes it nearly impossible to deliver high quality all the time with just one CDN

SIGCOMM ‘12

The Idea"   Where there is heterogeneity, there is room for

optimization"   For each viewer we want to decide what CDN to stream

from"   But it’s difficult to model the internet, and things can

rapidly change over time"   So we will make this decision based on the real-time data

that we collect


§  CDN variability makes it nearly impossible to deliver high quality everywhere with just one CDN

SIGCOMM ‘12


§  CDN variability makes it nearly impossible to deliver high quality all the time with just one CDN

SIGCOMM ‘12

The Idea"   Where there is heterogeneity, there is room for

optimization"   For each viewer we want to decide what CDN to stream

from"   But it’s difficult to model the internet, and things can

rapidly change over time"   So we will make this decision based on the real-time data

that we collect



The Idea

Avg. buff ra5o of users in Region[1] streaming from

CDN1

Avg. buff ra5o of users in Region[1] streaming from

CDN2

"   For each CDN, partition clients by City

"   For each partition compute Buffering Ratio

Seal

le San

Francisco

New

York

Lond

on Hon

g Ko

ng

The Idea"   For each partition select best CDN

and send clients to this CDN



Seal

le San

Francisco

New

York

Lond

on Hon

g Ko

ng





CDN2 >> CDN1

Seal

le San

Francisco

New

York

Lond

on Hon

g Ko

ng





Seal

le San

Francisco

New

York

Lond

on Hon

g Ko

ng

Best CDN (buffering raBo)

The Idea



"   For each partition select best CDN and send clients to this CDN

Seal

le San

Francisco

New

York

Lond

on Hon

g Ko

ng

The Idea




"   What if there are changes in performance? S

eall

e San

Francisco

New

York

Lond

on Hon

g Ko

ng

The Idea




"   Use online algorithm respond to changes in the network. S

eall

e San

Francisco

New

York

Lond

on Hon

g Ko

ng

Content owners (CMS & Origin)

CDN 1

CDN 2

CDN 3

Clients

Coordinator

Con5

nuou

s measuremen

ts

Busin

ess P

olicies

control

How?"   Coordinator implementing an optimization algorithm that

dynamically selects a CDN for each client based on§  Individual client§  Aggregate statistics§  Content owner policies

"   All based on real-time data

What processing framework do we use?"   Twitter Storm

§  Fault tolerance model affects data accuracy§  Non-deterministic streaming model

"   Roll our own§  Too complex§  No need to reinvent the wheel

"   Spark§  Easily integrates with existing Hadoop architecture§  Flexible, simple data model§  Writing map() is generally easier than writing update()

Spark Usage for Production

Decision Maker Decision Maker Decision Maker

"   Compute performance metrics in spark cluster

"   Relay performance information to decision makers

Spark Cluster Spark Node

Spark Node

Spark Node

Spark Node Spark

Node

Storage Layer

Clients

Results

75%

80%

85%

90%

95%

100%

0 500 1000 1500 2000 2500

Non-‐buffering views

Average Bitrate (Kbps)

Spark’s Role"   Spark development was incredibly rapid, aided both by its excellent

programming interface and highly active community"   Expressive:

§  Develop complex on-line ML decision based algorithm in ~1000 lines of code

§  Easy to prototype various algorithms"   It has made scalability a far more manageable problem"   After initial teething problems, we have been running Spark in a

production environment reliably for several months.

Problems we faced"   Silent crashes…"   Often difficult to debug, requiring some tribal knowledge"   Difficult configuration parameters, with sometimes inexplicable

results"   Fundamental understanding of underlying data model was essential

to writing effective, stable spark programs

Enforcing constraints on optimization"   Imagine swapping clients until an

optimal solution is reached

Constrained Best CDN (buffering raBo)



50%

50%

Enforcing constraints on top of optimization"   Solution is found after clients have already joined."   Therefore we need to parameterize solution to clients

already seen for online use."   Need to compute an LP on real time data"   Spark Supported it

§  20 LPs§  Each with 4000 decisions variables and 350 constraints§  5 seconds.

Tuning

"   Can’t select a CDN based solely on one metric.§  Select utility functions that best predict engagement

"   Confidence in a decision, or evaluation will depend on how much data we have collected§  Need to tune time window§  Select different attributes for separation

Tuning

"   Need to validate algorithm changes quickly"   Simulation of algorithm offline, is essential

Spark Usage for Simulation

Spark Node

Spark Node

Spark Node

Spark Node

Spark Node

HDFS with Produc5on traces

"   Load production traces with randomized initial decisions

"   Generate decision table (with artificial delay)

"   Produce simulated decision set"   Evaluate decisions against

actual traces to estimate expected quality improvement

Future of Spark and Conviva"   Leverage spark streaming"   Unify live and historical processing"   Develop platform to build various processing ‘apps’ (e.g.

Anomaly Detection, Customer Tailored Reporting)§  Can share the same data API§  Will all have consistent input

In Summary"   Spark was able to support our initial requirement of fast

fault tolerant performance computation for an on-line decision maker

"   New complexities like LP calculation ‘just worked’ in the existing architecture

"   Spark has become an essential tool in our software stack

Real-time Measurement from Every Viewer

Stopped/

Exit

Player

States Joining Playing Buffering Playing

Network/

stream

connection

established

Video

buffer

filled upVideo

buffer

empty

Buffer

replenished

sufficiently

time

User

action

Events

Player

Monitoring

Video download rate,

Available bandwidth,

Dropped frames,

Frame rendering rate, etc.

JoinTime (JT) BufferingRa2o(BR) RateOfBuffering(RB)

RenderingQuality(RQ)

An UK ISP: Top Metros by Metric over timeGreater London

Op5mizing for Buffering Ra5o

Op5mizing for Video Start Failures

Slough

Op5mizing for Average Bitrate

Birmingham

Greater London

Greater London

Slough

Slough

Birmingham

Birmingham

Time (1 min data-‐points over 24 hours)

Measurements of 3 Leading CDN Show Significant Variation Over Geo and ISP

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Washington DC

(Hagerstow

n):CMCS(33657)

Milw

aukee:RO

ADRU

NNER

-‐CEN

TRAL(20231)

Green Ba

y -‐ A

ppleton:SCRR

-‐7015(7017)

Denver:ASN

-‐QWEST(20

9)

Charlole:SCRR

-‐11426(11426)

Washington DC

(Hagerstow

n):ASN

-‐CXA

-‐ALL-‐

Philade

lphia:VZ

GNI-‐T

RANSIT(19262)

San Diego:SBIS-‐AS(7132)

Las V

egas:ASN

-‐CXA

-‐LV-‐13432-‐CB

S(13432)

Madiso

n:CH

ARTER-‐NET-‐HKY-‐NC(20115)

Indianapolis:SBIS-‐AS(7132)

Providen

ce -‐ New

Bed

ford:ASN

-‐CXA

-‐ALL-‐CCI-‐22773-‐

Washington DC

(Hagerstow

n):VZG

NI-‐T

RANSIT(19262)

Harrord & New

Haven

:SBIS-‐AS

(7132)

Houston:SBIS-‐AS(7132)

Grand Ra

pids -‐ Kalamazoo

-‐ Ba

lle Creek:SBIS-‐

Atlanta:BE

LLSO

UTH

-‐NET-‐BLK(6389)

Hono

lulu:HAW

AIIAN-‐TELCO

M(36149)

Atlanta:CO

MCA

ST-‐7725(7725)

Washington DC

(Hagerstow

n):SPC

S(10507)

Orla

ndo -‐ D

aytona Beach -‐ Melbo

urne

:EMBA

RQ-‐

Denver:CMCS(33652)

Norfolk -‐ Po

rtsm

outh -‐ New

port New

s:AS

N-‐CXA

-‐ALL-‐

Saint Lou

is:CH

ARTER-‐NET-‐HKY-‐NC(20115)

Cincinna5:FU

SE-‐NET(6181)

Phoe

nix:AS

N-‐QWEST(209)

Dallas -‐ Fort W

orth:VZG

NI-‐T

RANSIT(19262)

Pilsburgh:CCC

H-‐3(7016)

Saint Lou

is:SBIS-‐AS(7132)

San Francisco -‐ O

akland

-‐ San Jose:SBIS-‐AS

(7132)

San Diego:RO

ADRU

NNER

-‐WEST(20001)

Greenville -‐ Spartansburg -‐ A

sheville -‐

Kansas City

:SBIS-‐AS

(7132)

Columbu

s -‐ OH:SCRR

-‐10796(10796)

Louisville:INSIGH

T-‐CO

MMUNICAT

IONS-‐CO

RP-‐

Chicago:AT

T-‐INTERN

ET3(6478)

Kansas City

:ASN

-‐CXA

-‐ALL-‐CCI-‐22773-‐RDC

(22773)

Miami -‐ Fort Laude

rdale:BE

LLSO

UTH

-‐NET-‐BLK(6389)

Cleveland:NEO

-‐RR-‐CO

M(11060)

Chicago:VZ

GNI-‐T

RANSIT(19262)

Columbia -‐ SC:SCRR

-‐11426(11426)

Dallas -‐ Fort W

orth:SBIS-‐AS

(7132)

Detroit:S

BIS-‐AS

(7132)

Kansas City

:SCR

R-‐11955(11955)

Los A

ngeles:CHA

RTER

-‐NET-‐HKY-‐NC(20115)

Miami -‐ Fort Laude

rdale:CO

MCA

ST-‐20214(20214)

Cleveland:SCRR

-‐10796(10796)

West P

alm Beach -‐ Fort

San Diego:AS

N-‐CXA


(22773)

Seal

le -‐ Tacoma:AS

N-‐QWEST(209)

New

York:CA

BLE-‐NET-‐1(6128)

Portland

-‐ OR:AS

N-‐QWEST(209)

Oklahom

a City:SBIS-‐AS

(7132)

Phoe

nix:AS

N-‐CXA


(22773)

Chicago:SBIS-‐AS(7132)

Greensbo

ro -‐ High Point -‐ Winston

-‐Albany -‐ Sche

nectady -‐ T

roy:RR

-‐NYSRE

GION-‐

Tyler -‐ Lon

gview (Lut

in &

Grand Ra

pids -‐ Kalamazoo

-‐ Ba

lle Creek:CMCS(33668)

Houston:CM

CS(33662)

Harrord & New

Haven

:COMCA

ST-‐7015(7015)

Eugene

:ASN

-‐QWEST(209)

San Francisco -‐ O

akland

-‐ San Jose:CMCS(33651)

Documents

Ion Stoica, CTO (also at UC Berkeley and Databricks)pv2013.itec.aau.at/wp-content/uploads/2013/10/Stoica.pdf · 2018. 7. 25. · ContentPublisher Team PMC:&RealG5me! Visibility!for!CDNs!!