How Netflix directs 1/3rd of - qconsf.com...How Netflix directs 1/3rd of Haley Tucker Mohit Vora...

How Netflix directs 1/3rd of

Haley Tucker Mohit Vora

QCon San FranciscoNov 16, 2015

Playback Overview

DATA PLANE(CDN)

CONTROL PLANE

STREAMNETFLIX DEVICE

Project 366 #59; 280212 Days Gone By..., CC BY-SA, Pete 2012, Flickr

AUDIOVIDEO TEXT

STREAMS

How do we build a streaming “tape”?

Determine the preferred experience

DEVICETITLE

CONNECTIONS

COUNTRY

NETWORK

Broadband - wired or wifiCellular - Edge, 3G, LTE, ...

CUSTOMER

That’s exactly what I want ...now where can I get it?

Point the device to appropriate locations

Steering

GENERATE PLAYBACK MANIFEST

PLAYBACK MANIFEST

Uh-oh, the content is encrypted!

Keymaster, CC BY-SA, Sean McGrath 2007, Flickr

LICENSE

And...Action!

SESSION (START, STOP, PAUSE, RESUME, KEEPALIVE)

SESSION EVENTS

LICENSE

PLAYBACK MANIFEST

GENERATE PLAYBACK MANIFEST

SESSION (START, STOP, PAUSE, RESUME, KEEPALIVE)

PLAYBACK LIFECYCLE

Data Plane(CDN)

What is a Content Delivery Network?

Open ConnectA NETFLIX ORIGINAL

CONTENT RANK

PREDICTABLE VIEWING PATTERNS

FILLING WHEN YOU SLEEP

g…, C

FILLING WHEN YOU SLEEP

Open ConnectA NETFLIX ORIGINAL

READ XOR WRITE

Content Delivery Mechanisms

DATA PLANE(CDN)

CONTROL PLANE

STREAM

ISP DATA CENTER

ISP ROUTER

NETFLIX DEVICE

STREAM

ISP DATA CENTER

ISP ROUTER

NETFLIX DEVICE

ISP CO-LOCATION

STREAM

ISP DATA CENTER

ISP ROUTER

NETFLIX DEVICE

STREAM

ISP DATA CENTER

NETFLIX DEVICE

IXP DATA CENTER

NFLXROUTER

ISP ROUTER

NETFLIX

STREAM

ISP DATA CENTER

NETFLIX DEVICE

IXP DATA CENTER

NFLXROUTER

ISP ROUTER

NETFLIX

STREAM

ISP DATA CENTER

NETFLIX DEVICE

IXP DATA CENTER

NFLXROUTER

ISP ROUTER

IXP INTERCONNECTION

NETFLIX

Control Plane

OPEN CONNECTSTREAMNETFLIX DEVICE

CDN CONTROL

DEVICE CONTROL

PLANEDON’T KEEP SECRETS

Network ProximityContent Positioning

Load Distribution

Network Proximity

Social Network in a Course, CC BY-SA, Hans Põldoja 2010, Flickr

By Specification?

By Specification?Doesn’t scale

Border Gateway Protocol

TAKEAWAY

BGP ROUTE175.231.128.0/24

(+ proximity attributes)

Use BGP

ISP2 DATA CENTER

ISP2 BGP ROUTES

CONTROLPLANE

IXP DATA CENTER

ISP1 BGP ROUTES

ISP1 DATA CENTER ISP1

BGP ROUTE175.231.128.0/24 (+ proximity attributes)

Content Positioning

LOCALIZE TRAFFIC

ISP DATA CENTER

SERVE CACHE MISS

HOW DO WE DETERMINE WHAT CONTENT WILL BE POPULAR TOMORROW?

CHANGING CATALOG

EVOLVING MEMBER TASTES

MINIMIZE FILL CHURN

ISP DATA CENTER

OFF PEAKFILL

USE HISTORICAL DATA

CONTENT RANKBY

bytesStreamed/bytesStored

IS ONE DAY OF HISTORY ENOUGH?

EXPONENTIALLY WEIGHTED MOVING AVERAGE

DAYS AGO0 10 20 30 40

TAKEAWAY Weigh Recent Data Higher

HOW SHOULD CONTENT BE ALLOCATED?

MILLIONS OF FILES

THOUSANDS OF SERVERS

HOW SHOULD CONTENT BE ALLOCATED?

SVR2SVR1

TAKEAWAY

ALLOCATE MULTIPLE REPLICASRESILIENT TO CLUSTER CHANGES

REPEATABLE

Consistent Hashing

ISP2 DATA CENTER

WHAT TO FILL?

CONTROLPLANE

IXP DATA CENTER

WHERE TO FILL FROM?

ISP1 DATA CENTER

FILL OVER HTTP

Load Distribution

CONTENT RANKBY

LOTS OF THROUGHPUT

LOTS OF STORAGE

CONTENT WITH CONFLICTING CONSTRAINTS

SSD BASED

SPINNING DISK BASED

WITHIN CLUSTERS ON EACH SERVER

MEMORY

CONTENT RANK

SSD SPINNING DISK

TAKEAWAY Tier Infrastructure

ACROSS SERVERS WITHIN CLUSTERS

BALANCE BALANCE

ACROSS EQUIDISTANT CLUSTERS

HOW DO WE BALANCE LOAD?

OPEN CONNECTNETFLIX DEVICE

CDN CONTROL

DEVICE CONTROL

LOAD BALANCER

STREAM

USING CONTENT DISTRIBUTION

HOW DO WE BALANCE LOAD?

FLIP A COIN

AND WHEN WE HAVE EQUALLY ATTRACTIVE LOCATIONS TO SERVE FROM –

INCIDENT LOAD

INSANESANE

HOW DO WE LOAD SERVERS OPTIMALLY?

… AMIDST EVER CHANGING INTERNET WEATHER

… AND DAILY TRAFFIC EBBS AND FLOWS

+ SERVE STREAMS

FEEDBACK

-TRAFFIC EFFECT ON

SYSTEM METRICS

CONTROL

WE INTRODUCE A FEEDBACK LOOP

TAKEAWAY PID CONTROLLER

Process Variable

Set Point

Control Variable

Current RPM

Desired RPM

Input Voltage

System Metrics

System Metrics Max

Controlled Traffic

DC MOTOR

TAKEAWAY PID CONTROLLER

Process Variable

Set Point

Control Variable

System Metrics

System Metrics Max

Controlled Traffic

Current RPM

Desired RPM

Input Voltage

LOADING SERVERS

ISP2 DATA CENTER

CONTROL TO 80%

CONTROLPLANE

IXP DATA CENTER

NO CONTROL

ISP1 DATA CENTER

0.0 < CONTROL VAR < 1.0

NEXT HOP

TRAFFIC SHIFTS TO NEXT HOP LOCATION

Steering

CDN CONTROL

PLANEPLAYBACK SERVICES

STEERINGGot URLs for f1, f2, …, fn?

Yes, here’s the URLs

PROXIMITYHEALTH

CONTENT

OPEN CONNECT

ArchitectureEvolution

5 CHALLENGES

STEERING

SESSION

MANIFEST

DRMLICENSE

How did we evolve from here...

STEERING

SESSION

MANIFEST

DRMLICENSE

CLIENT SCRIPTS

SERVICE LAYER

INSIGHTS

...to here.

5 SOLUTIONS

DEVICE

CUSTOMER

NETWORK

Broadband - wired or wifiCellular - Edge, 3G, LTE, ...

CONNECTIONS

COUNTRY

High dimensionalityCHALLENGE

How can we quickly alter the playback experience in a targeted manner?

ALL STREAMS

FOR CONTENT

ENGINE

BEST STREAMS

FOR SESSION

Stream FilteringUSE CASE

EXAMPLE RULES

ENGINE

CONFIGURATION MANAGEMENT UI

UPDATING RULES

PUBLISH

RULESSUBSCRIBE

Dynamic Business Rules

STEERING

SESSION

MANIFEST

DRMLICENSE

TAKEAWAY

Pinpoint what is brokenCHALLENGE

3:00 AM : Pager goes off

METRICS AND ALERTING

OK...error code 105 is elevated. But why?

Indexed Logging

Detailed Domain Insights

STEERING

SESSION

MANIFEST

DRMLICENSE

INSIGHTS

TAKEAWAY

Large amount of stateCHALLENGE

How can we enable faster UIs and low-end devices?

We introduced a server-side caching tier

MANIFESTSC

Watch out for resiliency issues!!

Ping Pong project, CC BY-SA, Michael Knowles 2008, Flickr

STEERING

SESSION

MANIFEST

DRMLICENSE

INSIGHTS

Reduce client stateTAKEAWAY

Managing device protocolsCHALLENGE

Square peg, round hole, CC BY-SA, Simon Law 2006, Flickr

Can we allow devices to define their own protocols?

DYNAMIC SCRIPTING PLATFORM

SESSION

LICENSE

MANIFESTXBOX

iPHONE

HTML5 PLAYER

iphone.groovy

xbox.groovy

html5.groovy

STEERING

SESSION

MANIFEST

DRMLICENSE

RULESINSIGHTS

Client-driven protocols

CLIENT SCRIPTS

SERVICE LAYER

TAKEAWAY

Enabling high-velocity innovationCHALLENGE

CC BY-SA, Nathan E Photography 2008, Flickr

How can we expose new data with the least amount of churn?

API MANIFEST

Stream● Bitrate● Framerate● Dynamic Data

Stream’● Bitrate● Dynamic Data

This works from API:● stream.getBitrate()● stream.getDynamicData().get(“FRAME_RATE”)

Works both ways!

This works from CLIENT SCRIPT!● stream.getDynamicData().get(“BIT_RATE”)● stream.getDynamicData().get(“FRAME_RATE”)

CLIENT SCRIPT

Stream’’● Dynamic Data

Works both

API MANIFEST

Stream● Bitrate● Framerate● Dynamic Data

Stream’● Bitrate● Dynamic Data

Works both

CLIENT SCRIPTS

SERVICE LAYER

STEERING

SESSION

MANIFEST

DRMLICENSE

INSIGHTS

Data pass-thruTAKEAWAY

TAKEAWAYS

● BGP based proximity● Tiered Infrastructure● PID Controller● EWMA for historical data● Consistent Hashing

● Dynamic business rules● Detailed domain insights● Reduce client state● Client-driven protocols● Data pass-thru

TAKEAWAYS

● BGP based proximity● Tiered Infrastructure● PID Controller● EWMA for historical data● Consistent Hashing

● Dynamic business rules● Detailed domain insights● Reduce client state● Client-driven protocols● Data pass-thru

Questions?Haley Tucker @hwilson1204

Mohit Vora@mohitvora

STREAM

NETFLIX DEVICE

STREAM

SPINNING DISK SERVERS

SSD SERVERS

WHAT TO FILL?

WHERE TO FILL FROM?

CLIENT SCRIPTS

SERVICE LAYER

CACHE CONTROL

DON’T KEEP SECRETSSTEERING

SESSION

MANIFEST

DRMLICENSE

INSIGHTS

IXP DATA CENTER

ISP2 BGP ROUTES

ISP1 BGP ROUTES

CONTROL TO 80%

● Background image from https://www.flickr.com/photos/centralasian/4099515384, Image was cropped and red lines and dots were drawn on top, https://creativecommons.org/licenses/by/2.0/.

● Image from https://www.flickr.com/photos/28705377@N04/4142872268, No modifications made, https://creativecommons.org/licenses/by/2.0/.

● Image of cassette is from https://www.flickr.com/photos/comedynose/6939206771, Image was cropped, https://creativecommons.org/licenses/by/2.0/.

● Image of speaker is from https://www.flickr.com/photos/av_hire_london/5578975575, No changes made, https://creativecommons.org/licenses/by/2.0/.

● Image of television is from https://www.flickr.com/photos/jvcamerica/3660897684/, No changes made, https://creativecommons.org/licenses/by/2.0/.

● Image of text is from https://www.flickr.com/photos/dno1967b/5754743006, No changes made, https://creativecommons.org/licenses/by/2.0/.

● Background image from https://www.flickr.com/photos/mcgraths/866572532, Image was cropped, https://creativecommons.org/licenses/by/2.0/.

● Image from https://www.flickr.com/photos/thatguyfromcchs08/2300190277, Image is dimmed, https://creativecommons.org/licenses/by/2.0/.

● Image from https://www.flickr.com/photos/mknowles/3134373590, Image was cropped, https://creativecommons.org/licenses/by-sa/2.0/.

Image Attributions

How Netflix directs 1/3rd of - qconsf.com...How Netflix directs 1/3rd of Haley Tucker Mohit Vora...

Documents

Netflix 2009

Netflix Social

Netflix Final

FINALEBusiness - Netflix

Pragmatic Cloud Security - cdn.ttgtmedia.com Mogull_Pragmatic Cloud...Netflix Security Monkey . Review Lock down management plane Focus on IAM for admins ... Commercial, hosted, or

Netflix Pricing

Netflix Report

How to get Netflix for life - Netflix lifetime account

Capétiens directs

Netflix Pitch

Netflix Recommendations

Ppt Netflix

Netflix 2.0

Netflix (2)

NETFLIX TRAFFIC CHARACTERIZATIONcarey/CPSC641/slides/netflix/Netflix... · NetFlix –Video Delivery •HTML5 Player (transitioned away from Silverlight) •Requests to the Web interface

lifesavercpr.net · ACLS Surveys Directs team to administer 100% oxygen Directs team to apply monitor leads Directs IV or 10 access Directs appropriate defibrillation and drug treatment

Netflix support phone number | Netflix Support Help Centre

NETFLIX Project

Netflix prize

Netflix Exemplar