Netflix Edge Engineering Open House Presentations - June 9, 2016

Daniel Jacobson@daniel_jacobson

Satish Gudiboina@sgudiboina

Suudhan Rangarajan@suudhan

Vasanth Asokan@vasanthasokan

Edge Engineering Open House - June 9, 2016

190 Countries (not China and a few others)

81+ Million Subscribers

1000+ Different Device Types

Over 42 Billion Hours Streamed in 2015

Streaming Hours Per Year in Billions

Over 42 Billion Hours Streamed in 2015

Over 42 BillionSuccesses!

Of Course, There Are Failures Too…

Two Primary Drivers Behind Our Successes

People Desire to Watch Netflix

Systems Scale to Meet Desires

Sign-Up

Discovery / Browse

Sign-Up

Discovery / Browse

Playback

Edge Engineering provides data and

functionality to support these

three experiences

Designing APIs

EnablingPlayback Scaling

Routing

InsightsDX

Resiliency

Edge Engineering provides data and

functionality to support these

three experiences

DEVICES

ROUTING

DEVICES

ROUTING

DEVICES

ROUTING

API API API API API API

DEVICES

ROUTING

SERVICES

S2S2RecsS2S2Member

S2S2RatingsS2S2Playback LifecycleS2S2Authn/z

S2S2A/BS2S2Search

S2S2IdentityS2S2 S2S2Playback Data S2S2DRMMetadata

DEVICES

ROUTING

SERVICES

S2S2RecsS2S2Member

S2S2RatingsS2S2S2S2Authn/z

S2S2A/BS2S2Search

S2S2IdentityS2S2Metadata

S2S2Playback Data S2S2DRM

Ownedby Edge

Engineering

Playback Lifecycle

DEVICES

ROUTING

SERVICES

S2S2RecsS2S2Member

S2S2A/BS2S2Search

Playback Lifecycle

DEVICES

ROUTING

SERVICES

S2S2RecsS2S2Member

S2S2A/BS2S2Search

Playback Lifecycle

DEVICES

ROUTING

SERVICES

S2S2RecsS2S2Member

S2S2A/BS2S2Search

Playback Lifecycle

DEVICES

ROUTING

SERVICES

S2S2RecsS2S2Member

S2S2A/BS2S2Search

Playback Lifecycle

S2S2S2S2Authn/z

S2S2Playback Data S2S2DRM

INSIGHTS

Playback Lifecycle

42 Billion Hours2015

200 Billion Hours

Future

42 Billion Hours

The rest of

Netflix’s AWS Cloud Footprint by %

Talking About the Future of Edge Engineering

Satish GudiboinaAPI and Upcoming Re-Architecture

Suudhan RangarajanPlayback Experience

Vasanth AsokanDeveloper Tools, Velocity and Experience

The Netflix API Platform for Server-Side Scripting

Current and The FutureSatish Gudiboina

The Netflix API

Streaming Hours Per Year in Billions

Scale is multi-faceted

Growing number of users ( → RPS)

Growing number of device types

Growing number of A/B tests

Growing number of languages

Growing number of countries

What we need to build for

Velocity

Resiliency

Other requirements:PerformanceGreat developer experienceOperational insightsTooling

Js(mostly)

Client AClient BClient C

Client A

Client YClient Z

Netflix Microservices

script

Network boundary

API Server JVM

Today’s architecture

Resiliency with Hystrix

Developer Velocity: Decoupled deployments of versions

API device 1 device 2 device 3 device 4

i+1i+2i+3

Changing risk profile

Growing number of users ( → RPS)

Growing number of devices

Growing number of A/B tests

Growing number of languages

Growing number of countries

Growing number and complexity of scripts (scripts → apps)

Js(mostly)

Client A

Client YClient Z

script

Network boundary

API Server JVM

Today’s system (T-3yrs)

few, small scriptsfewer uploads

(mostly)java

Client A

Client YClient Z

script

Network boundary

API Server JVM

Today’s system (T)

scripts

hundreds of more complex scripts,10-50 uploads per day

What we need

Velocity

Resiliency?

Lack of process isolation is a growing risk.

Moving toward our ideal API:What will change

Scripts will run in containers

Scripts will call API remotely

(mostly)java

Client A

Client YClient Z

node script

Network boundary API Server JVM

The (near) future

node.js

process isolation

node for device teams

Why containers?

Process isolation

Fast startup

Consistent developer experience across environments

Isolated failures: scripts don’t affect each other

device 1 device 2 device 3 device 4Temporarily unavailable!

Independent autoscaling

device 1 device 2 device 3 device 4

Fast startup

New API server: minutesNew container: seconds

Fast rollout, fast rollback, fast MTTR

The Netflix API

Edge Developer ExperienceTranslating developer productivity to Netflix customer delight

Developer Experience?

DEVELOP(rapidly)

DEPLOY(reliably)

OPERATE(effectively)

Experimentation driven innovation

~700 apps, dozens of pushes a day15+ client teams, ~200 developers

~50 direct services, 100s of AB tests, dozens of new features

The Innovation Funnel

Devices

Netflix Services

Client Adaptor Applications

Why care about DevEx?

DeveloperProductivity

ProductInnovation

Automation

Insights

CustomerSatisfaction

App Development and Management

DEVELOP(rapidly)

DEPLOY(reliably)

dary API SERVER JVM

js java

Developer Ergonomics

Large / Complex

API SERVER JVM

Developer Ergonomics ...

js javajs

DOCKER CONTAINERS

ryNetflix

Microservices

Setup Canary

SupportProd Push

Pre-Prod

MetricsTracing

Lifecycle

Alerts

Bootstrap

API Discovery

Unit Test

SDK Debug Logging

Profiling

Audits

Security

Custom Routing

Dependency Management

Client Application Development Critical Component!

Dx Developer Experience

$ newt init

Just bring your Javascript business logic

NeWT: Netflix Workflow Toolkit

Continuous Integration

Deployment Pipelines

Autoscaling

Dashboards

Alerting

Logging

Lifecycle Management

Audits and Analytics

Container tooling

Canaries

Dependency Management

NeWT: Netflix Workflow Toolkit

Edge PaaS UI

$ newt auto-deploy -d

nodeJSproject

Docker Machine

node-inspector

DebuggerFile watcher / live reload trigger

File watcher agent

NeWT: Local Container Development

Local Container

docker build / run

$ newt auto-deploy -d

Docker Machine

NeWT: Local Container Development

Local Container

CloudMicroservices

Cloud Proxy

Terminate security

Service Discover

App Operations and Insights

DEVELOP(rapidly)

DEPLOY(reliably)

• Low Latency, High throughput, Highly Efficient• Handle bursty or large scale loads• Extensible programming model

600 jobs in production, 8M messages/sec at peak, 100Gbps network throughput

Mantis - Stream Processing Platform

Monitoring facets of aggregate application health, globally

Aggregate Insights

Analyze in real-time, requests matching a precise set of conditions

Surgical Insights

Surgical Insights - Real-time Stream Queries

Monitoring server side calling pattern and internal application profile

Session Tracing

Session Tracing - Request Profile

Session Tracing - Per Node Profile

Automatic monitoring of high cardinality data across multiple dimensions

Real-time Anomaly Detection

• Scaling developer productivity with business growth

• Provide fully managed PaaS experience to client developers • Shift Left Insights to power smart development• Curated, blended visualizations that simplify devops

In conclusion...

Tech Soup

Scaling Playback Services

Suudhan Rangarajan Senior Software Engineer, Playback Features

@suudhan

Playback Lifecycle

DECIDE

COLLECT & LEARN

AUTHORIZE

Decide

MANIFEST (Tracks and URLs)

Authorize

LICENSE

❏ Content usage / resolution policies

❏ Plan / device limits enforcement

❏ DRM / License generation

Collect & Learn

Bookmarks & Hours Watched

Streaming Errors and Metrics

Quality Of Experience metrics

Lets look at Play Decisions

DECIDE

MANIFEST

AUTHORIZE

COLLECT & LEARN

LICENSE

SESSION

Huge number of Streams

Resolutions - 720p, 1080p, 4K etcCodecs - H.264,HEVC etcBitrates - 230, 780, 3000 etc

Channels - Stereo, Surround SoundLanguages - English, French etc

Types - Subtitles, Closed Captions, Forced NarrativesLanguages - English, French etc

Streams to Tracks

- H.264 Main Profile- English 5.1 Audio- No Subtitle

- HEVC Dash Profile- French 2.0 Audio- English CC

- HDR Dash Profile- Spanish AAC Audio- English Forced Narrative

Decide & Filter

MANIFEST SERVICE

Many Many Dimensions

PLAYBACKMANIFEST

USER PREFERENCES

TITLEMETADATA

COUNTRY

DEVICE

NETWORK

Big Opportunity

Rich playback experiences

Tremendous increase in scale

Customer growth

Challenge: Efficient Scaling

Targeting sub-linear growth

# of Requests

Cloud Costs

Predictable Viewing Patterns

Key Insight

CONTENT RANK

Also..Manifest Request for one title

Current: Completely Real-time

Real-time manifest generation

With Caching

Real-time manifest generation

80% Cached20% Real-time

Challenges

How do we determine the optimal combination of attributes to cache on?

Challenges

Cache Considerations: ●When to populate?●When to bust?●How to scale for

cache-miss or failures?

Potential Win

10x increase in requests with only 4x increase in costs

Optimize computation

Can we re-imagine our service processing to dramatically increase throughput?

Anatomy of a Playback Manifest Request

Metadata Access

Tracks Generation

Streams Filtering

Serialization

Potential Win

10x increase in requests with just 2x increase in service costs

Two-pronged Strategy to Scaling

Cache Manifests

Re-architect code to reduce processing time

Scaling Problems Across Services

Decide Authorize Collect & Learn

Playback Features

Playback Access

Playback Data Systems

Thanks!

@suudhan

Come Talk to Us!

Image AttributionAll Images used are under creative commons or public domain license:

● Video icon - http://simpleicon.com/wp-content/uploads/video-camera-1.png● Speaker icon -

https://upload.wikimedia.org/wikipedia/commons/thumb/2/21/Speaker_Icon.svg/1024px-Speaker_Icon.svg.png

● Subtitle icon - https://thenounproject.com/term/subtitles/78795/ ● Uptrend image - https://pixabay.com/en/chart-line-line-chart-diagram-trend-148256/ ● Funnel image - https://commons.wikimedia.org/wiki/File:Funnel_Mech.svg ● Business Intelligence image -

https://pixabay.com/static/uploads/photo/2015/04/14/23/17/it-business-722950_960_720.png ● Key icon - https://pixabay.com/static/uploads/photo/2014/04/03/10/55/key-311738_960_720.png ● Person icon-

https://pixabay.com/static/uploads/photo/2015/12/22/04/00/photo-1103596_960_720.png ● Mobile icon-

https://upload.wikimedia.org/wikipedia/commons/thumb/1/14/Mobile_phone_font_awesome.svg/1024px-Mobile_phone_font_awesome.svg.png

● Globe image - https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Simple_Globe.svg/1024px-Simple_Globe.svg.png

● Devices icon- https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Simple_Globe.svg/1024px-Simple_Globe.svg.png

● wifi icon - https://pixabay.com/static/uploads/photo/2016/01/03/11/32/wireless-signal-1119306_960_720.png

● cell tower - https://pixabay.com/static/uploads/photo/2012/04/13/00/23/tower-31235_960_720.png

Netflix Edge Engineering Open House Presentations - June 9, 2016

Technology

FINALEBusiness - Netflix

Living on the Edge (Service): Bundling Microservices to Optimize Consumption for Devices with Spring Cloud and Netflix OSS

Netflix (2)

Netflix Exemplar

Netflix Brief

Siemens PLM Software Dynamic Presentations using Solid Edge Content€¦ · · 2018-01-05Dynamic Presentations using Solid Edge Content Siemens PLM Software #SEU15. ... Save the

Hacking Netflix - Netflix APIs

Case NetFlix

NETFLIX TRAFFIC CHARACTERIZATIONcarey/CPSC641/slides/netflix/Netflix... · NetFlix –Video Delivery •HTML5 Player (transitioned away from Silverlight) •Requests to the Web interface

TDC2016SP - Living on the Edge (Service): Bundling Microservices to Optimize Consumption for Devices with Spring Cloud & Netflix OSS

Maintaining the Front Door to Netflix : The Netflix API

2013 MUM Presentation - MUM - MikroTik User Meetingmum.mikrotik.com/presentations/US13/steve.pdf · Netflix-Cons disabled=no new-packet-mark=all_other_traffic passthrough=no Conﬁg

Netflix Sued

Ppt Netflix

Netflix Pricing

Netflix & grill

NETFLIX Project

Leading Edge 2016 presentations

Adrian Cole of Netflix: Living On The Edge from Geek Summer Camp

Netflix - Τarpov