#NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies &...

Josh Evans - Director of Operations Engineering

March, 2016

#NetflixEverywhereGlobal Architecture

December 24th, 2012

Disappointment

Outrage

Withdrawal

December 24th, 2012

Failure is inevitable

Never fail the same way twice

Failure-Driven Architecture

#NetflixEverywhere

Failure-Driven Architecture

• Introductions

• Failure-Driven Architecture

• Taking It Global

#NetflixEverywhere

Our Talk Today

• Introductions• Failure-Driven Architecture

#NetflixEverywhere

Our Talk Today

1999 – 2009• Ecommerce (DVD Streaming)

2009 – 2013• Playback Services (Activate, Manifests, DRM)

2013 - present• Operations Engineering

– CD, RTA, Chaos, Performance

Josh Evans – Director of Operations Engineering

jevans@netflix.com

Bringing movies & TV shows from all over the world to people all over the world

• Streaming, on demand, subscription• Global & regional licensing• Hollywood, independent, international

• Striving for global ubiquity

• Jan – Windows

• May – Roku

• Oct – LG, Samsung Blu-ray

• Oct – Apple Mac

• Nov – XBox 360

• Jun – LG DTV

• Nov –Sony PS3 (disc)

• Nov – Sony Bravia– DTV & Blu-ray

Device Ubiquity

• May – Android

• Nov – First e-readers– Kindle Fire, Nook

Device Ubiquity2010

• Mar – Nintendo Wii (disc)

• Apr – Apple iPad

• Aug – Apple iPhone

• Sep – Apple TV

• Oct – Sony PS3 (no disc)

• Oct – Nintendo Wii (no disc)

• Nov – Windows Phone 7

2010 - Canada

2011 - Latin America

2012 - UK, Ireland, Nordics

2013 – Netherlands

2014 - Austria, Belgium, France, Germany, Luxembourg, Switzerland

2015 - Australia, New Zealand, Japan, Spain, Italy, Portugal

Geographic Ubiquity

• English• Spanish (Latin American)• Portuguese (Brazilian)• Dutch• French• German• Japanese• Spanish (Castilian)• Italian• Portuguese (European)

Language Ubiquity - Subs, Dubs, UI

75,000,000

• Introductions

• Failure-Driven Architecture• Taking It Global

#NetflixEverywhere

Our Talk Today

August 2008

• No automation, virtualization, standardization• Manual, error prone, slow• Big iron & monoliths

DC22009

Undifferentiated Heavy Lifting

US-East-1

Amazon Web Services2010

• Scale & elasticity• Virtual, programmable• Global footprint

• Micro-services

• Database

• Cache

• Traffic

Architectural Pillars

• Micro-services

• Database

• Cache

• Traffic

FITFault-Injection

Test Framework

Micro-service Failure

• Micro-services

• Database

• Cache

• Traffic

NoSQL but…

• Not web scale

• Throttling

Modest scale• 100s of play starts / second• 10,000s of requests / second• 10s of billions of records

SimpleDB

• Micro-services

• Database

• Cache

• Traffic

Ephemeral Volatile memCache (EVCache)

Clustered memcached optimized for Netflix use cases

EVCache Server

Memcached

Prana (Sidecar)

Monitoring & Other Processes

Eureka

Client Application

Client Library

EVCache Client

Shards, consistent hashing

TTLs & LRU

EVCache Architecture

Zone A

Client Application

Client Library

EVCache Client

Zone B

Client Application

Client Library

EVCache Client

Zone C

Client Application

Client Library

EVCache Client

. . .. . .. . .

Zone A Zone B Zone C

. . .. . .. . .

Writes

Client Application

Client Library

EVCache Client

Client Application

Client Library

EVCache Client

Client Application

Client Library

EVCache Client

1. Read from cache2. On cache miss call service3. Service calls DB & responds4. Service updates cache

Client Application

Client Library

EVCache Client Service Client

S S S S. . .

DB DB DB DB. . .

Fronting Micro-services

Linear Scaling

• 30 million requests/sec

• 2 trillion requests per day globally

• Hundreds of billions of objects

• Tens of thousands of memcached instances

• Milliseconds of latency per request

US-East-1

Canada

International Expansion2011

Latin America

US-East-1 EU-West-1

Cloud Islands2012

• Micro-services

• Database

• Cache

• Traffic

US-East-1 EU-West-1

UK/IE, Nordics, Netherlands

Latin America

DNS Geo Mapping

Canada

• Micro-services

• Database!

• Caching

• Traffic

Why Cassandra?• NoSQL at scale

• Open source

• Multi-region

• Multi-directional

• CAP Choices

– Availability

– Partition tolerance

– Eventual consistency*

Scalable, Durable, Global

Single Region, Multiple AZs

1. Client writes to any node

2. Coordinator replicates to nodes

3. Nodes ack to coordinator

4. Coordinator acks to client

5. Write to commit log

Zone A

Zone B

Zone C

Zone B

Zone C

Client

Zone A

• Hinted handoff to offline nodes

Local Quorum(Typical)

Not quite fast enough

December 24th, 2012

US-East-1US-West-2 EU-West-1

IsthmusSpring 2013

Survive a regional ELB outage

AZ1 AZ2 AZ3

US-EAST-1 ELBs

Data Data Data

Geo-locatedstate/province

AZ1 AZ2 AZ3

US-WEST-2 ELBs

Data Data Data

Americas Internet Traffic

Eastern NA +

LatAm Traffic

• Zuul routes locally or remotely

• Eureka - multi-region aware

Isthmus

Active-Active2013 - 2014

Survive a large-scale regional service outage

Active-Active Data Replication

Region BRegion A

Zone A

Zone B

Zone C

Zone B

Zone C

Zone A

Zone B

Zone C

Client Client

Zone AZone B

Multi-Region Writes

Bi-directional

Nightly compare & repair

Local Quorum(Typical)

EVCache Replication

Repl Writer

Application

Client

EVCache Replication

Repl Writer

1. Set or delete

2. send metadata

3. poll msg

6. set or delete

Application

Client

7. read

EVCache Cross-Region Replication

Region BRegion A

Active-Active Traffic Management

ELB US-West-2 ELB US-East-1 ELB EU-West-1

DNS api-global.netflix.com UltraDNS

Route53

DNS api-global.netflix.com • Remove state from geo bucket

api-global.netflix.comDNS • Remove state from geo bucket• Add state to geo bucket • Log event

• For each end point

api-global.netflix.com

api-global.us-west-2.prodaa.neflix.com

api-global.us-east-1.prodaa.neflix.com

api-global.eu-west-1.prodaa.neflix.com

ELB ELB ELB

Active-Active Failover

• Introductions

• Failure-Driven Architecture

#NetflixEverywhere

Our Talk Today

January 6th, 2016

Geographic Ubiquity

Before Global• English• Spanish (Latin American)• Portuguese (Brazilian)• Dutch• French• German• Japanese• Spanish (Castilian)• Italian• Portuguese (European)

Global• Chinese

• Korean

• Arabic

Language Ubiquity

March 18th, 2016

Daredevil Season 2

All episodes, all devices, all countries

Simultaneously

Content Ubiquity

Ubiquitous, Resilient Architecture

Reliably and efficiently serve any customer from any region

Netflix Global 2015

Ubiquitous Data

EVCache Replication

Repl Writer

Application

Client

• High latency

• Read once

• Low latency

• Multiple readers

• > 1M replications/sec

US Ring US Ring

EU Ring

EU-West-1US-East-1

1. Extend US ring to EU region & run repairs

Client2. Dual Write

3. Forklift

EU-West-1US-West-2

Global Ring

US-East-1

Global Ring

Ubiquitous Traffic Management

us-east-1-na

• East US

• East CA

• MX

us-west-2

• APAC

• West US

• West CA

eu-west-1

• Europe

• Mid East

• Africa

us-east-1-sa

• LatAm

• Not MX

Virtual DNS Regions

• Fixed virtual modules

• Origin tier

• Standardized names

api-global.us-east-1-sa.prodaa.neflix.com

api-global.us-east-1-na.prodaa.neflix.com

api-global.eu-west-1.prodaa.netflix.com

api-global.us-west-2.origin.prodaa.neflix.com

api-global.us-east-1.origin.prodaa.neflix.com

api-global.eu-west-1.origin.prodaa.neflix.com

ELB ELB ELB

Virtual

Origin

DNS Tiers

ELB ELB ELB

Virtual

Origin

Split Failover

ELB ELB ELB

Virtual

Origin

Split Failover

ELB ELB ELB

Virtual

Origin

Cascading Failover

ELB ELB ELB

Virtual

Origin

Cascading Failover

ELB ELB ELB

Virtual

Origin

Cascading Failover

ELB ELB ELB

Virtual

Origin

Cascading Failover

Multi-region Failover

January 6th, 2016

“Going global is just like having a baby.”- Reed Hastings, Netflix CEO

What’s next?• Global latency• Edge computing• ML-based monitoring• Self-healing systems• Capacity utilization• Fast, autonomous traffic• Integrate DB & caching

#NetflixEverywhere

Takeaways

Christmas Eve2012

Know your resiliency patterns

Pattern Properties

DC SPoF, infrastructure heavy lifting

Cloud (one region) Multiple DCs, one control plane

Islands Regional containment

Isthmus Regional ELB bypass

Active-active Regional failover

Global Ubiquity, resiliency, efficiency

Invest in architectural pillars

• Micro-services

• Database

• Caching

• Traffic

#NetflixEverywhere

Think globally, act locally

netflix.github.io

Netflix Tech Blog

techblog.netflix.com

Josh Evansjevans@netflix.com@ops_engineering

#NetflixEverywhereGlobal Architecture

#NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies &...

Documents

THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · 2018-01-11 · FAILURE AT NETFLIX VELOCITY THE ART OF CHAOS ENGINEERING Dave Hahn — Senior SRE, Netflix dhahn@netflix.com

Netflix RaaS: Reliability as a Service - USENIX RaaS: Reliability as a Service Coburn Watson. ... Build and Maintain own CDN ... cwatson@netflix.com, @coburnw Netflix Open Source Portal

Netflix Instance Performance Analysis Requirements Instance Performance Analysis Requirements Brendan Gregg Senior Performance Architect Performance Engineering Team bgregg@netflix.com

CEO and co-founder, Gremlin Kolton Andrus · Gremlin, Inc. (Formerly Netflix) kolton@gremlininc.com Chris Sanden Casey Rosenthal Ali Basiri Lorin Hochstein Netflix, Inc. csanden @netflix.com

Communication Skills CHAPTER 5 to Elicit Physical Activity Behavior Change: How to Talk to the Client · The client can’t The client The client feels The client The client The client

Client Registration Client Login Client ID & Password

· .Founding Netflix.com to offer online movie rentals. .0ffering selling and subscription services. Events .Launching a personalized movie recommendation service (CineMatch) .lssuing

wednesday general evans faultinjection 19 - Meet us in ... · EmbracingFailure FaultInjec,on&and&Service&Resilience& ... Netflix jevans@netflix.com, @josh_evans_nflx . Title: wednesday_general_evans_faultinjection_19.pptx

The Performance People Performance Management with Free and Bundled Tools Adrian Cockcroft Netflix Inc. acockcroft@netflix.com (Co-authored with Mario

Netflix.com A Strategic Management Case Study Tony Gauvin

Variational Autoencoders for Collaborative Filtering · Variational Autoencoders for Collaborative Filtering Dawen Liang Netflix Los Gatos, CA dliang@netflix.com Rahul G. Krishnan

[HTML5DevConf2014] Scaling AB Testing on Netflix.com with Node.js

#1123 12/20 - Production Weekly · NETFLIX 5800 Sunset Boulevard, Hollywood, CA 90028 310-734-2900 lahiring@netflix.com A family comedy with a wide-eyed, ill-guided 18-year-old autistic

[NodeConf.eu 2014] Scaling AB Testing on Netflix.com with Node.js

Latest Threats Against Mobile Devices Dave Jevans Founder, Chairman and CTO

Client Information Client Information

Minority Male Success Initiative€¦ · Minority Male Success Initiative application, please contact John J. Evans at (919) 807-7212 and/or via e-mail at JEvans@nccommunitycolleges.edu

IXP Member List API - RIPE Network Coordination Centre · IXP Member List API RIPE69, London elisa@netflix.com nick@inex.ie

Netflix, Inc.: Rule 14a-8 no-action letter - SEC · Netflix, Inc. rthompson@netflix.com . Re: Netflix, Inc. Incoming letter dated February 3, 2017 . Dear Mr. Thompson: This is in

Climate Change and Biodiversity in Ecosystems of Eastern ...faculty.washington.edu/wgold/BIS 358/111306.pdf · Ecosystems of Eastern Washington Jim Evans jevans@tnc.org “To preserve