Intuit CTOF 2011 - Netflix for Mobile in the Cloud

mobile first

Netflix for Mobile in the Cloud

Siddharth “Sid” Anand

May 2011

Intuit Proprietary & Confidential

mobile first

A Very Special Thanks to …

… the following folks for helping with the presentation: •  Ashish Khivesara – IPhone/Android UI Development

•  Daniel Jacobson – Director of API •  Shyan-Ming Perng – PS3/IPad/IPhone UI Development

•  Mike Cohen – API Development

•  Yury Izrailevsky – Director of Cloud Systems

… the entire company for pulling together to make Netflix for Mobile in the Cloud a reality!

mobile first

Our Business

No Longer Going Postal!

Intuit Proprietary & Confidential

mobile first

Our Business

In 1999 • Netflix launched DVD rentals-by-mail – Unlimited rentals for a flat monthly fee

– No due dates – No late fees

– No shipping or handling fees

– First attempt to target long-tail rental market • We can ship any of >100K titles, whereas Brick-and-Mortar are constrained by how much space they can rent - Only 1-2k rentals

Intuit Proprietary & Confidential

mobile first

Our Business

Fast Forward to 2008 •  Netflix forecasts the gradual end of the DVD and

starts switching to Movie Streaming •  Upside? –  We spend $500MM to $600MM annually on US

Postage for DVD mailing

–  Streaming a movie is a fraction of the cost of shipping a DVD

–  Easier to craft a business model for international expansion

Intuit Proprietary & Confidential

mobile first

Our Business

•  We have 22M+ subscribers in the US and Canada

•  May account for 20% of US peak downstream bandwidth usage

•  A top CDN user in the US •  Limelight, Akamai, & Level 3 deliver images and movies

•  Grew subscribers 2008-2010 by partnering with device makers

Intuit Proprietary & Confidential

mobile first

Our Business

•  A world of connected devices, some mobile

mobile first

Our Problem

Intuit Proprietary & Confidential

mobile first

Our Problem

• Circa late 2008, Netflix had a single data center –  Single-point-of-failure (a.k.a. SPOF)

–  Approaching limits on cooling, power, space, traffic capacity

• Alternatives – Build more data centers – Outsource the majority of our capacity planning and scale out • Allows us to focus on core competencies

Intuit Proprietary & Confidential

mobile first

Our Problem’s Solution!

• Winner : Outsource the majority of our capacity planning and scale out – Leverage a leading Infrastructure-as-a-service provider • Amazon Web Services

• Footnote : As it has taken us a while (i.e. ~2+ years) to realize our vision of running on the cloud, we needed an interim solution to handle growth – We did build a second data center along the way • We did outgrow it soon afterward!

mobile first

Cloud Migration Strategy

What to Migrate? How?

Intuit Proprietary & Confidential

mobile first

Cloud Migration Strategy

• Components – Applications and Software Infrastructure

– Data

• Migration Considerations – Security • PII and PCI DSS stays in our DC, rest can go to the cloud

– Web Scale – this goes!

Intuit Proprietary & Confidential

mobile first

Cloud Migration Strategy

Intuit Proprietary & Confidential

mobile first

Cloud Migration Strategy

Intuit Proprietary & Confidential

mobile first

Cloud Migration Strategy

Intuit Proprietary & Confidential

mobile first

Cloud Migration Strategy

Examples of Applications that can be moved • Web Scale – Personalized Video Discovery (e.g. browsing movie lists on the IPhone)

– Streaming Video Playback (e.g. watching a movie on the IPhone)

• Off-line – Video Encoding • Encode and store in final form in CDN

• ~40 encodes from a single mezzanine : bit rates, screen resolutions, codecs, etc…

•  Infrastructure – Log Capture & Analysis, Monitoring, Alerting

Intuit Proprietary & Confidential

mobile first

Cloud Migration Strategy

Examples of Data that can be moved • Video-centric data – Critics’ and Users’ reviews

– Video Metadata (e.g. director, actors, plot description, etc…)

• User-video-centric data – some of our largest data sets – Video Queue

– Watched History

– Video Ratings (i.e. a 5-star rating system)

– Video Playback Metadata (e.g. streaming bookmarks, activity logs)

mobile first

Cloud Migration

What Does the Cloud Offer?

Intuit Proprietary & Confidential

mobile first

“The cloud lets its users focus on delivering differentiating business value instead of wasting valuable resources on the undifferentiated heavy lifting that makes up most of IT infrastructure.”

- Werner Vogels, CTO, Amazon August 25, 2009, ‘All Things Distributed’

Cloud Migration

mobile first

AWS Services We Use Messaging Services

Intuit Proprietary & Confidential

mobile first

Messaging Services

Data Center Amazon Web Services WebLogic Messaging SQS (Message Queues)

SNS (Message Topics)

Intuit Proprietary & Confidential

mobile first

Messaging Services

•  Use-cases for Message Queues –  System Dependency Decoupling •  e.g. Decouple Cloud and DC applications -  When user adds a movie to the instant queue in the cloud, we send the update back to the DC for DC-resident applications -  Message consumers are decoupled from message producers

–  Fire-and-forget Write Scalability •  e.g. Application Logging -  Write to BI storage -  Consumers can process messages at their own rate

Intuit Proprietary & Confidential

mobile first

Messaging Services

•  SQS –  Infinitely Scalable but does not guarantee in-order message delivery

–  Never worry about overflowing the queue and crashing or becoming unresponsive! •  Often a problem with enterprise messaging systems

–  Messages are automatically dropped from queues after a maximum of 4 days

Intuit Proprietary & Confidential

mobile first

Messaging Services

•  SQS –  API for Queue Management •  CreateQueue •  ListQueues •  DeleteQueue

– API for Message Management •  SendMessage (up to 64K in size) •  ReceiveMessage (up to 10 messages in a batch) •  DeleteMessage (a.k.a. ACK Message) •  SetVisibilityTimeout – after which, a message becomes visible to other ReceivedMessage calls

mobile first

AWS Services We Use Persistence

Intuit Proprietary & Confidential

mobile first

Persistence

Data Center Amazon Web Services

Oracle SimpleDB

S3

Cassandra

Intuit Proprietary & Confidential

mobile first

Persistence

An ideal storage solution should have the following features: þ Hosted

þ Managed Distribution Model

þ Works in AWS þ AP from CAP

þ Handles a majority of use-cases accessing high-growth, high-traffic data þ Specifically, key access by customer id, movie id, or both

Intuit Proprietary & Confidential

mobile first

Persistence

• We picked SimpleDB and S3 – SimpleDB was targeted as the AP equivalent of our RDBMS

databases in our Data Center

– S3 was used for data sets where item or row data exceeded SimpleDB limits and could be looked up purely by a single key (i.e. does not require secondary indices and complex query semantics) • Video encodes

• Streaming device activity logs (i.e. CLOB, BLOB, etc…)

• Compressed (old) Rental History

Intuit Proprietary & Confidential

mobile first

Persistence

SimpleDB Hash Table Relational Databases

Domain Hash Table Table

Item Entry Row

Item Name Key Mandatory Primary Key

Attribute Part of the Entry Value

Column

Terminology

Intuit Proprietary & Confidential

mobile first

Persistence

Soccer Players

Key Value

ab12ocs12v9 First Name = Harold Last Name = Kewell Nickname = Wizard of Oz

Teams = Leeds United, Liverpool, Galatasaray

b24h3b3403b First Name = Pavel Last Name = Nedved Nickname = Czech Cannon

Teams = Lazio, Juventus

cc89c9dc892 First Name = Cristiano Last Name = Ronaldo

Teams = Sporting, Manchester United, Real Madrid

SimpleDB’s salient characteristics

•  SimpleDB offers a range of consistency options

•  SimpleDB domains are sparse and schema-less

•  The Key and all Attributes are indexed

•  Each item must have a unique Key

•  An item contains a set of Attributes

•  Each Attribute has a name

•  Each Attribute has a set of values

•  All data is stored as UTF-8 character strings (i.e. no support for types such as numbers or dates)

Intuit Proprietary & Confidential

mobile first

Persistence What does the API look like? • Manage Domains – CreateDomain – DeleteDomain – ListDomains – DomainMetaData

• Access Data – Retrieving Data • GetAttributes – returns a single item • Select – returns multiple items using SQL syntax

– Writing Data • PutAttributes – put single item • BatchPutAttributes – put multiple items

– Removing Data • DeleteAttributes – delete single item • BatchDeleteAttributes – delete multiple items

mobile first

AWS Services We Use Compute

Intuit Proprietary & Confidential

mobile first

Compute

Data Center Amazon Web Services

Hadoop EMR & S3

Servers EC2 – heavy use of reserved instances

Citrix Load Balancer •  Elastic Load Balancer (ELB) •  Cloud Watch •  Auto Scale Group (ASG)

mobile first

Filling the Gaps Netflix’s Undifferentiated Heavy Lifting

Intuit Proprietary & Confidential

mobile first

Filling the Gaps

AWS IAAS • Persistence

•  SimpleDB, RDS, S3, EBS

• Compute • EC2, EMR, ASG, ELB

• Monitoring • CloudWatch

? Netflix Applications

AWS provides various IAAS offerings, but applications need more!

Intuit Proprietary & Confidential

mobile first

Filling the Gaps

AWS IAAS • Persistence •  SimpleDB, RDS, S3, EBS

• Load Handling • EC2, EMR, ASG, ELB

• Monitoring • CloudWatch

Netflix Platform • Platform.jar • Middle-tier Load

Balancing • Discovery • Encryption Services • Caching • Configuration • GeoLocation

Netflix Applications

AWS provides various IAAS offerings, but applications need more!

Hence the need for Netflix’s infrastructure team to bridge the gap!

Intuit Proprietary & Confidential

mobile first

?

Discovery & MTLB

S3 SimpleDB

SQS

EC2 EC2

AWS IAAS

EC2 EC2 EC2 EC2

Netflix mid-tier services (e.g. Q or Merch services)

AWS ELB Netflix edge services (e.g. API)

Q

API

Merch

Intuit Proprietary & Confidential

mobile first

Discovery & MTLB

S3 SimpleDB

SQS

EC2 EC2

AWS IAAS

EC2 EC2 EC2 EC2

Netflix mid-tier services (e.g. Q or Merch services) w/

platform.jar

AWS ELB

Discovery Middle Tier Load Balancer

Netflix Platform

Netflix edge services (e.g. API)

Q Merch

API

mobile first

Building Mobile Apps Apple IPhone

Intuit Proprietary & Confidential

mobile first

Mobile App Development Strategy

• We develop mobile web pages that run in Web Kit where available. Examples include: •  PS3

•  IPhone/IPad/etc…

• Enables • A/B Testing

•  Fast Deployments

•  Leveraging code across devices (e.g. IPhone & Android)

Intuit Proprietary & Confidential

mobile first

IPhone

• When you click on the app, you see the splash screen

• It will disappear once we start loading the Home Page

Intuit Proprietary & Confidential

mobile first

IPhone • Home Page Loading: Fetch

1.  Time service

2.  User resource

3.  Rental History for resume feature

4.  Fetch up to 12 Lists of Movies

1.  Get 3 movies for the top-most list (e.g. Children & Family)

2.  After it loads, fetch 3 movies for remaining 11 lists in parallel – slows scrolling

5.  Bulk rating request for all titles to fetch the star bars

•  Average 11 API calls on Home Page

Intuit Proprietary & Confidential

mobile first

IPhone • User Interaction Reasoning (in red)

•  Rental History for resume feature

•  Promotes fastest movie launch

•  Fetch up to 12 Lists of Movies

•  Get 3 movies for the top-most list (e.g. Children & Family)

•  After it loads, fetch 3 movies for remaining 11 lists in parallel

•  Remaining 11 may slow scrolling

•  Bulk rating request for all titles to fetch the star bars

•  Absence is least obvious to user

mobile first

Tips & Tricks Apple IPhone

Intuit Proprietary & Confidential

mobile first

Tips & Tricks - Prefetching

Intuit Proprietary & Confidential

mobile first

Tips & Tricks – Prefetching • On any list (e.g. Home) page, we prefetch

•  Box Shot Image

•  Title

•  Ratings (i.e. star bar)

•  Year, MPAA rating, Duration

•  Synopsis

• On the Movie Display Page, we load/reload

•  “In Q” button status

•  Play or Resume button status

• Cast/Director

• Episode List (for TV Shows) or Simliars (for Movies)

Intuit Proprietary & Confidential

mobile first

Tips & Tricks – Memory Savings

• To save memory, when we navigate away from the MDP, we unload the following: • Similars (for Movies)

• Episodes (for TV Shows)

• We keep the Cast & Director

• On future visit to this MDP, we will reload the following:

• Anything not loaded

• Button status for both buttons

Intuit Proprietary & Confidential

mobile first

Tips & Tricks – Bandwidth Savings

•  Server responses are GZIP compressed –  Processor speed ample enough to handle the decompression

–  Less power hungry than wireless 3G transfers of uncompressed data

–  Rules of thumb, keep payload 20-100KB per call

Intuit Proprietary & Confidential

mobile first

Tips & Tricks – Bandwidth Savings

•  Revising Netflix REST Model – Too chatty

– Too many round-trips

–  Or it returns data that is not needed • A lot of processing is needed to skip extraneous data

•  Moving to a more structured query language (e.g. SQL-like)

• This will reduce memory, bandwidth use, and round-trips • Great blog entry on the topic from Netflix’s Director of API

@daniel_jacobson –  http://techblog.netflix.com/2011/02/redesigning-netflix-

api.html

Intuit Proprietary & Confidential

mobile first

Tips & Tricks – Image Sprites

•  Image Sprites are used to reduce memory and bandwidth by giving the user the impression of viewing different images, when in fact there is only one –  Ratings Star Bars –  Footer (i.e. Home, Genres, Search, & Q)

Intuit Proprietary & Confidential

mobile first

Tips & Tricks – Image Sprites

With Some Markup

<span title="1 of 5" class="rating r1"><img width="0" height="1" src="sprite.gif" alt=""/>1 out of 5</span>

<span title="2 of 5" class="rating r2"><img width="0" height="1" src="sprite.gif" alt=""/>2 out of 5</span>

…..

And Some CSS

.rating {

position: relative;

height: 1.6em;

…..

Intuit Proprietary & Confidential

mobile first

Tips & Tricks – Security

•  All traffic is over secure channels – https – Deny Man-in-the-Middle attacks

•  All requests are signed with a time –  Use Netflix Time Service to avoid clock-

skew

–  Avoids replay attacks

mobile first

Netflix IPhone App Demo

Intuit Proprietary & Confidential

mobile first

Netflix IPhone App Demo

•  Link can be found at the location below: –  http://blog.netflix.com/2010/08/netflix-now-available-on-your-iphone.html

mobile first

Questions?