[JSDC 2016] Codex: Conditional Modules Strike Back

Alex Liu

A N E T F L I X E N G I N E E R I N G O R I G I N A L

@[email protected]

codex

conditionaL ModuleS Strike bacK

#netflixeverywhere

contentsI. The Problem

II. Codex: Conditional Bundling

III. Scaling for Netflix

IV. Looking to the Future

i. the Problem

It's all about building JS bundles.

// bundle.js var a = require('a'); var b = require('b');

bundle.js

var bundles = [ 'bundle.js' ];

bundle.js

home.jsprofile.jsfeed.js album.js signup.js

home.js profile.js feed.js

signup.js account.js setting.js

album.js photo.js login.js

var bundles = [ 'home.js', 'profile.js', 'feed.js', 'signup.js' 'account.js', 'settings.js', 'album.js', 'photo.js', 'login.js' ];


/home /profile /feed


/home /profile /feed

home.js profile.js feed.js

html5shiv es5-shim

var bundles = [ 'home.js', 'homeIE.js', 'profile', 'profileIE', 'feed.js', 'feedIE.js' 'signup.js', 'signupIE.js', ... ];


/home


/home

home.js

NO


/home

home.js homeIE.js

YESNO

Netflix AB testing!

AB Test w/ multiple cells

Cells Control (Cell 1) Cell 2 Cell 3

Movie Cover Art

AB Test w/ multiple cells

Cells Control (Cell 1) Cell 2 Cell 3

Movie Cover Art

14% 6%

Netflix AB testing!

home.js

home.js

newSearch.jsoldSearch.js

home.js

oldSearch.js

home.js

newSearch.js

home.js


home.js


jQuery React~80KB ~120KB

larger bundle size: file sizes time to download memory usage time to interactive (TTI)

old school Lego bricks were generic

new Lego is about specialization

hard to reuse specialized bricks

home.js


// starting to look like a // lot of bundles... var bundles = [ 'homeNewSearch.js', 'homeNewSearchIE.js', 'homeOldSearch.js', 'homeOldSearchIE.js', ... ];

// starting to look like a // lot of bundles... var bundles = [ 'homeNewSearch.js', 'homeNewSearchIE.js', 'homeOldSearch.js', 'homeOldSearchIE.js', ... ];

4x variations already!

Netflix runs hundreds of AB tests

Netflix runs hundreds of AB tests

but we personalize on many other dimensions too

|S1| |S2| |Sn| = |S1 S2 Sn|

|S1| |S2| |Sn| = |S1 S2 Sn|

3100 = 5.1537752e47

40,000,000,000 bricks to reach the


7,600,000,000,000,000 bricks to reach


7,600,000,000,000,000 bricks to reach

Enough bricks to reach 6.7812832e32 times.

thats a #$*%^ ton of bundles!

https://xkcd.com/303/

Website's full bundle is 10MB+

how do we deal with conditional modules?

ii. codex conditional Bundling

:

what if we generate on-demand?

what if we generate on-demand?

1. identify the UI variation 2. generate the bundle

how do we identify the UI variation?

AB Tests

truthsnoun, plural [trooth z, trooths]

a bucket of boolean flags used to build a personalized Netflix experience

{ "webfonts": false, "instantSearch": true, "socialFeatures": false, "motionBanner": true, "html5video": true, "customScrollbar": true }

{ "webfonts": false, "instantSearch": true, "socialFeatures": false, "motionBanner": true, "html5video": true, "customScrollbar": true }

inputs and outputs are NOT 1:1

how do we generate the bundle?

home.js


// home.js if (truths.isNewSearch === true) { require('./newSearch'); } else { require('./oldSearch'); }

home.js


home.js


conditioncondition

conditioncondition

isNewSearch!isNewSearch

// home.js if (truths.isNewSearch === true) { require('./newSearch'); } else { require('./oldSearch'); }

home.js


isNewSearch!isNewSearch

home.js

newSearch.js

newEntryPoint.js

oldSearch.js

!isNewSearch isNewSearch

git

codex(node.js module)

git

codexartifact

(node.js module)

artifact

{ "home.js": { "deps": [ "dep1.js", "dep2.js", "dep3.js", ], "conditionalDeps": { "newSearch.js": { "name": "isNewSearch", "value": true },

"dep3.js", ], "conditionalDeps": { "newSearch.js": { "name": "isNewSearch", "value": true }, "oldSearch.js": { "name": "isNewSearch", "value": false } } } }

it's a conditional map!

web/v1 web/v2 web/v3 web/v4

artifacttruths

http://codex.nflxext.com/web/v1/83af

codex

? i got this!


codex

http://codex.nflxext.com/web/v1/83af http://codex.nflxext.com/{team}/{version}/{truths}

codexweb/v1


codex { 83: newSearchTest, af: isChrome }

web/v1


home.js

oldSearch.js

home.js


home.js

home.js


response times

codex


codex

here you go


codex

cached! here you go


Recap Build Time: build conditional graph (artifact)

Run Time: apply truths to artifact

Conditional bundling is transparent, universal, configuration free!

iii. Scaling For Netflix

web/v1 web/v2 tv/v5 tv/v7

Storage Metadata

Amazon S3Amazon DynamoDB

Storage Metadata

Build Time: Codex Artifact Management

web/v1


web/v1web/v1


saved!

SAVED!

web/v1web/v1


saved!

activate web/v1 web/v1

activated!


Run Time: Codex Bundler

web/v1

here are the active build ids


here are the artifacts

web/v1

here are the active build ids


prod prod-new canary

16GB ought to be enough for us!__

400+ artifacts!

400+ artifacts!

and we ran out of memory

32GB ought to be enough for us!__

800+ artifacts!

800+ artifacts!

and we ran out of memory

64GB ought to be enough for us?__

1600+ artifacts!

1600+ artifacts!

and we ran out of memory. again.

1600+ artifacts!Our teams will use as

much as we give them.

and we ran out of memory. again.

Q: What's cheap, plentiful, and fast

enough?

Q: What's cheap, plentiful, and fast

enough?A: Disk.

codex

LevelDB

100% CPU usage.

what's the problem?

v8::internal::Runtime_ParseJson

codex

LevelDB

codex

LevelDB

JSON.parse

JSON.parse is slow. And blocks the CPU!

codex

LevelDB

JSON.parse

LRU Cache: Saving the CPU

Breaking change to conditional graph

traversal algorithm!


old algorithm?

new algorithm?

old algorithm?

new algorithm?

http://codex.nflxext.com/1.0.0/web/v1/83afhttp://codex.nflxext.com/2.0.0/web/v1/83af

1.0.0

2.0.0

zuul

zuul 1.0.0

2.0.0

Good for now.

Good for now.

Continue to look for engineering wins.

What about operational resiliency?

eu-west-1

us-west-2

us-east-1

eu-west-1

us-west-2

???

???

???

???

codex

???

Recap Management plane necessary at scale

Performance is critical (TTI)

Redundancy across 3 AWS zones

Resilient against CDN failure

iv. Looking To The Future

why not {bundler}?

how do we support tree

shaking?

don't be afraid to challenge common

convention.

don't make assumptions about the upper limits.

don't optimize before you understand the system.

use the scientific method:

1. gather data 2. formulate hypothesis 3. test hypothesis 4. repeat

engineer for fault tolerance

Netflix scale is challenging.

https://www.flickr.com/clement127/ https://www.flickr.com/jose_antonio_hidalgo_jimenez/ https://www.flickr.com/reiterlied/

Lego Photo Credits

Image Credits

Image Credits

Artist: alecive (Alessandro Roncone) Iconset Homepage: https://github.com/alecive/FlatWoken

Alex Liu [email protected] @stinkydofu

fin

Internet

[JSDC 2016] Codex: Conditional Modules Strike Back