Presenters - QCon New York · Presenters Phillipa Avery Senior Software Engineer Project Technical...

Presenters

Phillipa AverySenior Software Engineer

Project Technical Lead and EngineerDownloads License Accounting

pavery@Netflix.com / @PhillipaAvery

Robert RetaSenior Software Engineer

Event Sourcing System ArchitectDownloads License Accounting

rreta@netflix.com / @rreta04

Phillipa

Robert

Overview

1. Why do we need a Downloads License Accounting Service?

2. Event Sourcing overview.3. Deep dive into the Event Sourcing Architecture.4. What’s it like working with the License Accounting

Service after release?

LICENSE

CREATE PLAYBACK CONTEXT

SESSION EVENTS(START, PAUSE,

RESUME, KEEPALIVE)

Streaming Playback Lifecycle

RELEASELICENSE

SESSION EVENT(STOP)

STOP

Streaming Playback Lifecycle

LICENSE

CREATE PLAYBACK CONTEXT

DOWNLOADED ENCODED MOVIE

Downloads Playback Lifecycle

~ 1 year

SESSION EVENTS(START, PAUSE, RESUME,

KEEPALIVE, STOP)

RENEWLICENSE?

X

RELEASELICENSE

Downloads Playback Lifecycle

Download Business Requirements

● Devices with downloads

Download Business Requirements

● Devices with downloads● Downloads per studio

Download Business Requirements

● Devices with downloads● Downloads per studio● Movie downloads or playback over year

~ 1 year

License Accounting Service Requirements

❏ Flexible

❏ Debuggable

❏ Reliable

❏ Scalable

Flexibility

Debuggability

Scalability

Reliability

License Accounting Service Requirements...

✓ Flexible: data model can be changed❏ RDBMS❏ Document Model

License Accounting Service Requirements...

✓ Flexible: data model can be changed❏ RDBMS❏ Document Model

✓ Debuggable❏ Event Sourcing

License Accounting Service Requirements...

✓ Flexible: data model can be changed❏ RDBMS❏ Document Model

✓ Debuggable❏ Event Sourcing

✓ Reliable❏ Fallbacks

License Accounting Service Requirements...

✓ Flexible: data model can be changed❏ RDBMS❏ Document Model

✓ Debuggable❏ Event Sourcing

✓ Reliable❏ Fallbacks

✓ Scalable

Event Sourcing

Domain Model

Make Purple

Command

Commands

Commands

Make Purple

Command

Command Handler

Added Red Event

Added Blue Event

Make Purple

Command

Command Handler

Commands

Events

Added Red Event

Added Blue Event

Event Handler

Events

Added Blue Event

Event Handler

Domain Model

Aggregate

Commands

Make Red Command

Command Handler

Command Handler

Commands

Make Red Command

Aggregate ServiceREST

Endpoint

Aggregate Repository

Event StoreQuery Query Query

Event Store

Row ID Events

Event Store

Row ID Events

Row ID Events

e1 e2 e3 e5 e6 e7e4

Event Store

Aggregate id 1 Aggregate id 2

Aggregate Repository

Query Event Store

Aggregate Repository

Event StoreQuery

select * from eventswhere rowId =

Aggregate Repository Event StoreResponse

e1 e2 e3 e5 e6 e7e4

Aggregate Repository

e1 e2 e3 e5 e6 e7e4

Aggregate id 1 Aggregate id 2

e1 e2 e5 e6 e3 e4 e7

Aggregate Repository

Aggregate id 2

Aggregate id 1

e1 e2 e5 e6

e3 e4 e7

Aggregate 1

Aggregate 2

Aggregate Repository

Aggregate 2

Aggregate 1

Aggregate id 2

Aggregate id 1

e1e2 e5 e6

e3 e4 e7

Aggregate Repository

Aggregate 2

Aggregate 1

Aggregate id 2

Aggregate id 1

e2e5 e6

e3 e7

Aggregate Repository

Aggregate 2

Aggregate 1

Aggregate id 2

Aggregate id 1

e5e6

e7

Aggregate Repository

Aggregate 2

Aggregate 1

Aggregate id 2

Aggregate id 1

e6

Aggregate Repository

Aggregate 2

Aggregate 1

Aggregate Repository

Command

Aggregate 1

Command Handler

Aggregate Repository

Command Handler

e8 e9

Aggregate Repository

e8 e9Aggregate 1

Aggregate Repository

Aggregate 1e8 e9

Aggregate Repository

Aggregate 1e9

Aggregate Repository

Aggregate 1

Aggregate Repository Event StoreUpdate

append events

where rowId =

e8 e9

Event Store

Aggregate id 1 Aggregate id 2

Row ID Events

e1 e2 e3 e5 e6 e7e4 e8 e9

Aggregate RepositoryQuery

Aggregate Service

Get all aggregates for a customer

Aggregate Repository

Aggregate Service Response

Aggregate 2Aggregate 1

Aggregate

DownloadedAggregate

License Aggregate

License Service

Acquire License

EndpointQuery

customerId: BayleytitleId: GlowS01E01

License Service

Query

customerId: BayleytitleId: GlowS01E01

Downloaded Service

isAllowed?

Query

customerId: BayleytitleId: GlowS01E01since: 6/27/2016

Downloaded Service

Downloaded Repository

getAggregates

Downloaded Service

Downloaded Repository

Response

DownloadedAggregate

customerId: BayleytitleId: GlowS01E01date: 2/15/2017

DownloadedAggregate

customerId: BayleytitleId: GlowS01E01date: 5/25/2017

License Service

Downloaded Service

Response

isAllowed?

aggregates.size() < yearly limit

License Service

Downloaded Service

Response

isAllowed?

Trueaggregates.size() < yearly limit

(2) < (3)

License Service

Update

License Repository

Create Aggregate

Create License Command

customerId: BayleytitleId: GlowS01E01date: 6/27/2017

License Aggregate

customerId:titleId:expires:released:

License Repository

Command HandlerCreate License

CommandcustomerId: BayleytitleId: GlowS01E01date: 6/27/2017

License Aggregate

customerId:titleId:expires:released:

License Repository

Command HandlerCreate License

CommandcustomerId: BayleytitleId: GlowS01E01date: 6/27/2017

License Aggregate

customerId:titleId:expires:released: License Created

Event

customerId: BayleytitleId: GlowS01E01date: 6/27/2017

License Repository

Event Handler

License Aggregate

customerId:titleId:expires:released:

License Created Event

customerId: BayleytitleId: GlowS01E01date: 6/27/2017

License Aggregate

customerId: BayleytitleId: GlowS01E01expires: 7/27/2017released: false

License Repository Event StoreSave

License Created Event

customerId: BayleytitleId: GlowS01E01date: 6/27/2017

License Service

License Repository

Create Aggregate

CreatedAcquire License

EndpointResponse

License Service License Repository

Response

isAllowed?

aggregates.filter(GlowS01E01) .size()

< yearly limit

Get all licenses

Table

Partition Keys Clustering Columns Columns

Event Table

rowIdtext

Event Timetimestamp

Event Datablob

Aggregate Idtext

Event MapperString

● A fast and efficient object graph serialization framework for Java

○ https://github.com/EsotericSoftware/kryo

● Able to preconfigure the library to your data models for extra compaction

● Custom serializers

AggregateIdtext

Event Datablob

rowIdtext

Event Timetimestamp

House of Cards LicenseCreatedMatt:LicenseAggregate 0

Buddy Thunderstruck LicenseCreated1

The Hardy Boys LicenseCreatedJeff:LicenseAggregate 2

AggregateIdtext

Event Datablob

rowIdtext

Event Timetimestamp

House of Cards LicenseCreatedMatt:LicenseAggregate 0

House of Cards LicenseReleased3

Buddy Thunderstruck LicenseCreated1

The Hardy Boys LicenseCreatedJeff:LicenseAggregate 2

AggregateIdtext

Event Datablob

rowIdtext

Event Timetimestamp

House of Cards LicenseCreatedMatt:LicenseAggregate 0

House of Cards LicenseReleased3

Buddy Thunderstruck LicenseCreated1

The Hardy Boys LicenseCreatedJeff:LicenseAggregate 2

AggregateIdtext

Event Datablob

rowIdtext

Event Timetimestamp

House of Cards LicenseCreatedMatt:LicenseAggregate 0

House of Cards LicenseReleased3

Buddy Thunderstruck LicenseCreated1

The Hardy Boys LicenseCreatedJeff:LicenseAggregate 2

Snapshotting

House of Cards LicenseCreatedMatt:LicenseAggregate 0

House of Cards LicenseReleased3

Buddy Thunderstruck LicenseCreated1

Master of None LicenseCreated0

Master of None LicenseReleased3

Bojack LicenseCreated1

Lucha Underground LicenseCreated0

Lucha Underground LicenseReleased3

Grace & Frankie LicenseCreated1

House of Cards LicenseCreated0

Snapshot Table

rowIdtext

versionint

Snapshot Datablob

Versionint

Event Datablob

rowIdtext

1 binaryDataMatt:LicenseAggregate

AggregateIdtext

Event Datablob

rowIdtext

Event Timetimestamp

House of Cards LicenseCreatedMatt:LicenseAggregate:0 0

House of Cards LicenseReleasedMatt:LicenseAggregate:1 3

Jessica Jones LicenseCreated1

The Hardy Boys LicenseCreatedJeff:LicenseAggregate:0 2

Working With Event Sourcing

Flexibility in Practice

● Changes to the Data Model are trivial!○ Device deactivation

New Device Deactivation Requirement

Deactivate Device

EndpointDevice Service

Device Repository

Device Aggregate

Device Deactivated

EventDeactivate

Device Command

New Device Deactivation Requirement...

Deactivate Device

Endpoint

Device ServiceDeactivate Device

Can Deactivate?

DeviceRepository

Response

DeviceAggregate

CustomerId: BayleyDeviceId: iPhoneDeactivations: 1

New Device Deactivation Requirement...

Deactivate Device

Endpoint

Device ServiceDeactivate Device

Can Deactivate

License ServiceRelease Licenses

Device Repository

Command Handler

Deactivate Device CommandcustomerId: BayleydeviceId: iPhone

Device Deactivated Event

customerId: BayleydeviceId: iPhone

DeviceAggregate

CustomerId: BayleyDeviceId: iPhoneDeactivations: 1

New Device Deactivation Requirement...

Debugging

● Possibly the biggest win

● Current tooling is rudimentary

{ "Event" : "LicenseAcquiredEvent", "Event Time" : "2017-06-25 05:23:00 PM", "aggregateId" : "aggregateId1", "committed" : true, "eventTimeStamp" : 1498411380903, "eventOrderNumber" : 0}

Debugging...

Acquire License Event

customerId: BayleydeviceId: iPhonedate: 01/05/2017

Release License Event

customerId: BayleydeviceId: iPhonedate: 01/05/2017

Renew License Event

customerId: BayleydeviceId: iPhonedate: 01/06/2017

License Aggregate

customerId:Bayleyexpires: 01/07/2017released: True

Reliability: Fallbacks

Edge API Layer

License Accounting Service Client

License Accounting

Service CassandraFallback Response

How Did it Scale?

Predictive Scaling

Scalability with Cassandra SSD (I2) Nodes

Scalability with Cassandra SSD Nodes...

● Could handle much higher loads● Storage use ramping up very quickly

Storage Optimizations

● TTLs● HDD (D2) clusters:

○ More storage○ Higher latency (up to 1 second)

Storage Optimizations...

● Partitioned approach○ Snapshot and subsequent events saved to i2 (SSD) cluster○ Archive events to D2 (HDD) cluster

Snapshot DataVersion 3

Event DataVersion 3

Event DataVersion <= 2

i2

D2

Storage Optimizations...Partitioned approach using CQRS (Command Query Responsibility Segregation)

● Write segregation○ Event handlers that determine partitioned usage

● Query segregation○ Query SSD or HDD (or both)○ Uninterrupted event stream

Event Data

i2

D2

Snapshot + Events

Event Handler

Key Event Sourcing Take-Aways

● Flexible: Adapting to change can be simple

Key Event Sourcing Take-Aways

● Flexible: Adapting to change can be simple

● Debugging: Debugging data / state transitions greatly simplified

Key Event Sourcing Take-Aways

● Flexible: Adapting to change can be simple

● Debugging: Debugging data / state transitions greatly simplified

● Reliable: Fallbacks provide service reliability

Key Event Sourcing Take-Aways

● Flexible: Adapting to change can be simple

● Debugging: Debugging data / state transitions greatly simplified

● Reliable: Fallbacks provide service reliability

● Scalable: Service scaled well, but good architectural solutions for data storage solutions should be considered.

Questions?

Phillipa AverySenior Software Engineer

Project Technical Lead and EngineerDownloads License Accounting

pippa@netflix.com / @PhillipaAvery

Robert RetaSenior Software Engineer

Event Sourcing System ArchitectDownloads License Accounting

rreta@netflix.com / @rreta04