Managing Data in Microservices - QCon New York · • Most applications deployed multiple times per day ... MySQL, etc.}, owned and operated by the service team ... o Analytic systems

Managing Data in Microservices

Randy Shoup @randyshoup

linkedin.com/in/randyshoup

Background• VP Engineering at Stitch Fix

o Combining “Art and Science” to revolutionize apparel retail

• Consulting “CTO as a service”o Helping companies scale engineering organizations and technology

• Director of Engineering for Google App Engineo World’s largest Platform-as-a-Service

• Chief Engineer / Distinguished Architect at eBayo Multiple generations of eBay’s infrastructure

@randyshoup linkedin.com/in/randyshoup

Stitch Fix


Stitch Fix


Stitch Fix


Stitch Fix


Combining Art and [Data] Science

• 1:1 Ratio of Data Science to Engineeringo Almost 100 software engineerso Almost 100 data scientists and algorithm developerso Unique in our industry

• Apply intelligence to *every* part of the businesso Buyingo Inventory managemento Logistics optimizationo Styling recommendationso Demand prediction

• Humans and machines augmenting each other


Styling at Stitch Fix

Personal styling

Inventory


PersonalizedRecommendations

InventoryAlgorithmic

recommendations

Machine learning


Expert HumanCuration

Human curation

Algorithmic recommendations


How do we work, and why does it work?

Modern SoftwareDevelopment

Practices

CultureTechnology

Organization


TDD and Continuous

Delivery

DevOpsMicroservices

Small Teams


TDD and Continuous

Delivery

DevOpsMicroservices

Small Teams

Small“Service” Teams

• Teams Aligned to Business Domainso Clear, well-defined area of responsibilityo Single service or set of related services

• Cross-functional Teamso All skill sets needed to do the job

• Depend on other teams for supporting services, libraries, and tools



TDD and Continuous

Delivery

DevOpsMicroservices

Small Teams

Test-Driven Development

• Tests help you go fastero Tests “have your back”o Development velocity

• Tests make better codeo Confidence to break thingso Courage to refactor mercilessly

• Tests make better systemso Catch bugs earlier, fail faster


“Do you have time to do it twice?”

“We don’t have time to do it right!”

Test-Driven Development

• Do it right (enough) the first timeo The more constrained you are on time and resources, the more important

it is to build solid featureso Build one great thing instead of two half-finished things

• Right ≠ Perfect (80 / 20 Rule)

• è Basically no bug tracking system (!)o Bugs are fixed as they come upo Backlog contains features we want to buildo Backlog contains technical debt we want to repay


ContinuousDelivery

• Most applications deployed multiple times per day

• More solid systemso Release smaller units of worko Smaller changes to roll back or roll forwardo Faster to repair, easier to understand, simpler to diagnose

• Rapid experimentationo Small experiments and rapid iteration are cheap



TDD and Continuous

Delivery

DevOpsMicroservices

Small Teams

DevOps• End-to-end Ownership

o Team owns service from design to deployment to retirement

• Responsible for o Featureso Qualityo Performanceo Operationso Maintenance


You Build It, You Run It.-- Werner Vogels


TDD and Continuous

Delivery

DevOpsMicroservices

Small Teams

Evolution to Microservices

• eBay • 5th generation today• Monolithic Perl à Monolithic C++ à Java à microservices

• Twitter• 3rd generation today• Monolithic Rails à JS / Rails / Scala à microservices

• Amazon• Nth generation today• Monolithic Perl / C++ à Java / Scala à microservices


First Law of Distributed Object Design:

Don’t distribute your objects!-- Martin Fowler

If you don’t end up regretting your early technology decisions, you probably over-engineered.

-- me

Microservices

• Single-purpose• Simple, well-defined interface• Modular and independent

A

C D E

B

Microservices are nothing more than SOA done properly.

-- me

Microservices

• Single-purpose• Simple, well-defined interface• Modular and independent• Isolated persistence (!)

A

C D E

B

MicroservicePersistence

• Approach 1: Operate your own data storeo Store to your own instance(s) of {Postgres, MySQL, etc.}, owned and

operated by the service team

• Approach 2: Use a persistence serviceo Store to your own schema in {Dynamo, RDS, Spanner, etc.}, operated as a

service by another team or by a third-party providero Isolated from all other users of the service

• è Only external access to data store is through published service interface


Events asFirst-Class Construct

• “A significant change in state”o Statement that some interesting thing occurredo 0 | 1 | N consumers subscribe to the event, typically asynchronously

• Traditional 3-tier systemo Presentation è interface / interactiono Application è stateless business logico Persistence è database

• Fourth fundamental building blocko State changes è events


Events asFirst-Class Construct

• Events represent how the real world workso Financeo Software development processo Any “workflow”


Microservicesand Events

• Events are a first-class part of a service interface

• A service interface includeso Synchronous request-response (REST, gRPC, etc)o Events the service produceso Events the service consumeso Bulk reads and writes (ETL)

• The interface includes any mechanism for getting data in or out of the service (!)


Extracting Microservices

• Problem: Monolithic shared DB

• Clients• Shipments• Items• Styles, SKUs• Warehouses• etc.

stitchfix.com Styling app Warehouse app Merch app

CS app Logistics app Payments service Profile service



• Decouple applications / services from shared DB

• Clients• Shipments• Items• Styles, SKUs• Warehouses• etc.

stitchfix.com Styling app Warehouse app Merch app

CS app Logistics app Payments service Profile service



• Decouple applications / services from shared DB

Styling app Warehouse app

core_item

core_sku

core_client



• Step 1: Create a service


core_item

core_sku

core_client

client-service



• Step 2: Applications use the service


core_item

core_sku

core_client

client-service



• Step 3: Move data to private database


core_item

core_sku

client-service

core_client



• Step 4: Rinse and Repeat


core_sku

client-service

core_client

item-service

core_item





client-service

core_client

item-service

core_item

style-service

core_sku





client-service

core_client

item-service

core_item

style-service

core_sku


Microservice Techniques:Shared Data

• Problemo Monolithic database makes it easy to leverage shared datao Where does shared data go in a microservices world?



• Principle: Single System of Recordo Every piece of data is owned by a single serviceo That service is the canonical system of record for that data

• Every other copy is a read-only, non-authoritative cache


customer-servicestyling-service

customer-search

billing-service


• Approach 1: Synchronous Lookupo Customer service owns customer datao Fulfillment service calls customer service in real time

fulfillment-service

customer-service



• Approach 2: Async event + local cacheo Customer service owns customer datao Customer service sends address-updated event when customer address

changeso Fulfillment service consumes event, caches current customer address

fulfillment-servicecustomer-service



• Approach 3: Shared metadata library o Read-only metadata, basically immutableo E.g., size schemas, colors, fabrics, US States, etc.

receiving-serviceitem-service

style-service


Microservice Techniques:Joins

• Problemo Monolithic database makes joins very easyo Splitting the data into separate services makes joins very hard



• Approach 1: Join in Client Applicationo Get a single customer from customer-serviceo Query matching orders for that customer from order-service

• Best for single A, multiple Bs (1:N join)

AB

order-history-page

customer-service order-service


• Many common systems do thiso Web application “mashup”



• Approach 2: “Materialize the View”o Listen to events from item-service and order-feedback-serviceo Maintain denormalized join of items and order feedback in local storage

• Best for high cardinality A and B (M:N join)

Items Order Feedback

item-feedback-serviceorder-feedback-service

item-service


• Many common systems do thiso Most NoSQL approacheso “Materialized view” in database systemso Search engineso Analytic systemso Log aggregators


Microservice Techniques:Workflows and Sagas

• Problemo Monolithic database makes transactions across multiple entities easyo Splitting data across services makes transactions very hard



• Transaction è Sagao Model the transaction as a state machine of atomic events

• Reimplement as a workflow

• Roll back by applying compensating operations in reverse

A B C D

A B C D



• Many common systems do thiso Payment processingo Expense approval o Any multi-step workflow



• Ideal use for Functions as a Service (“Serverless”)o Very lightweight logico Statelesso Triggered by an event

A B C D

A B C D

ƛ

ƛ

ƛ

ƛ

ƛ

ƛ

ƛ

ƛ



TDD and Continuous

Delivery

DevOpsMicroservices

Small Teams

Thanks!• Stitch Fix is hiring!

o www.stitchfix.com/careerso Based in San Franciscoo Hiring everywhere!o More than half remote, all across USo Application development, Platform engineering,

Data Science

• Please contact meo @randyshoupo linkedin.com/in/randyshoup

Documents

Managing Data in Microservices - QCon New York · • Most applications deployed multiple times per day ... MySQL, etc.}, owned and operated by the service team ... o Analytic systems