Replacing a Jet Engine Mid-flight

Or, how we launched new architecture for

planet-scale distributed system at Google

Aysylu GreenbergAugust 22, 2016

Aysylu Greenberg

@aysylu22

Build System StoryPhoto by imanka / CC BY

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Towards Distributed Build System

Build System StoryPhoto by imanka / CC BY

Architecture UpgradePhoto by magilla03 / CC BY SA

BuildRabbit:From Experimental Project to Key Piece of Infrastructure

Photo by magilla03 / CC BY SA

From Push to Pull: Client / Server

Client for

User

BuildRabbit

Scheduler

BuildRabbit

Worker

RPCeventstream

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

From Push to Pull: Build Service

RPCeventstream

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

From Push to Pull: Build Service

Client

for

User

BuildRabbit

Scheduler

BuildRabbit

Worker

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

Client

for

User

BuildRabbit

Scheduler

BuildRabbit

Worker

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

Client

for

User

BuildRabbit

Scheduler

BuildRabbit

Worker

RESILIENCE

LOGIC

RESILIENCE LOGIC

Architecture UpgradePhoto by magilla03 / CC BY SA

Continuous

integration

system

BuildRabbit Blaze

Source

System

Google

engineers &

teams

Release

infrastructure

Integration

testing

infrastructure

Build

artifact

storage

Continuous

integration

system

BuildRabbit

Source

System

Google

engineers &

teams

Release

infrastructure

Integration

testing

infrastructure

Build

artifact

storage

IN THE

CLOUD

REPLACING A JET ENGINE MID-FLIGHT

Confidential + Proprietary

Photo by moonjazz / CC BY SA

Migration with Zero Downtime

MIGRATE BACKENDS FIRST

RPCeventstream

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

RPCeventstream

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

LOTS OF THROWAWAY CODE

MIGRATE BACKENDS FIRST

Photo by goosmurf / CC BY

FOCUS ON MIXED MODE

LOTS OF THROWAWAY CODE

MIGRATE BACKENDS FIRST

FOCUS ON MIXED MODE

LOTS OF THROWAWAY CODE

MIGRATE BACKENDS FIRST

>>> Compatibility window is several months

TARGET LAUNCH-FRIENDLY CLIENTS

FOCUS ON MIXED MODE

LOTS OF THROWAWAY CODE

MIGRATE BACKENDS FIRST

RPCeventstream

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

RPCeventstream

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

DECOUPLE LAUNCH OF SERVICES

TARGET LAUNCH-FRIENDLY CLIENTS

FOCUS ON MIXED MODE

LOTS OF THROWAWAY CODE

MIGRATE BACKENDS FIRST

RPCeventstream

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

RPCeventstream

User

Persistent

Queue

BuildRabbit

Worker

Build Artifacts

Build Progress

Info

HAVE A SOLID ROLLBACK PLAN

DECOUPLE LAUNCH OF SERVICES

TARGET LAUNCH-FRIENDLY CLIENTS

FOCUS ON MIXED MODE

LOTS OF THROWAWAY CODEMIGRATE BACKENDS FIRST

MAXIMUM VISIBILITY INTO SYSTEM STATE

HAVE A SOLID ROLLBACK PLAN

DECOUPLE LAUNCH OF SERVICES

TARGET LAUNCH-FRIENDLY CLIENTS

FOCUS ON MIXED MODELOTS OF THROWAWAY CODE

GRADUAL ROLLOUT

MAXIMUM VISIBILITY INTO SYSTEM STATE

HAVE A SOLID ROLLBACK PLAN

DECOUPLE LAUNCH OF SERVICES

TARGET LAUNCH-FRIENDLY CLIENTSFOCUS ON MIXED MODE

PRACTICE BEFORE LAUNCH

GRADUAL ROLLOUT

MAXIMUM VISIBILITY INTO SYSTEM STATE

HAVE A SOLID ROLLBACK PLAN

DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTS

PRACTICE BEFORE LAUNCH

GRADUAL ROLLOUT

MAXIMUM VISIBILITY INTO SYSTEM STATE

HAVE A SOLID ROLLBACK PLAN

DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTS

PRACTICE BEFORE LAUNCH, A LOT

GRADUAL ROLLOUT

MAXIMUM VISIBILITY INTO SYSTEM STATE

HAVE A SOLID ROLLBACK PLAN

DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTS

Photo by moonjazz / CC BY SA

Migration with Zero Downtime

Replacing a Jet Engine Mid-flight

Or, how we launched new architecture for

planet-scale distributed system at Google

Aysylu Greenberg@aysylu22