View
357
Download
3
Category
Preview:
Citation preview
Replacing a Jet Engine Mid-flight
Or, how we launched new architecture for
planet-scale distributed system at Google
Aysylu GreenbergAugust 22, 2016
Aysylu Greenberg
@aysylu22
Build System StoryPhoto by imanka / CC BY
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Towards Distributed Build System
Build System StoryPhoto by imanka / CC BY
Architecture UpgradePhoto by magilla03 / CC BY SA
BuildRabbit:From Experimental Project to Key Piece of Infrastructure
Photo by magilla03 / CC BY SA
From Push to Pull: Client / Server
Client for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
From Push to Pull: Build Service
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
From Push to Pull: Build Service
Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
RESILIENCE
LOGIC
RESILIENCE LOGIC
Architecture UpgradePhoto by magilla03 / CC BY SA
Continuous
integration
system
BuildRabbit Blaze
Source
System
engineers &
teams
Release
infrastructure
Integration
testing
infrastructure
Build
artifact
storage
Continuous
integration
system
BuildRabbit
Source
System
engineers &
teams
Release
infrastructure
Integration
testing
infrastructure
Build
artifact
storage
IN THE
CLOUD
REPLACING A JET ENGINE MID-FLIGHT
Confidential + Proprietary
Photo by moonjazz / CC BY SA
Migration with Zero Downtime
MIGRATE BACKENDS FIRST
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
Photo by goosmurf / CC BY
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
>>> Compatibility window is several months
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODEMIGRATE BACKENDS FIRST
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODELOTS OF THROWAWAY CODE
GRADUAL ROLLOUT
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTSFOCUS ON MIXED MODE
PRACTICE BEFORE LAUNCH
GRADUAL ROLLOUT
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTS
PRACTICE BEFORE LAUNCH
GRADUAL ROLLOUT
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTS
PRACTICE BEFORE LAUNCH, A LOT
GRADUAL ROLLOUT
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTS
Photo by moonjazz / CC BY SA
Migration with Zero Downtime
Replacing a Jet Engine Mid-flight
Or, how we launched new architecture for
planet-scale distributed system at Google
Aysylu Greenberg@aysylu22
Recommended