VINCENT KOK • DEVELOPMENT MANAGER • ATLASSIAN • @VINCENTKOK

Releasing the Monolith On a Daily Basis

@vincentkok

http://geekandpoke.typepad.com/geekandpoke/2011/08/hudson-status-monitor.html

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Cool stories

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Got issues?

https://flic.kr/p/fSopv4

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Journey

https://flic.kr/p/8WQutt

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Monolith

https://flic.kr/p/nA4S49

C O N F L U E N C E P I P E L I N E

M I C R O S E RV I C E E N V Y

4 P R O B L E M S

Agenda

C U LT U R E

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Confluence Pipeline

https://flic.kr/p/jus4Bt

Nr of builds last week

15000

Nr of jobs last week

81000

Nr of tests

31992

Nr of build agents

1000

Flakes

NaN

Size of Pipeline team

3

A typical pipeline

ReleaseInner Outer

Publish Deploy

The Confluence pipeline

ReleaseInner Outer

Publish Deploy

The Confluence pipeline

ReleaseInner Outer

Publish Deploy

Staging deploy

The Confluence pipeline

ReleaseInner Outer

Publish Deploy

Staging deploy

Manual release branch

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Culture

https://flic.kr/p/DsetEn

GreenDex

GreenDex

The Oz Principle

Experience

The Oz Principle

The way you’ve been shaped

BeliefsExperience

The Oz Principle

The way you’ve been shaped

Beliefs are based on your experience

BeliefsExperience Actions

The Oz Principle

The way you’ve been shaped

Beliefs are based on your experience

You take actions based on your beliefs

ResultsBeliefsExperience Actions

The Oz Principle

The way you’ve been shaped

Beliefs are based on your experience

You take actions based on your beliefs

The outcome of our actions

Clear goal Early results Momentum

Theory in real life

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Microservice Envy

https://flic.kr/p/b7ak8e

Smaller Clear ownership Independent Lifecycle

Microservice traits

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Monolith

https://flic.kr/p/nA4S49

< 1 hour

< 1 day

Release cadence Test automation

Ownership Complexity

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

#1: Complexity

https://flic.kr/p/cFM3cd

The real version

ReleaseInner Outer

Publish Deploy

Staging deploy

Manual release branch

Manual tasks Hard to understand

Inconsistent feedback

Impact

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Fix the model

https://flic.kr/p/8ykpkW

Fork / Join

Release

Publish Deploy

Inner Outer

Fork

Join

Fork/Join details

Inner loop Build times Build stability

Iterate

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

#2: Ownership

https://flic.kr/p/bfRuoX

Getting the model wrong

Fire fighting Broken windows No improvements

Impact

Dev Speed Feedback loop MTTR

Impact

Ownership

Development Teams Site Reliability Engineering

Ownership

Development Teams

Sustainable environment

Site Reliability Engineering

Pipeline team - BuildOps

Where does your team sit?

Busywork

Improving team life

Improving team life

Enforce, don’t fix

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Sustainable System

https://flic.kr/p/awVLLs

Early result

Move the needle

Measure everything

VisibilityCreate awareness in the wider team

Gather data to allow data driven decisions and prioritisation

Focus on quick wins to move the needle

Define rules Shared OwnershipEnforce by automation

Iterate

Shared ownership

Shared ownership

First responder

First responder

First responder

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

#3: Testing

https://flic.kr/p/cBbxuQ

Unit

Service

UI

Unit

Service

UI

Slowness Duplication Flakes

Impact

The impact of flakes

Broken windows

Culture

Build timesDue to restarts build times get out of control really quickly

Builds will go red people will start to ignore this; Culture of green out of the window

It starts with one, many will follow soon

Retries as a pain killerAutomatic retries to relieve some pain. Remember this is a workaround

Early result

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Fix the root cause

https://flic.kr/p/J44Nm

Iterate

Test strategy Flake policy

Fix or RemoveIssueDetect Ignore

Flake policy

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Deleting tests, seriously?

https://flic.kr/p/8BYXV4

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

#4: Releases

https://flic.kr/p/91MFkh

Release cycle

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Congestion

https://flic.kr/p/bRgBZP

Dogfooding becomes hard

Big bangEnds at master Cancellations

Impact

For once a free lunchFork and Join solved most of our problems already

Early result

Iterate

Reduce build times Fix dogfooding

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Dogfooding

https://flic.kr/p/dxpLTm

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Feedback

https://flic.kr/p/dxaAdk

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Soak time

https://flic.kr/p/fmfG2q

Decouple code and features

Merge often One process

Feature flags

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

How far in are we?

https://flic.kr/p/8WQutt

Release cadence Test automation

Ownership Complexity

Clear goal Early results Momentum

Changing beliefs

• Big cool statistic

• 2,569

• Add-Ons in Marketplace

Sustainable System

https://flic.kr/p/awVLLs

http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef015392a77686970b-pi

Thank you!

VINCENT KOK • DEVELOPMENT MANAGER • ATLASSIAN • @VINCENTKOK