Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer

© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts

@lyndspsimple software solutions to big business problems.

Making Software. Better.

Smoothing the continuous delivery pathA tale of two teams


@lyndsp

What’s this all about?

2

Continuous

DeliveryDone Well


@lyndsp

What’s this (Continuous Delivery) all about?

3

“... producing valuable

software in short cycles and ensure that the software

can be reliably released

at any time”

- https://en.wikipedia.org/wiki/Continuous_delivery


@lyndsp

Continuous doughnuts

4

Fast feedback

Risk Reduction

T: 0:00:000 T: 2:00:000


@lyndsp

There’s no (CD) silver bullet

5

1 (php) monolith50 deploys / day1.5 billion page views / monthhttp://goo.gl/XIB29i

>600 java microservices>100 deploys / day~2 billion requests per day

http://goo.gl/ZJBiK6


@lyndsp

Prefer Continuous Improvement over Continuous Delivery

▪ .Net monolith▪ 26 -> 24 -> 34 releases

per year▪ https://goo.gl/KF9xXw

6


@lyndsp 7

People first, practices/process second, products last


@lyndsp

.Net monolith team - how they roll

8

▪ Private sector▪ Significant legacy code▪ High daily traffic (~100 million

visits)▪ ~8 product teams▪ Weekly deployments:

– Pick release candidate– Regression test– Dog food– Deploy


@lyndsp

Scala microservices team - how they roll

9

▪ .Gov - HMRC Digital▪ 2 years old (greenfield)▪ Calendar spikes (Visits peak is ~4

million)▪ >30 product teams▪ Multiple deployments per day

– Teams “own” their own services– “You build it, you run it” -

https://goo.gl/DyfeOf


@lyndsp

Continuous

DeliveryDone Well

Patterns for successful practice

10


@lyndsp

Best practice #1 - Healthy CI - mind & body

11


@lyndsp

Best practice #1 - Healthy CI - mind & body

12

Almostgreen!


@lyndsp

Best practice #2 - Testing as an activity, not a phase

13

Product Owner

Define

Dev

Build

QA

Test

Web Ops

Deploy


@lyndsp

Best practice #2 - Testing as an activity, not a phase

14

Product Owner

Define

Dev

Build

QA

Test

Web Ops

Deploy

Test


@lyndsp

Best practice #3 - Healthy automated tests

15

Unit Tests

API

UI


@lyndsp

Best practice #3 - (Un)healthy automated tests

16

API

UI

Unit Tests

The “Hour Glass”


@lyndsp

UI

Best practice #3 - (Un)healthy automated tests

17

API

The “Ice Cream Cone”

Unit

Tests


@lyndsp

Best practice #3 - Healthy automated tests

18

Unit Tests

API

UI

The “Tear Drop”


@lyndsp

Best practice #4 - Low (manual) cost deployment (and rollback)

19

VS.


@lyndsp

Best practice #5 - Metrics, monitoring and alerting

20

http://seluxkanaur.deviantart.com/art/Sauron-s-Eye-6-420535318

“Big data”Social mediaUser behaviourBusiness metricsService performanceService errorsDisk and Network

Monitor all things!


@lyndsp

Metrics, monitoring and alerting - getting it right

21

DetectAlert Respon

d

DisplayAnalyse

Time & Pain increasing!


@lyndsp

Metrics, monitoring and alerting - getting it wrong

22

DetectRespon

d

DisplayAnalyse

Time & Pain increasing!Alert


@lyndsp

Metrics, monitoring and alerting - where and how?

23

DetectAlert Respon

d

Display Analyse


@lyndsp

Metrics, monitoring and alerting - stories from the trenches

24

DetectAlert Respon

d

Display Analyse


@lyndsp

Best practice summary

1. Healthy CI (mind and body)

2. Testing as an activty, not a phase

3. Tear drop shape automated tests (focus on behaviour, with loose

coupling)

4. Low (manual) cost deployment and rollback

5. Metrics, monitoring and reporting

25

Product Owner

Define

Dev

Build

QA

Test

Web Ops

Deploy

Test

Unit Tests

API

UI

Detect

Alert Respond

Display Analyse


@lyndsp

Shared Pain

26

Continuous

DeliveryDone Well


@lyndsp

Shared pain #1 - Zero down-time releases

27

https://en.wikipedia.org/wiki/VO2_max


@lyndsp

Shared pain #2 - Manual deployment steps

28

https://en.wikipedia.org/wiki/Tailor https://en.wikipedia.org/wiki/Zara_(retailer)

VS.

https://en.wikipedia.org/wiki/Tailor


@lyndsp

Shared pain summary

1. Zero down time releases

2. Manual deployment steps

29

VS.


@lyndsp

Team specific accelerators

30

Continuous

DeliveryDone Well


@lyndsp

Accelerators - 5 why’s

31

● Create a timeline.● Gather those involved.● Walk through the timeline.● Choose an entry point event:

○ Event○ Why?○ Process improvement

● Repeat x5 (ish)


@lyndsp

Accelerators - 5 why’s - example

32

Event Why? Improvement

Error threshold alert triggered.

Calls to service X were timing out.

Implement failover / circuit breaker.

Calls to service X were timing out.

Queries run by Service X were taking too long.

Alert on long running queries.

Queries run by Service X were taking too long.

Index was missing for a new field.

...

... ... ...


@lyndsp

Accelerators - automated release notes - the problem #1

33

Much confusion and

pain


@lyndsp

Accelerators - automated release notes - the problem #2

34

Agile Project Management

Tool

Source control system

Features

Stories

Defects

LoginReporting

PaymentsAPI’s


@lyndsp

Accelerators - automated release notes - the solution

35


Tool


Features

Stories

Defects

Login

1. Identify source changes2. Identify stories/defects3. Identify areas4. Group by story/defect5. For each, summarise

areas and authors6. Group by:

a. What’s doneb. What’s in progressc. Untracked work

Reporting

PaymentsAPI’s


@lyndsp

Team specific accelerators

36

Continuous

DeliveryDone Well


@lyndsp

Accelerators - teams “own” their own microservices

37

▪ “You build it, you run it” - https://goo.gl/DyfeOf

▪ Prefer service over library dependencies

https://goo.gl/DyfeOf


@lyndsp

Accelerators - backward compatible API’s (principle of least surprise)

38

https://en.wikipedia.org/wiki/VO2_max

ProductionStaging

QADev

Everyone shares(and cares)


@lyndsp

Team specific accelerators summary

Monolith:1. 5 -why’s2. Automated release noteMicroservices:3. Microservice dependencies, not library

dependencies4. Backward compatible API’s

39


Tool


Features

Stories

Defects

LoginReporting

PaymentsAPI’s


@lyndsp

References and resources

40

▪ Build Quality In (promo code - LP2015 -

https://goo.gl/ZRuQKw)

▪ Continuous Delivery

▪ Implementing Lean Software

Development

▪ Release It


@lyndspsimple software solutions to big business problems.

Making Software. Better.

Thank You

Twitter@EqualExperts@lyndsp

LinkedInlinkedin.com/company/equal-expertsuk.linkedin.com/in/lyndsp

UNITED KINGDOM

+44 203 603 [email protected] Experts UK Ltd30 Brock StreetLondon NW1 3FG

INDIA

+91 20 6607 [email protected] Experts India Private LtdOffice No. 4-CCerebrum IT Park No. B3Kumar City, Kalyani NagarPune, 411006

Webwww.equalexperts.comwww.lyndsayp.com

CANADA

+1 403 775 [email protected] Experts Devices Inc205 - 279 Midpark way S.E.T2X 1M2Calgary, Alberta

PORTUGAL

+ [email protected] Experts PortugalRua Tomás da Fonseca - Torres de LisboaTorre G, 5º Andar1600-209 Lisboa

Thank You

mailto:[email protected]


http://www.equalexperts.com/

http://www.equalexperts.com/

http://www.lyndsayp.com/



