Lyndsay Prewer - Smoothing the continuous delivery path - a tale of two teams

Smoothing the continuous delivery path

Smoothing the continuous delivery path

A tale of two teams

SLOW and CLEAR speachPLAIN languageBreathe!

Introduce myself (lyndsayp Ltd and EE)Roles

Context

Average time between production releases

Establish:- what roles are present- how many deliver to web

Show of hands - production deployment capability: one or more per year, per quarter, per month, per week, per day

Whats this all about?

ContinuousDeliveryDone WellPrivate sector .Net MonolithGovernment Scala microservices

Both teams are doing a great job of getting their doughnuts (of differing sizes and shapes) out to Production.

This is an experience report will compare and contrastWhat best practices help these team go faster, fasterWhat areas of pain slow them down

Whats this (Continuous Delivery) all about?

... producing valuable software in short cycles and ensure that the software can be reliably released at any time

- https://en.wikipedia.org/wiki/Continuous_delivery

TODO - show / hide russian / lithuanian logoEmphasise:Fast feedback Risk reduction

Continuous doughnuts

Fast feedback

Risk Reduction

T: 0:00:000

T: 2:00:000

Valuable software in short cycles reliably released at any time

Fast feedback - shortest time from concept to cash

Risk reduction - smallest value possible

Theres no (CD) silver bullet

1 (php) monolith50 deploys / day1.5 billion page views / month

http://goo.gl/XIB29i

>600 java microservices>100 deploys / day~2 billion requests per day

http://goo.gl/ZJBiK6

Prefer Continuous Improvement over Continuous Delivery

.Net monolith

26 -> 24 -> 34 releases per year

https://goo.gl/KF9xXw

2 hours East of here theres an IT company that makes an awesome product, Target Process.

Last month, their CEO published an insightful post on how theyve evolved over the last 90 months.Its a fascinating read, and covers the gory details of their journey through various agile best practices.

Over the last three years theyve improved their delivery frequency, but still have some way to go. Theyve made lots of improvements in other areas though.

The article reminds us that when improving ones Continuous Delivery, its important not to do so at the expense of other practices.

The overarching goal has to be Continuous Improvement, not Continuous Delivery.

.Net monolith team - how they roll

Private sector

Significant legacy code

High daily traffic (~100 million visits)

~8 product teams

Weekly deployments:

Pick release candidate

Regression test

Dog food

Deploy

Scala microservices team - how they roll

.Gov - HMRC Digital

2 years old (greenfield)

Calendar spikes (Visits peak is ~4 million)

>30 product teams

Multiple deployments per day

Teams own their own services

You build it, you run it - https://goo.gl/DyfeOf

HMRC Digital (Public sector)Gov.uk tax interfaces for citizens and businesses (70% of all gov transactions, > 1 billion transactions online)

Traffic varies across the year, with peaks at key tax calendar events

30+ Product Teams

Tech StackScalaMicroservicesDockerMongo DB

Deployment frequencyYou build it, you run it - Amazon CTO Werner Vogels

ContinuousDeliveryDone WellPatterns for successful practice

Best practice #1 - Healthy CI - mind & body

CI that worksCulture - Mind share

Best practice #1 - Healthy CI - mind & body

Almostgreen!

CI that works - knowing what to automate - a story about NVM

Best practice #2 - Testing as an activity, not a phase

Product OwnerDefineDevBuildQATestWeb OpsDeploy

Cross functional teamsPicture of team mixPicture of testing all the time

Story about Elateral

Best practice #2 - Testing as an activity, not a phase

Product OwnerDefineDevBuildQATestWeb OpsDeployTest

Cross functional teamsPicture of team mixPicture of testing all the time

NVM and HMRC

Best practice #3 - Healthy automated tests

Unit TestsAPIUIThe test automation pyramid

API and/or integration tests

Best practice #3 - (Un)healthy automated tests

APIUIUnit TestsThe Hour Glass

Picture different test mixesCommon automated testing problems

UIBest practice #3 - (Un)healthy automated tests

APIThe Ice Cream Cone

Unit TestsPicture different test mixesCommon automated testing problemsPicture different test mixes, highlighting problems

Best practice #3 - Healthy automated tests

Unit TestsAPIUI

The Tear Drop

Story about NVM

Tear drop shapeBehavioural focus

Reference to TDD - where did it all go wrong?

Best practice #4 - Low (manual) cost deployment (and rollback)

VS.

Story about Elateral, then NVMStory about HMRC (rollback)

Best practice #5 - Metrics, monitoring and alerting

http://seluxkanaur.deviantart.com/art/Sauron-s-Eye-6-420535318

Monitor all things!Eye of Sauron, looking at the whole stack (user behaviour, down to disk space)Highlight:What to focus on (behaviour instead of environment)Leading vs. trailing indicators

Metrics, monitoring and alerting - getting it right

DetectAlertRespondDisplayAnalyseTime & Pain increasing!

DetectAlertRespondDisplayAnalyseDiagram of right and wrong mix of the above

Metrics, monitoring and alerting - getting it wrong

DetectRespondDisplayAnalyseTime & Pain increasing!

AlertDetectAlertRespondDisplayAnalyseDiagram of right and wrong mix of the above

Metrics, monitoring and alerting - where and how?

DetectAlertRespondDisplayAnalyse

ToolsLogos and Pictures of toolsNew RelicGraphanaKIbanaLine of logging codeElastic SearchGraphiteSensuPapertrailPagerdutySplunkTell a storyMongo query defect at HMRCKibanaChange in success rateKibana / splunk

Metrics, monitoring and alerting - stories from the trenches

DetectAlertRespondDisplayAnalyse

Tell a storyMongo query defect at HMRCKibanaChange in success rateKibana / splunk

Best practice summary

Healthy CI (mind and body)

Testing as an activty, not a phase

Tear drop shape automated tests (focus on behaviour, with loose coupling)

Low (manual) cost deployment and rollback

Metrics, monitoring and reporting

Product OwnerDefineDevBuildQATestWeb OpsDeployTestUnit TestsAPIUIDetectAlertRespondDisplayAnalyse

Shared Pain

ContinuousDeliveryDone Well

Shared pain #1 - Zero down-time releases

https://en.wikipedia.org/wiki/VO2_max

Making changes to Running machine / aeroplane

Stories and Diagram ofCookie changeDatabase change

Shared pain #2 - Manual deployment steps

https://en.wikipedia.org/wiki/Tailor

https://en.wikipedia.org/wiki/Zara_(retailer)

VS.

Story about no more manual scripts / steps

Shared pain summary

Zero down time releases

Manual deployment steps

VS.

Team specific accelerators


Accelerators - 5 whys

Create a timeline.

Gather those involved.

Walk through the timeline.

Choose an entry point event:

Event

Why?

Process improvement

Repeat x5 (ish)

Root cause analysis (5 why's)Child asking why5 whys template

Accelerators - 5 whys - example

Accelerators - automated release notes - the problem #1

Much confusion and pain

Automated release noteSpot the difference (automated release note)Diagram of how it worksDiagram of layout

Accelerators - automated release notes - the problem #2

Agile Project Management ToolSource control systemFeaturesStoriesDefects

LoginReportingPaymentsAPIs

Accelerators - automated release notes - the solution

Agile Project Management ToolSource control systemFeaturesStoriesDefectsLoginIdentify source changes

Identify stories/defects

Identify areas

Group by story/defect

For each, summarise areas and authors

Group by:

Whats done

Whats in progress

Untracked work

ReportingPaymentsAPIs

Team specific accelerators


Accelerators - microservice architecture

Teams own their own services

You build it, you run it - https://goo.gl/DyfeOf

Prefer service over library dependencies

Microservice architectureAmazon quote - Build It, Run It, Own itSeparation of features and concernsPrefer service dependencies over library dependencies

Accelerators - backward compatible APIs (principle of least surprise)

https://en.wikipedia.org/wiki/VO2_max

Everyone shares(and cares)

Principle of least surpriseService API's must always be backwardly compatible

Story of when we were young, teams wanted to test against fixed versions of other services

Team specific accelerators summary

Monolith:

5 -whys

Automated release note

Microservices:

Microservice dependencies, not library dependencies

Backward compatible APIs

Agile Project Management ToolSource control systemFeaturesStoriesDefectsLoginReportingPaymentsAPIs

References and resources

Build Quality In (promo code - LP2015 - https://goo.gl/ZRuQKw)

Continuous Delivery

Implementing Lean Software Development

Release It

Promote Build Quality InExcellent chapter in ILSD on VSMs

Thank You

Twitter@EqualExperts@lyndsp

LinkedInlinkedin.com/company/equal-expertsuk.linkedin.com/in/lyndsp

UNITED KINGDOM

+44 203 603 [email protected] Experts UK Ltd30 Brock StreetLondon NW1 3FG

INDIA

+91 20 6607 [email protected] Experts India Private LtdOffice No. 4-CCerebrum IT Park No. B3Kumar City, Kalyani NagarPune, 411006Webwww.equalexperts.comwww.lyndsayp.com

CANADA

+1 403 775 [email protected] Experts Devices Inc205 - 279 Midpark way S.E.
T2X 1M2
Calgary, AlbertaPORTUGAL

+ [email protected] Experts PortugalRua Toms da Fonseca
- Torres de LisboaTorre G, 5 Andar1600-209 LisboaThank You

Equal Experts UK Ltd and lyndsayp ltd 2015

@EqualExperts

@lyndsp


@EqualExperts

@lyndsp

simple software solutions to big business problems.

Making Software. Better.


@EqualExperts

@lyndsp

Thank You

Twitter@EqualExperts

LinkedInlinkedin.com/company/equal-experts

UNITED KINGDOM


INDIA

+91 20 6607 [email protected] Experts India Private LtdOffice No. 4-CCerebrum IT Park No. B3Kumar City, Kalyani NagarPune, 411006Webwww.equalexperts.com

CANADA

T2X 1M2



@EqualExperts

@lyndsp


@EqualExperts

@lyndsp


@EqualExperts

@lyndsp


@EqualExperts

@lyndsp




@EqualExperts

@lyndsp

Thank You



UNITED KINGDOM


INDIA


CANADA

T2X 1M2



@EqualExperts

@lyndsp


@EqualExperts

@lyndsp


@EqualExperts

@lyndsp


@EqualExperts

@lyndsp




@EqualExperts

@lyndsp

Thank You



UNITED KINGDOM


INDIA


CANADA

T2X 1M2



@EqualExperts

@lyndsp


@EqualExperts

@lyndsp