52
SEACON:UK 2017 Breakout Track Exploring different approaches to Continuous Delivery Pipelines

Seacon Continuous Delivery Pipeline Tools Track

Embed Size (px)

Citation preview

Page 1: Seacon Continuous Delivery Pipeline Tools Track

SEACON:UK 2017 Breakout Track

Exploring different approaches to Continuous Delivery Pipelines

Page 2: Seacon Continuous Delivery Pipeline Tools Track

Diversion into Inclusion & Diversity1.It should be down to all of us to champion this at every

opportunity.2.We all suffer from Unconscious Bias and even if this has just a

0.1% on the opportunities that we give others, cumulatively this stacks up. (Take a test: https://implicit.harvard.edu/implicit/selectatest.html)

3.Mirco aggression is a term for behaviour that intimidates others and make it harder for them to perform a their best in the working environment. Be vigilant and call it out. (Real examples here: http://www.microaggressions.com/)

Other links:• https://www.fastcompany.com/3044738/hit-the-ground-running/7-simple-methods-to-fight-against-your-

unconscious-biases• https://codeascraft.com/2016/10/19/being-an-effective-ally-to-women-and-non-binary-people/• http://weprinciples.org/files/attachments/

WHITE_PAPER_Nudging_the_Unconscious_Mind_Nielsen_&_Kepinski_Oct_2014.pdf• https://managingbias.fb.com/• http://www.cmu.edu/news/stories/archives/2015/june/bias-blind-spot.html

Page 3: Seacon Continuous Delivery Pipeline Tools Track

Agenda• Introduction to Continuous Delivery In Practice

• This Exercise

• XebiaLabs - XL Deploy and XL Release

• ADOP and Jenkins

• Thoughtworks GoCD

• General Q&A

Page 4: Seacon Continuous Delivery Pipeline Tools Track

The Objective: Continuous Delivery of Ideas

FEEDBACK

Page 5: Seacon Continuous Delivery Pipeline Tools Track

BuildStatic Code Unit Tests

Source CodeRepo

CI pipeline

Deploy Code

Run Test Harness

Deploy CodePPE

Perf,Security

TestDeploy

Prod

CD pipeline(rest of the way to Prod)

Automating the Workflow from Commit to Release

Page 6: Seacon Continuous Delivery Pipeline Tools Track

The Workflow needs to hold up

How do we control flow?

What versions

were tested

together?

What should

we release?

Page 7: Seacon Continuous Delivery Pipeline Tools Track

Tests configurationBuild tools configuration

Deployment software configurationMiddleware configuration

Operating System configuration Virtual infrastructure configuration

Hardware configuration

Business

Application

Business

Application

Business

Application

Business

Application

Business

Application

Compile Code AnalysisCommitter: jdoe

Story:25

Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test

Compile Code AnalysisCommitter: jdoe

Story:25

Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test

Compile Code AnalysisCommitter: jdoe

Story:25

Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test

Compile Code AnalysisCommitter: jdoe

Story:25

Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test

Compile Code AnalysisCommitter: jdoe

Story:25

Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test

We should probably do this all the way down!

Page 8: Seacon Continuous Delivery Pipeline Tools Track

Commits of any Source Code

triggers a pipeline

???

One pipeline won’t work - the tools need handle complexity

BuildStatic Code Unit Tests

Deploy Code

Run Test

Harness

Deploy CodePPE

Perf,Securit

y Test

Deploy Prod

Page 9: Seacon Continuous Delivery Pipeline Tools Track

Luckily in 2017 they all do now support a lot things e.g.:

• Agents for secure remote execution on different platforms

• Application security

• Role Based Access Control

• Reporting and metrics

• Automated and manual triggers and flow control

• Commercial support

• Extensibility

Page 10: Seacon Continuous Delivery Pipeline Tools Track

But how do they differ?

THE EXCERCISE

Page 11: Seacon Continuous Delivery Pipeline Tools Track

Application DesignReference Application Stack• Frontend App• Backend App (tightly coupled to frontend – both must be released

together)• Configuration Management = Chef or Puppet• Provisioning = TerraformOrchestration & Automation Scripts = (you choose) Tools• Software configuration Management = (you choose but not of

concern)• Binary storage = (you choose but not of concern)• Testing = (you choose and but of concern)• Monitoring = (you choose but not of concern)• Automation of build = (you choose but not of concern)• IaaS platform = (you choose but not of concern)Automation of deployment = (you choose)Continuous Delivery Pipeline = (you choose)

Page 12: Seacon Continuous Delivery Pipeline Tools Track

Example PipelineFrontend

Code, Config

BackendCode, Config

Config MgmtScripts

ProvisioningScripts

Orchestration& Automation

Scripts

Build & Package

Unit TestCode Analysis

Build & Package

Unit TestCode Analysis

Unit Test &Code Analysis

Sandbox Testing(various)

Install &ConfigureNon-Prod

Runtime Testing(various)

Install &ConfigureProduction

Unit Test &Code Analysis

Sandbox Testing(various)

ProvisionNon-Prod

Runtime Testing(various)

ProvisionProduction

Unit Test &Code Analysis

Sandbox Testing(various)

UpdateJobs /Flows

MonitoringBackupHealingetc.

System TestDeploy

System TestDeploy

Install &ConfigureSystem Test

Runtime Testing(various)

Non-ProdDeploy

Runtime Testing(various)

ProductionDeploy

Runtime Testing(various)

Runtime Testing(various)

ProvisionSystem Test

Runtime Testing(various)

Page 13: Seacon Continuous Delivery Pipeline Tools Track

FrontendCode, Config

BackendCode, Config

Config MgmtScripts

ProvisioningScripts

Orchestration& Automation

Scripts

Build & Package

Unit TestCode Analysis

Build & Package

Unit TestCode Analysis

Unit Test &Code Analysis

Sandbox Testing(various)

Install &ConfigureNon-Prod

Runtime Testing(various)

Install &ConfigureProduction

Unit Test &Code Analysis

Sandbox Testing(various)

ProvisionNon-Prod

Runtime Testing(various)

ProvisionProduction

Unit Test &Code Analysis

Sandbox Testing(various)

UpdateJobs /Flows

MonitoringBackupHealingetc.

System TestDeploy

System TestDeploy

Install &ConfigureSystem Test

Runtime Testing(various)

Non-ProdDeploy

Runtime Testing(various)

ProductionDeploy

Runtime Testing(various)

Runtime Testing(various)

ProvisionSystem Test

Runtime Testing(various)

Visualisation and release Tracking1. How can “what is deployed where” be visualised and tracked (over time) and how can tested combinations of components be tracked and locked-in?

2. How can it be enforced that a consistent use of automation is used across deployments to different environments?

Page 14: Seacon Continuous Delivery Pipeline Tools Track

FrontendCode, Config

BackendCode, Config

Config MgmtScripts

ProvisioningScripts

Orchestration& Automation

Scripts

Build & Package

Unit TestCode Analysis

Build & Package

Unit TestCode Analysis

Unit Test &Code Analysis

Sandbox Testing(various)

Install &ConfigureNon-Prod

Runtime Testing(various)

Install &ConfigureProduction

Unit Test &Code Analysis

Sandbox Testing(various)

ProvisionNon-Prod

Runtime Testing(various)

ProvisionProduction

Unit Test &Code Analysis

Sandbox Testing(various)

UpdateJobs /Flows

MonitoringBackupHealingetc.

System TestDeploy

System TestDeploy

Install &ConfigureSystem Test

Runtime Testing(various)

Non-ProdDeploy

Runtime Testing(various)

ProductionDeploy

Runtime Testing(various)

Runtime Testing(various)

ProvisionSystem Test

Runtime Testing(various)

Developing and Testing Automation3. What does the activity of creating automation and orchestration entail (Coding, DSL, Drag and Drop etc.)?

4. How is automation and orchestration tested and new versions released?

5. How can people share and re-use automation packages?

Page 15: Seacon Continuous Delivery Pipeline Tools Track

FrontendCode, Config

BackendCode, Config

Config MgmtScripts

ProvisioningScripts

Orchestration& Automation

Scripts

Build & Package

Unit TestCode Analysis

Build & Package

Unit TestCode Analysis

Unit Test &Code Analysis

Sandbox Testing(various)

Install &ConfigureNon-Prod

Runtime Testing(various)

Install &ConfigureProduction

Unit Test &Code Analysis

Sandbox Testing(various)

ProvisionNon-Prod

Runtime Testing(various)

ProvisionProduction

Unit Test &Code Analysis

Sandbox Testing(various)

UpdateJobs /Flows

MonitoringBackupHealingetc.

System TestDeploy

System TestDeploy

Install &ConfigureSystem Test

Runtime Testing(various)

Non-ProdDeploy

Runtime Testing(various)

ProductionDeploy

Runtime Testing(various)

Runtime Testing(various)

ProvisionSystem Test

Runtime Testing(various)

Build & Package

Unit TestCode Analysis

System TestDeploy

Operational Concerns7. How are the Continuous Delivery Pipeline tools upgraded / patched and typically how often does this happen and why?

6. How can the platform be extended e.g. Via Plugins?

Page 16: Seacon Continuous Delivery Pipeline Tools Track

The main event!

• Ben Hählen – Xebia Labs – XL Deploy and XL Release

• Hitesh Joshi – Accenture – ADOP and Jenkins

• Marius Ciotlos – Betfair – Thoughtworks GoCD

Page 17: Seacon Continuous Delivery Pipeline Tools Track
Page 18: Seacon Continuous Delivery Pipeline Tools Track

XL Release and XL Deploy

Page 19: Seacon Continuous Delivery Pipeline Tools Track

XL Release

Page 20: Seacon Continuous Delivery Pipeline Tools Track

XL Release

Page 21: Seacon Continuous Delivery Pipeline Tools Track

XL Release

Page 22: Seacon Continuous Delivery Pipeline Tools Track

XL Release

Page 23: Seacon Continuous Delivery Pipeline Tools Track

XL Deploy

Page 24: Seacon Continuous Delivery Pipeline Tools Track

XL Deploy

Page 25: Seacon Continuous Delivery Pipeline Tools Track

XL Deploy

Page 26: Seacon Continuous Delivery Pipeline Tools Track

To facilitate comparison we would like to propose the following scenario. The objective of the exercise is to design (no need to implement) a solution for achieving continuous delivery of the following.

Reference Application Stack• App 1 = Front-End• App 2 = Back-End• Configuration Management = Chef or Puppet• Provisioning = Terraform Orchestration & Automation Scripts = XL Release Tools• Software configuration Management = Git, SVN• Binary storage = Nexus, Artifactory, etc• Testing = Sonar, Cucumber, etc• Monitoring = ELK, Splunk, etc• Automation of build = Jenkins, VSTS, TFS, etc• IaaS platform = On-premise, Azure, Kubernetes, Docker, etc Automation of deployment = XL Deploy Continuous Delivery Pipeline = XL Release

Application Design

Page 27: Seacon Continuous Delivery Pipeline Tools Track

1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?

Page 28: Seacon Continuous Delivery Pipeline Tools Track

1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?

Page 29: Seacon Continuous Delivery Pipeline Tools Track

1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?

Page 30: Seacon Continuous Delivery Pipeline Tools Track

1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?

Page 31: Seacon Continuous Delivery Pipeline Tools Track

2. How can it be enforced that a consistent use of automation is used across deployments to different environments?

Page 32: Seacon Continuous Delivery Pipeline Tools Track

3. What does the activity of creating automation and orchestration entail (coding, DSL, drag and drop etc.)?

Page 33: Seacon Continuous Delivery Pipeline Tools Track

3. What does the activity of creating automation and orchestration entail (coding, DSL, drag and drop etc.)?

Page 34: Seacon Continuous Delivery Pipeline Tools Track

4. How is automation and orchestration tested and new versions released?

Page 35: Seacon Continuous Delivery Pipeline Tools Track

5. How can people share and re-use automation packages?

Page 36: Seacon Continuous Delivery Pipeline Tools Track

6. How can the platform be extended e.g. via plugins?

Page 37: Seacon Continuous Delivery Pipeline Tools Track

7. How are the Continuous Delivery Pipeline tools upgraded / patched and typically how often does this happen and why?

Short Term Support : Every 2 months

Long Term Support : Every 6 monthsMaintenance Release : Every 2 months

Page 38: Seacon Continuous Delivery Pipeline Tools Track

THANKS

Page 39: Seacon Continuous Delivery Pipeline Tools Track

About Me

RAPID MOBILISATION

GLOBAL SHARED SERVICES TEAM

SUPPORTING 24 / 7

Software Configuration Management

Release ManagementAssessments

Environment Management

DevOps TransformationBuild & Deploy Automation Continuous Delivery

Rational and Jira

Infrastructure as code CloudPaaS Tooling

Page 40: Seacon Continuous Delivery Pipeline Tools Track

Jenkins

Page 41: Seacon Continuous Delivery Pipeline Tools Track

Jenkins with ADOP• All open source• Commercial Jenkins option though CloudBees• Accenture created ADOP so we can help

1: December 2016: https://www.cloudbees.com/jenkins/about

Stats• 150,000 Jenkins active installations, 1M users, 12.5M jobs (1)

• Jenkins plugins: 1200+• Jenkins Pipeline installations: 77,000+• ADOP: ~550 forks + ~200 stars• ADOP plugins (cartridge) > 50

Page 42: Seacon Continuous Delivery Pipeline Tools Track

On Premise

Docker Machine

Docker Compose

ADOP Platform Extension

ADOP Cartridge

AWS Cloud

AWS regionAvailability

Zone

Internet

Selenium

Platform for Reusable Automation

Cartridges

Page 43: Seacon Continuous Delivery Pipeline Tools Track

Solution ComponentsReference Application Stack• App 1 = Hybris• App 2 = SOLR• Configuration Management = Chef or Puppet• Provisioning = Terraform Orchestration & Automation Scripts = ADOP, Jenkins, Ansible

Tools• Software configuration Management = Git• Binary storage = Nexus (with Lifecycle)• Testing = Cucumber, Selenium Grid• Monitoring = Sensu and ELK• Automation of build = Ant and Maven• IaaS platform = AWS Automation of deployment = Ansible Continuous Delivery Pipeline = ADOP, Jenkins

Page 44: Seacon Continuous Delivery Pipeline Tools Track

Visualisation and release tracking1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?

Components

Environments

Page 45: Seacon Continuous Delivery Pipeline Tools Track

Visualisation and release tracking2. How can it be enforced that a consistent use of automation is used across deployments to different environments?

Page 46: Seacon Continuous Delivery Pipeline Tools Track

Developing and Testing Automation3. What does the activity of creating automation and orchestration entail (coding, DSL, drag and drop etc.)?

Page 47: Seacon Continuous Delivery Pipeline Tools Track

$ tree.|-- README.md|-- jenkins| `-- jobs| |-- dsl| | |-- environment_provisioning.groovy| | `-- hybris_jobs.groovy`-- src `-- urls.txt

Developing and Testing Automation5. How can people share and re-use automation packages?

A Cartridge encapsulates a reference complete implementation for a particular technology or product, typically consisting of:• Reference application• Build/Deploy Pipeline• Environment Creator – depending on technology

Hybris Cartridge

CreateEnvironment(AWS EC2,

RDS) Base Hybris Build

(Platform + Code) Deploy Hybris

Docker image with config +

code

Page 48: Seacon Continuous Delivery Pipeline Tools Track

Developing and Testing Automation4. How is automation and orchestration tested and new versions released?

Same Continuous Delivery Pipeline for Jenkins and Ansible

Page 49: Seacon Continuous Delivery Pipeline Tools Track

Operational Concerns6. How can the platform be extended e.g. via plugins?

Page 50: Seacon Continuous Delivery Pipeline Tools Track

Operational Concerns7. How are the Continuous Delivery Pipeline tools upgraded / patched and typically how often does this happen and why?

Page 51: Seacon Continuous Delivery Pipeline Tools Track

THANKS

Page 52: Seacon Continuous Delivery Pipeline Tools Track

THANKS