MADRID · NOV 21-22 · 2014

Designing a Release Pipeline

Jose Luis Soria

Continuous Improvement Manager at Ria Financial

@jlsoriat

MADRID · NOV 21-22 · 2014

What is a

Release Pipeline?

MADRID · NOV 21-22 · 2014

What is a Release Pipeline?

Automated manifestation of

your delivery process.

Feedback mechanism.

Detection of unfit release

candidates.

Pull system.

Useful for CD, or any other

delivery model.

MADRID · NOV 21-22 · 2014

Pipeline design considerations

Emergent design. No BDUF.

Start early.

Start simple and evolve with the system.

Begin with the most valuable assets.

Address the bottlenecks.

MADRID · NOV 21-22 · 2014

#1 Define

Components

MADRID · NOV 21-22 · 2014

MADRID · NOV 21-22 · 2014

What is a (software) component? (*)

A set of artifacts (binaries, dynamic code,

configuration files, other supporting files) that

can be deployed and verified together without

affecting other areas of the application.

(*) in a release pipeline

MADRID · NOV 21-22 · 2014

Choosing components

Deploy and test the smallest independent

entity.

Rely on the architecture:Logical / physical.

Layers / tenants.

See the whole.

MADRID · NOV 21-22 · 2014

What do we need to know?

For each component:Meaningful name.

Description.

Priority / order (when to address it?)

Source (most likely, version control.)

Target (where it gets deployed.)

Pre-requisites.

Dependencies.

Configuration tokens.

MADRID · NOV 21-22 · 2014

Configuration tokens

1. Make a list of environment-dependent

information.

2. Tokenize it in the configuration.

3. Gather the values for all the environments.

MADRID · NOV 21-22 · 2014

Component sheet

MADRID · NOV 21-22 · 2014

Activity: defining components

Our application consists on:

A web site built on top this technology stack:

MVC framework.

Client-side logic (HTML5, JavaScript.)

Entity model mapped to the DB using a ORM.

Data model residing in a DB.

MADRID · NOV 21-22 · 2014

Activity: defining components

Our application consists on:

A web site built on top this technology stack:

MVC framework.

Client-side logic (HTML5, JavaScript.)

Entity model mapped to the DB using a ORM.

Data model residing in a DB.

Key business logic resides on a web services layer.

We also maintain a mobile client for two platforms.

MADRID · NOV 21-22 · 2014

Activity: defining components

Our application consists on:A web site built on top this technology stack:

MVC framework.

Client-side logic (HTML5, JavaScript.)

Entity model mapped to the DB using a ORM.

Data model residing in a DB.

Key business logic resides on a web services layer.

We also maintain a mobile client for two platforms.

For some operations we make calls to third-party web

services.

MADRID · NOV 21-22 · 2014

Example: component sheet

MADRID · NOV 21-22 · 2014

Implementation notes

It should be possible to independently build, deploy and

test anything defined as a component.

You should decide how dependencies will be made

available:

Source Control.

Artifact repositories.

Deployed (third-party) artifacts.

Etc.

MADRID · NOV 21-22 · 2014

Tooling

Version control systemsGit, Mercurial, SVN, TFS…

Artifact repositories

Artifactory, Nexus, Frog, NuGet…

MADRID · NOV 21-22 · 2014

#2 Identify

Sub-pipelines

MADRID · NOV 21-22 · 2014

Single pipeline

A single pipeline servicing all the components

and teams.

May be able to detect which component has

changed and operate only on that one.

MADRID · NOV 21-22 · 2014

One pipeline per component

Each component has its own pipeline.

Different pipelines may have different designs.

Individual pipelines may fan-in to a system pipeline.

More flexible but more complex.

MADRID · NOV 21-22 · 2014

One pipeline per team

Each team has its own pipeline.

Different pipelines may have different designs.

Individual pipelines may fan-in to an integration pipeline.

MADRID · NOV 21-22 · 2014

Mixed approach

Different teams building different components.

Keep it simple!

MADRID · NOV 21-22 · 2014

Implementation notes

It is easier if you use a tool that allows to define

sub-pipelines, fan-in, fan-out, etc.

MADRID · NOV 21-22 · 2014

#3 Define

Stages & Orchestration

MADRID · NOV 21-22 · 2014

MADRID · NOV 21-22 · 2014

What is a stage?

A set of steps or activities that are performed on a

release candidate.

It lets any release candidate advance towards

production, or discards it.

When a release candidate passes through a stage,

our confidence on it is increased.

It is a source for feedback.

Frequently taken for environments (but they’re not

the same)

MADRID · NOV 21-22 · 2014

What is orchestration?

It is the way we arrange the stages so release

candidates flow through them, in their way to

production.

MADRID · NOV 21-22 · 2014

Tips for stages & orchestration (I)

Feedback is the key. Arrange stages and

orchestration based on the feedback we need.

Stages are filters. The orchestration should be

arranged to stop the pipeline if a stage fails.

Stages can contain both manual and

automated steps.

MADRID · NOV 21-22 · 2014

Tips for stages & orchestration (II)

Stages can be manually or automatically

triggered (think about approvals.)

Automate as much as possible. Including the

approvals.

MADRID · NOV 21-22 · 2014

Tips for stages & orchestration (III)

Grow your pipeline wide, not long

http://bit.ly/1jsNGP5

Build only once.

Use environment-agnostic binaries.

Version everything.

MADRID · NOV 21-22 · 2014

What do we need to know?

For each stage:∘ Meaningful name.

∘ Clear goal.

∘ Does it need a manual approval to be triggered?

∘ Does it need a manual verification when it has finished?

∘ Sources.

∘ Flow (orchestration.)

MADRID · NOV 21-22 · 2014

Pipeline-level orchestration

(examples)

Commit Manual testing Release

Minimum pipeline

Fully automated

Partially automated, or manual

Legend:

MADRID · NOV 21-22 · 2014

Pipeline-level orchestration

(examples)

Commit Acceptance testing Release

Exploratory testing

Capacity testing Security testingUser Acceptance

testing

Complex pipelineFully automated

Partially automated, or manual

Legend:

MADRID · NOV 21-22 · 2014

Stage-level

orchestration

(example) Code is builtOnce and Only Once:

In the first stage.

Subsequent stagesare run in parallel

if configured that way.

Pipeline gets triggered:When a developer does a check-in,

or manually

A new instance of the pipeline is

created

Get next stage, relate it to the

pipeline instance, prepare

parameters, notify for monitoring

Automatically-triggered stage?

Trigger stageWait for the user to

trigger the stage

Gather stage results and notify them for

monitoring

Succeded?

Stop the pipeline instance

Any stages left?

NOYES

NO

NO

YES

YES

MADRID · NOV 21-22 · 2014

About sources

Version control.

Artifact repositories.

Environment libraries.

MADRID · NOV 21-22 · 2014

Which stages do I need?

Think about the kind of feedback you need.

Think about what should stop a release

candidate to get to production.

Create a Value Stream Map.

MADRID · NOV 21-22 · 2014

Value stream map (example)

Assessment Approval Planning

Capacity tests

ReleaseAcceptance

tests

CodeSpecification

2 days 1 dayValue-added time

Waittime

Development: cycle time ~ ?

3 days 3 days ?

? ?

?

4 days 1 day 2 days 2 weeks ? ?

Delivery: lead time ~ ?

Exploratory tests

?

UAT

?

?

MADRID · NOV 21-22 · 2014

Prevalent stages: the Commit stage

Eliminate early release candidates that are unfit for

production.

Close to (or the same as) a CI build.

Quick validations: build, unit testing, static analysis, etc.

Packaging.

For Continuous Delivery, it runs on each commit (no

branches – feature toggles.)

For other models, decide when it gets triggered (for

example, on each merge to trunk.)

MADRID · NOV 21-22 · 2014

Prevalent stages: the Commit stage

http://bit.ly/1jsSkwA

MADRID · NOV 21-22 · 2014

Prevalent stages: the Automated

Acceptance Test stage

http://bit.ly/1jsSkwA

MADRID · NOV 21-22 · 2014

Prevalent stages: the Manual

Test stage

http://bit.ly/1jsSkwA

MADRID · NOV 21-22 · 2014

Prevalent stages:

non-Functional Testing stages

http://bit.ly/1jsSkwA

MADRID · NOV 21-22 · 2014

Activity: defining stages &

orchestration

“We have a basic suite of automated

acceptance tests that we plan to grow along

with the system.”

“The team does (manual) functional testing.”

MADRID · NOV 21-22 · 2014

Activity: defining stages &

orchestration

“We have a basic suite of automated

acceptance tests that we plan to grow along

with the system.”

“The team does (manual) functional testing.”

“We need to support 2,000 concurrent users.”

MADRID · NOV 21-22 · 2014

Implementation notes

Choose a tool that allows to easily model and

visualize the flow.

Choose a tool that supports what you need for

orchestration:∘ Approvals.

∘ Validations.

∘ Parallelization.

∘ Alerts.

∘ Etc.

MADRID · NOV 21-22 · 2014

#4 Define

Environments

MADRID · NOV 21-22 · 2014

What is an environment?

A set of servers, devices or any other

resources we need in order to run and validate

a release candidate in its way to production.

MADRID · NOV 21-22 · 2014

Tips for defining environments

Prepare for deployment automation.

Lock down environments. Restrict access.

Different stages could target the same environment if

needed.

Prepare for auto-provision.

Make environments disposable. Don’t turn them into

bottlenecks.

Environments may not be tied to stages. It should be easy

to point any stage to any environment.

MADRID · NOV 21-22 · 2014

Activity: defining environments

“We have a basic suite of automated

acceptance tests that we plan to grow along

with the system.”

“The team does (manual) functional testing.”

“We need to support 2,000 concurrent users.”

MADRID · NOV 21-22 · 2014

Implementation notes

Use virtualization.

Use cloud-based environments.

Use tools for managing templates,

configuration, auto-provision, etc.

MADRID · NOV 21-22 · 2014

Tooling

Phisical machines

Virtualization tools

VmWare, VirtualBox, Hyper-V, etc.

Containers

Docker

Cloud

Amazon, Google, Azure…

Environment definition

Vagrant, PowerShell DSC…

MADRID · NOV 21-22 · 2014#5 Define Steps

MADRID · NOV 21-22 · 2014

What is a step?

Any activity that is done in the context of a

stage, that allows us to get feedback and prove

the fitness of the release candidate.

Examples:∘ Deploy a component.

∘ Run automated tests.

∘ Run manual tests.

∘ Update metrics.

∘ Alert the user of some event.

∘ Etc.

MADRID · NOV 21-22 · 2014

Tips for defining steps (I)

Consider:

The goal of the stage.

The kind of feedback you need.

Sources.

Targets (environments.)

Build and package only in the Commit stage.

MADRID · NOV 21-22 · 2014

Tips for defining steps (II)

Consider:

Most times, deployment is present, but not

always.

(Automated) Smoke Testing should follow

any deployment.

Think about both automated and manual

steps.

MADRID · NOV 21-22 · 2014

Activity: defining steps

“We want to filter out anything producing static

analysis warnings.”

“We want to try exploratory testing.”

“We may use the same environment for load

testing and security testing.”

MADRID · NOV 21-22 · 2014

#6 Define

Automation & Tooling

MADRID · NOV 21-22 · 2014

Tips for step automation (I)

Automate everything.

Automate everywhere (for all the

environments.)

Preference for automation:

Fully automated steps.

Manually triggered automatic steps.

Manual steps.

MADRID · NOV 21-22 · 2014

Tips for step automation (II)

Build only once.

Version everything.

This includes the automations.

Have environment lockdown in mind.

MADRID · NOV 21-22 · 2014

Deployment automation

considerations

Deploy the same way to every environment.

The target environment should be a (implicit)

parameter for the automations.

Set up (tool-agnostic) one-click deployments.

Treat configuration tokens as parameters for

the automations.

Prepare for rollbacks.

MADRID · NOV 21-22 · 2014

Database deployment

considerations

Database deployment is not the same as

database development.

Decide about the deployment strategy:

Schema & Data compare.

Delta scripts (better for Continuous

Delivery.)

ORM tools (schema update, migrations, etc.)

MADRID · NOV 21-22 · 2014

Test automation considerations

Q2 tests are not necessarily run through the UI.

Smoke tests may be run through the UI.

Frequently, non-functional testing can be

automated.

Leave environments and data in a known state.

A few things can’t be automated (UAT & Q3

testing.)

MADRID · NOV 21-22 · 2014

What do we need to know

For each step to be automated:

Automation tool or technology.

Execution model.

Parameters (at least you’ll have the

configuration tokens.)

Source / target.

MADRID · NOV 21-22 · 2014

About execution models

Native OS tool.

Agent.

Remote execution.

MADRID · NOV 21-22 · 2014

Activity: defining automation &

tooling

“Production environment is in an isolated

network”

“Operations people won’t allow us to install

anything there”

MADRID · NOV 21-22 · 2014

Tooling

Environment provision

Puppet, Chef, Ansible

Virtualization tools (VM templates, etc.)

App deployment

Scripting (Unix shell, PowerShell, etc.)

Puppet, Chef, Ansible, VS Release Management

DB Deployment

RedGate, DBDeploy, etc.

Testing

Testing frameworks

UI automation frameworks

Non-functional testing tools

MADRID · NOV 21-22 · 2014

#7 Define Execution model,

monitoring & metrics

MADRID · NOV 21-22 · 2014

Continuous delivery flow model

Pipeline instances are created on each commit.

Any commit is a release candidate.

One-piece continuous flow model.

There is no way back. Any error makes the

release candidate to be discarded.

Fixes are treated as new release candidates.

They are run through the entire pipeline from

the beginning.

MADRID · NOV 21-22 · 2014

Other flow models

Pipeline instances are created as needed. A

release candidate might comprise several

commits.

Decide on the batch size. Larger batches may

be cheaper but limit feedback.

Errors might be fixed in the context of the stage

where they arise.

MADRID · NOV 21-22 · 2014

Monitoring the pipeline

Transparency.

Rely on a proper tool.

Set up alerts for key events.

Use a Cumulative Flow Diagram (CFD.)

Gather metrics and act on them.

MADRID · NOV 21-22 · 2014

Primary metrics

Cycle time.

Mean Time Between Failures (MTBF.)

Mean Time To Recover (MTTR.)

MADRID · NOV 21-22 · 2014

Secondary metrics

Test coverage.

Duplication of code.

Coupling.

Compilation warnings.

Code churn.

Build frequency

Etc.

MADRID · NOV 21-22 · 2014

Tooling

Out of the box metrics in the orchestration tool.

Ad-hoc reporting.

Sonar, FxCop, Ndepend, FindBugs…

MADRID · NOV 21-22 · 2014

#8 Plan for future enhancements

MADRID · NOV 21-22 · 2014

Examples – DevOps culture

Improve branching model if needed.

Debugging optimization, symbol servers.

Canary releases.

Blue/green deployments.

A/B Testing.

MADRID · NOV 21-22 · 2014

Examples – DevOps culture

Preventive profiling.

Telemetry, analytics, Application Performance

Monitoring (APM.)

Proactive resiliency enablement (Simian army

http://nflx.it/SPeTGj)

MADRID · NOV 21-22 · 2014

Tooling

Application profilers

APMNew Relic

App Insights

Raygun

Compuware APM

MADRID · NOV 21-22 · 2014

Artifact and metadata repositories

Binaries repository

TFS Build Drops folder

Reporting system (receives data from all stages) TFS Reporting – tracks Cycle

Time, MTTR & MTBF Custom TFS Reporting on Work Items

Commit stage TFS Build Definition, customized from

Default template

CodeGet dependencies (package

restore)Build

Commit testsContinuous integration

Code analysisBasic functional tests (manual)

Version artifactsName the pipeline instance

(set the build number)Update the artifact repository

Update the Symbol Server

Acceptance test stage TFS Build Definition, customized

from Lab template.MTM for Test Case

management

Change configuration (automated)

Deployment (automated)MSBuild for WP

Powershell for AzureAcceptance tests

(automated)MSTest, CodedUI (?)

Release stage TFS Build Definition, customized from Lab

template

Branch or merge to Release branch (for archiving)

Deployment (automated)MSBuild for WP

Powershell for AzureSmoke testing (automated)

CodedUI (?)Error detection and recovery

(resiliency enablement)SCOM-TFS Integration

Intellitrace in ProductionPreemptive AnalyticsCapacity test stage TFS Build

Definition, customized from Lab template

Deployment (automated)MSBuild for WP

Powershell for AzureSmoke testing (automated)

CodedUI (?)Performance & load testing

(automated)Visual Studio perf & load

tests

Environments (automatic provision, locked down so only automated deployments are allowed) Using Lab Manager SCVMM Environments

Development (isolated)

Local deploymentto WP and Azure emulators

Test Production

Iteration 6 Pipeline: further improvements

Manualtrigger

Automatictrigger

Version Control (code and configuration) TFS Version Control

Main branch (w/ feature

toggles)

Release branch (for archiving)

Staging

Deployment & testingtriggered by Test Agent

Pipeline monitoring

Custom solution

UAT stage

UAT (manual)Acceptance criteria from

TFS requirementsTFS Feedback tool

Manualtrigger

Deployment triggeredby Test Agent

Using deploymentfrom Capacity test stage

Deployment & testingtriggered by Test Agent

Security test stage

Security tests (manual)WACA

Any additional tools?

Automatictrigger

Using deploymentfrom Capacity test stage

Exploratory test stage TFS Build Definition, customized

from Lab template

Deployment (automated)MSBuild for WP

Powershell for AzureExploratory testing (manual)

MTM

Manualtrigger

ExploratoryTest

Deployment & testingtriggered by Test Agent

Artifact repository

NuGet Server

Symbol Server

TFS Symbol Server

MADRID · NOV 21-22 · 2014

#9 Inspect & Adapt

MADRID · NOV 21-22 · 2014

Continuously improve the pipeline

Component or architectural changes.

New skills in the team.

New resources, tools, environments.

Reserve time, and make the team accountable

for improvement.

MADRID · NOV 21-22 · 2014

Summary

1. Define components.

2. Identify sub-pipelines.

3. Define Stages & Orchestration.

4. Define Environments.

5. Define Steps.

6. Define Automation & Tooling.

7. Define execution model, monitoring and metrics.

8. Plan for future enhancements.

9. Inspect & adapt.

MADRID · NOV 21-22 · 2014

Thanks!

Jose Luis SoriaContinuous Improvement Manager at Ria Financial

jlsoriat@gmail.com - @jlsoriat

http://geeks.ms/blogs/jlsoria

Slides: http://www.slideshare.net/jlsoria

http://aka.ms/releasepipeline