Modern Release Engineering in a Nutshell

Why Researchers should Care

Bram Adams Shane McIntosh,MC IS

Back in 2009 …

On average, we release new code fifty

times a day.

Release engineering aims to …

codechange

Release Engineering

integrating code changes

Release Engineering

integrating code changes building/testing (CI)

Release Engineering

deploying a new release

Release Engineering

releasing to the user deploying a new release

Nowadays …

Time-boxed releases

6 months

Time-boxed releases

6 weeks

6 months

Time-boxed releases

2 weeks (mobile)

6 weeks

6 months

Time-boxed releases

twice/day (web)

2 weeks (mobile)

6 weeks

6 months

Time-boxed releases

twice/day (web)

2 weeks (mobile)

6 weeks

6 months

http://www.informit.com/articles/article.aspx?p=1833567

Jez Humble

reduce the risk of releasing software

if it hurts, do it more frequently, and

bring the pain forward

Jez Humble

• How quickly can we ship a chemspill release?

• 4-6 weeks 11 hours

• How long to ship a “new feature” release?

• 12-18 months 6 weeks

• How many active code lines?

• 1 1/2 42

• How many checkins per day?

• ~15 per day 325 per day

Before & After

Thursday, June 6, 13

http://oduinn.com/images/2013/blog_2013_RelEngAsForceMultiplier.pdf 9

• 1 1/2 42

Before & After

• 1 1/2 42

Before & After

https://xebialabs.com/periodic-table-of-devops-tools/ & https://xebialabs.com/the-ultimate-devops-tool-chest/

industry Release engineering innovation

led by industry!

industry

academia

Release engineering innovation

led by industry!

What can you do for release engineering?

RELENG: International Workshop on Release Engineering

230 participants3 editions

dozens of industry

& academic talks

dozens of industry

& academic talks

next RELENG: in Fall 2016

http://releng.polymtl.ca

dozens of industry

& academic talks

How to predict integration effort?

git repo

myapp v.1

git repo

myapp v.1

git repo

myapp v.1

feature branch 1git repo

myapp v.1

master branch

myapp v.1

master branch

myapp v.1

master branch

myapp v.1

master branch

myapp v.1

master branch

release branch

myapp v.1

master branch

release branch

myapp v.1

master branch

release branch

myapp v.1

feature branch 2

master branch

release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

master branch

3rd party dependency

release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

master branch

release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

master branch

release branch

MERGE CONFLICT?

myapp v.1

feature branch 2sy

https://xkcd.com/303/

user acceptance/system testsperformance tests

(selection of) unit tests

all unit testsintegration tests

time to run (hours)

full compilation

incremental compilation

Why do CI builds take so long?

time to run (hours)

full compilation

local developer build

time to run (hours)

full compilation

time to run (hours)

full compilation

` CI build of merged change

time to run (hours)

full compilation

CI build of merged change

time to run (hours)

full compilation

closer to release

time to run (hours)

full compilation

closer to release

Even worse …

build machinery run

on each commit!

Even worse …

build machinery run

on each commit!

Even worse …

build machinery run

on each commit!different feature configurations

Even worse …

build machinery run

on each commit!

not just build and test, also code quality builds, nightly builds, etc.

different feature configurations

Even worse …

build machinery run

on each commit!

not just build and test, also code quality builds, nightly builds, etc.

not all builds succeed

different feature configurations

What are deployment best practices?

v.1 v.1v.1

v.1v.2 v.1

v.2 v.1v.2

v.2 v.1v.1

v.1v.1v.1

Infrastructure code

# Install PostgreSQL server and client include_recipe "postgresql::server" include_recipe "postgresql::client"

# Make postgresql_database resource available include_recipe "database::postgresql"

# Create database for Rails app db = node["practicingruby"]["database"] postgresql_database db["name"] do connection( :host => db["host"], :port => node["postgresql"]["config"]["port"], :username => db["username"], :password => db["password"], ) end

Infrastructure code

Infrastructure code smells?

Is the release in good

shape?

Was the release a success?

shape?

The release is botched, how can we

roll back?

What’s the optimal cycle time for us?

shape?

The release is botched, how can we

roll back?

Chuck Rossi

Continuous delivery for mobile apps is a serious

challenge!

Chuck Rossi

What can you do for release engineering?

What can release

engineering do for you?

When noble intentions meet reality

Harmful assumptions about release pipelines that can impact predictive modelling

1. All releases are equal

Look at Project X! It had a crazy number of

bugs in the typical six-month post-release window!

Look at Project X! It had a crazy number of

bugs in the typical six-month post-release window!

Well, Project X releases every 6 weeks, so

you’re counting several releases…

Release cycles vary amongpopular studied systems

Fig. 3: Integration delay is computed by counting the re-leases that occur between when an issue status changes toRESOLVED-FIXED and the the date of the release note thatlists that issue.

Fig. 4: Delays in days between releases of ArgoUML, Eclipse,and Firefox. The number shown over each boxplot is themedian interval

project. For example, Figure 4, shows that Firefox releasesconsistently every 42 days (six weeks), whereas the timesbetween ArgoUML releases vary from 50 to 220 days. Theconsistency of Firefox releases may lead to more delayedissues, since they rigidly adhere to a six-week release scheduledespite accumulating issues that could not be integrated.

34% to 60% of addressed issues in the traditionalrelease cycle systems were delayed by one or more releases.Figure 2 shows that 98% of the addressed issues in Firefoxare delayed by one or more releases. Firefox is expected tohave delayed issues due its rapid release cycles. However,98% is still a considerably large percentage. Furthermore, evenfor the systems that adopt a more traditional release cycle,34% (ArgoUML) to 60% (Eclipse) of the addressed issues aredelayed by one or more releases. This result indicates that eventhough an issue is addressed, integration could be delayed byone or more releases.

Many delayed issues were addressed well before releasesfrom which they were omitted. Addressed issues could bedelayed from integration because they were addressed latein the release cycle, e.g., one day or one week before theupcoming release date. In order to compare the rapid andtraditional release cycles regarding whether delayed issuesare addressed late in the release schedule, we computed the

Fig. 5: Distribution of days between when an issue wasaddressed and the next missed release divided by the releasewindow time.

Addressing Stage metric (AS) for each issue. The AS metricis calculated using the following equation: days to next release

release window

,where days to next release is the number of days when an issueis addressed before the next release (e.g., the time between t3to t4 in Figure 3), and the release window is the time in daysbetween the next upcoming release and the respective previousrelease (e.g., t4 to t2). An AS value close to 1 means that anissue was addressed too close to the next release, whereas avalue close to 0 means that an issue was addressed at thebeginning of a release cycle. Figure 5 shows the distributionof the AS metric for each project. The smallest AS medianis observed for Eclipse, which is 0.45. For ArgoUML andFirefox, the median is 0.52 and 0.53, respectively. The ASmedians are roughly in the middle of the release. Moreover,the boxes extend to cover between 0.25 and 0.75. The resultsuggests that, in the studied projects, delayed issues are usuallyaddressed 1

4 to 34 of the way through a release. Hence, it is

unlikely that most addressed issues miss the next release solelybecause they were addressed too close to an upcoming releasedate.

The integration of 34% to 60% of the addressed issuesin the traditionally releasing systems and 98% in therapidly releasing system were delayed by one or morereleases. Furthermore, we find that many delayed issueswere addressed well before releases from which they wereomitted from.

RQ2: Can we accurately predict when an addressed issuewill be integrated?Motivation. Several studies proposed approaches to inves-tigate the time required to address an issue [2–7]. Thesestudies could help to estimate when an issue will be addressed.However, we find that integration delays when an addressedissue will be delivered to users. Even though several issues areaddressed well before the next release date, their integration isdelayed. For users and contributors, however, knowing the re-lease in which an addressed issue will be integrated is of great

An Empirical Study of Delays in the Integration of Addressed IssuesD. A. da Costa et al.

[ICSME 2014]

Release cycles can evenvary within systems!

1.02.02.53.03.54.05.06.07.08.09.0

10.00 200 400 600 800

Days since prior release 27

The rapid release cycle of modern software systems

Often release several timesin one day!

1. All releases are equal

1. All releases are equal2. All branches are equal

Weird… The size of this project fluctuates between

50k and 45k lines!

Weird… The size of this project fluctuates between

50k and 45k lines!

Hmm, did you select the relevant branch? Several

are developed in parallel!

An example of defect prediction

Feature developmentDefect repairingMerge

Commit types

Pre-release

Commit types

Pre-release

Post-release

Commit types

Pre-release

Post-release

Handling the intricacies of a multi-branch release pipeline

Stable

Commit types

Stable

Commit types

Stable

Commit types

Stable

Commit types

Stable

Commit types

Stable

Commit types

Stable

Commit types

Stable

Commit types

Stable

Commit types

Stable

Commit types

Branching can get really complex…

3. All files are equal

This popular web browser has an enormous codebase!

>30M lines!

This popular web browser has an enormous codebase!

>30M lines!

Yes, but the codebase contains several systems! The build configuration

decides which one is produced!

Many files are conditionally included in deliverables

Tracing Software Build Processes to Uncover License Compliance

InconsistenciesS. van der Berg et al.

[ASE 2014]

OpenSSL

FFmpeg

% Excluded Files

0% 20% 40% 60% 80%

Many files are conditionally included in deliverables

Fixes in these files may have a smaller impact (if any) on customers

Tracing Software Build Processes to Uncover License Compliance

InconsistenciesS. van der Berg et al.

[ASE 2014]

OpenSSL

FFmpeg

% Excluded Files

0% 20% 40% 60% 80%

Understanding conditionally included files using the build system

Design recovery and maintenance of build systemsB. Adams et al.

[ICSM 2007]

1. All releases are equal,2. All branches are equal,

but some are more equal than others

My nightmare

Amassing and indexing a large sample of version control systems

Audris Mockus [MSR 2009]

Boa: a language and infrastructure for analyzing ultra-large-scale software repositories

R. Dyer et al. [ICSE 2013]

The GHTorrent Dataset and Tool Suite

G. Gousios [MSR 2013]

My nightmare

Amassing and indexing a large sample of version control systems

Audris Mockus [MSR 2009]

Boa: a language and infrastructure for analyzing ultra-large-scale software repositories

R. Dyer et al. [ICSE 2013]

The GHTorrent Dataset and Tool Suite

G. Gousios [MSR 2013]

We collect all of the data in the world, but it’s meaningless without context!

Release Engineering

Jez Humble

Release Engineering

releasing to the user deploying a new releasehttp://www.informit.com/articles/article.aspx?p=1833567

Jez Humble

Release Engineering

Jez Humble

master branch

How to Predict Integration Effort?

release branch

MERGE CONFLICT?

myapp v.1

feature branch 2sy

Release Engineering

Jez Humble

master branch

release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

Release Engineering

Jez Humble

master branch

release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

syncbut some are more equal than others

Release Engineering

Jez Humble

master branch

release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

but some are more equal than others

Modern Release Engineering in a Nutshell - Why Researchers should Care!

Engineering

ITIL in Nutshell

Maven nutshell

Ofbiz Nutshell

Nutshell All 19042010

In a nutshell . .

MBA Fundamentals Program for Researchers€¦ · topics - in a nutshell Great opportunity to network with other young researchers Stand out from the crowd The MBA Fundamentals Program,

Modern Particle Physics Event Generation with WHIZARDreuter/downloads/2014_ACAT.pdf · Modern Particle Physics Event Generation with ... WHIZARD in a Nutshell WHIZARD is a universal

Two Over One in a Nutshell: Modern Concepts for Winning ... · PDF fileTwo Over One in a Nutshell: Modern Concepts for Winning Bridge, 2003, Dee Berry, 0973164506, 9780973164503, PDI

PezzeYoung Ch01 Nutshell

IslanDoRa In Nutshell

Migration In a Nutshell Migration In a Nutshell Migration In a Nutshell D.S. Macpherson

Nutch in Nutshell

NUTSHELL - nharriscounty-tx.aauw.net

Program and Abstracts - University of Floridareg.conferences.dce.ufl.edu/docs/PDSTM/PDSTMProgram.pdf · properties, materials researchers applying modern fabrication methods and device

NUTSHELL EXCEL

Ias Nutshell

Global Nutshell

BCLP Nutshell 201008

In a nutshell …

Networking in Nutshell