Modern Release Engineering in a Nutshell
Why Researchers should Care
Bram Adams Shane McIntosh,MC IS
Back in 2009 …
On average, we release new code fifty
times a day.
Release engineering aims to …
4
Release engineering aims to …
4
codechange
Release engineering aims to …
4
codechange
Release engineering aims to …
4
codechange
Release engineering aims to …
4
codechange
FAST
Release Engineering
Release Engineering
Release Engineering
integrating code changes
Release Engineering
integrating code changes building/testing (CI)
Release Engineering
integrating code changes building/testing (CI)
deploying a new release
Release Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
Nowadays …
Time-boxed releases
7
Time-boxed releases
6 months
7
Time-boxed releases
6 weeks
6 months
7
Time-boxed releases
2 weeks (mobile)
6 weeks
6 months
7
Time-boxed releases
twice/day (web)
2 weeks (mobile)
6 weeks
6 months
7
Time-boxed releases
twice/day (web)
daily
2 weeks (mobile)
6 weeks
6 months
7
http://www.informit.com/articles/article.aspx?p=1833567
Jez Humble
http://www.informit.com/articles/article.aspx?p=1833567
reduce the risk of releasing software
if it hurts, do it more frequently, and
bring the pain forward
Jez Humble
• How quickly can we ship a chemspill release?
• 4-6 weeks 11 hours
• How long to ship a “new feature” release?
• 12-18 months 6 weeks
• How many active code lines?
• 1 1/2 42
• How many checkins per day?
• ~15 per day 325 per day
Before & After
Thursday, June 6, 13
http://oduinn.com/images/2013/blog_2013_RelEngAsForceMultiplier.pdf 9
• How quickly can we ship a chemspill release?
• 4-6 weeks 11 hours
• How long to ship a “new feature” release?
• 12-18 months 6 weeks
• How many active code lines?
• 1 1/2 42
• How many checkins per day?
• ~15 per day 325 per day
Before & After
Thursday, June 6, 13
http://oduinn.com/images/2013/blog_2013_RelEngAsForceMultiplier.pdf 9
• How quickly can we ship a chemspill release?
• 4-6 weeks 11 hours
• How long to ship a “new feature” release?
• 12-18 months 6 weeks
• How many active code lines?
• 1 1/2 42
• How many checkins per day?
• ~15 per day 325 per day
Before & After
Thursday, June 6, 13
http://oduinn.com/images/2013/blog_2013_RelEngAsForceMultiplier.pdf 9
https://xebialabs.com/periodic-table-of-devops-tools/ & https://xebialabs.com/the-ultimate-devops-tool-chest/
11
industry Release engineering innovation
led by industry!
11
industry
academia
Release engineering innovation
led by industry!
12
What can you do for release engineering?
RELENG: International Workshop on Release Engineering
13
RELENG: International Workshop on Release Engineering
13
230 participants3 editions
dozens of industry
& academic talks
RELENG: International Workshop on Release Engineering
13
230 participants3 editions
dozens of industry
& academic talks
RELENG: International Workshop on Release Engineering
13
230 participants3 editions
next RELENG: in Fall 2016
http://releng.polymtl.ca
dozens of industry
& academic talks
How to predict integration effort?
14
How to predict integration effort?
14
git repo
How to predict integration effort?
14
git repo
myapp v.1
How to predict integration effort?
14
git repo
myapp v.1
?
How to predict integration effort?
14
git repo
myapp v.1
How to predict integration effort?
14
feature branch 1git repo
myapp v.1
How to predict integration effort?
14
feature branch 1git repo
myapp v.1
How to predict integration effort?
14
feature branch 1git repo
myapp v.1
master branch
How to predict integration effort?
14
feature branch 1git repo
myapp v.1
master branch
How to predict integration effort?
14
feature branch 1git repo
MERGE
myapp v.1
master branch
How to predict integration effort?
14
feature branch 1git repo
MERGE
myapp v.1
master branch
How to predict integration effort?
14
feature branch 1git repo
MERGE
myapp v.1
master branch
How to predict integration effort?
14
feature branch 1git repo
release branch
MERGE
myapp v.1
master branch
How to predict integration effort?
14
feature branch 1git repo
release branch
MERGE
myapp v.1
master branch
How to predict integration effort?
14
feature branch 1git repo
release branch
MERGE
myapp v.1
feature branch 2
master branch
How to predict integration effort?
14
feature branch 1git repo
release branch
MERGE CONFLICT?
myapp v.1
feature branch 2
master branch
How to predict integration effort?
14
3rd party dependency
feature branch 1git repo
release branch
MERGE CONFLICT?
myapp v.1
feature branch 2
master branch
How to predict integration effort?
14
3rd party dependency
feature branch 1git repo
release branch
MERGE CONFLICT?
myapp v.1
feature branch 2
master branch
How to predict integration effort?
14
3rd party dependency
feature branch 1git repo
release branch
MERGE CONFLICT?
myapp v.1
feature branch 2sy
nc
15
https://xkcd.com/303/
user acceptance/system testsperformance tests
(selection of) unit tests
all unit testsintegration tests
time to run (hours)
full compilation
incremental compilation
`
Why do CI builds take so long?
16
user acceptance/system testsperformance tests
(selection of) unit tests
all unit testsintegration tests
time to run (hours)
full compilation
incremental compilation
`
local developer build
Why do CI builds take so long?
16
user acceptance/system testsperformance tests
(selection of) unit tests
all unit testsintegration tests
time to run (hours)
full compilation
incremental compilation
`
local developer build
Why do CI builds take so long?
16
user acceptance/system testsperformance tests
(selection of) unit tests
all unit testsintegration tests
time to run (hours)
full compilation
incremental compilation
` CI build of merged change
local developer build
Why do CI builds take so long?
16
user acceptance/system testsperformance tests
(selection of) unit tests
all unit testsintegration tests
time to run (hours)
full compilation
incremental compilation
CI build of merged change
local developer build
Why do CI builds take so long?
16
user acceptance/system testsperformance tests
(selection of) unit tests
all unit testsintegration tests
time to run (hours)
full compilation
incremental compilation
closer to release
CI build of merged change
local developer build
Why do CI builds take so long?
16
user acceptance/system testsperformance tests
(selection of) unit tests
all unit testsintegration tests
time to run (hours)
full compilation
incremental compilation
closer to release
CI build of merged change
local developer build
Why do CI builds take so long?
16
Even worse …
17
Even worse …
17
build machinery run
on each commit!
Even worse …
17
build machinery run
on each commit!
Even worse …
17
build machinery run
on each commit!different feature configurations
Even worse …
17
build machinery run
on each commit!
not just build and test, also code quality builds, nightly builds, etc.
different feature configurations
Even worse …
17
build machinery run
on each commit!
not just build and test, also code quality builds, nightly builds, etc.
not all builds succeed
different feature configurations
What are deployment best practices?
18
What are deployment best practices?
18
v.1 v.1v.1
What are deployment best practices?
18
v.1v.2 v.1
What are deployment best practices?
18
v.2 v.1v.2
What are deployment best practices?
18
v.2 v.1v.1
What are deployment best practices?
18
v.1v.1v.1
Infrastructure code
19
# Install PostgreSQL server and client include_recipe "postgresql::server" include_recipe "postgresql::client"
# Make postgresql_database resource available include_recipe "database::postgresql"
# Create database for Rails app db = node["practicingruby"]["database"] postgresql_database db["name"] do connection( :host => db["host"], :port => node["postgresql"]["config"]["port"], :username => db["username"], :password => db["password"], ) end
Infrastructure code
19
# Install PostgreSQL server and client include_recipe "postgresql::server" include_recipe "postgresql::client"
# Make postgresql_database resource available include_recipe "database::postgresql"
# Create database for Rails app db = node["practicingruby"]["database"] postgresql_database db["name"] do connection( :host => db["host"], :port => node["postgresql"]["config"]["port"], :username => db["username"], :password => db["password"], ) end
Infrastructure code
19
# Install PostgreSQL server and client include_recipe "postgresql::server" include_recipe "postgresql::client"
# Make postgresql_database resource available include_recipe "database::postgresql"
# Create database for Rails app db = node["practicingruby"]["database"] postgresql_database db["name"] do connection( :host => db["host"], :port => node["postgresql"]["config"]["port"], :username => db["username"], :password => db["password"], ) end
Infrastructure code smells?
19
Is the release in good
shape?
Was the release a success?
Is the release in good
shape?
Was the release a success?
Is the release in good
shape?
The release is botched, how can we
roll back?
Was the release a success?
What’s the optimal cycle time for us?
Is the release in good
shape?
The release is botched, how can we
roll back?
Chuck Rossi
21
Continuous delivery for mobile apps is a serious
challenge!
Chuck Rossi
21
What can you do for release engineering?
22
What can you do for release engineering?
22
What can release
engineering do for you?
When noble intentions meet reality
23
Harmful assumptions about release pipelines that can impact predictive modelling
24
Harmful assumptions about release pipelines that can impact predictive modelling
1. All releases are equal
24
25
Look at Project X! It had a crazy number of
bugs in the typical six-month post-release window!
25
Look at Project X! It had a crazy number of
bugs in the typical six-month post-release window!
Well, Project X releases every 6 weeks, so
you’re counting several releases…
25
Release cycles vary amongpopular studied systems
Fig. 3: Integration delay is computed by counting the re-leases that occur between when an issue status changes toRESOLVED-FIXED and the the date of the release note thatlists that issue.
Fig. 4: Delays in days between releases of ArgoUML, Eclipse,and Firefox. The number shown over each boxplot is themedian interval
project. For example, Figure 4, shows that Firefox releasesconsistently every 42 days (six weeks), whereas the timesbetween ArgoUML releases vary from 50 to 220 days. Theconsistency of Firefox releases may lead to more delayedissues, since they rigidly adhere to a six-week release scheduledespite accumulating issues that could not be integrated.
34% to 60% of addressed issues in the traditionalrelease cycle systems were delayed by one or more releases.Figure 2 shows that 98% of the addressed issues in Firefoxare delayed by one or more releases. Firefox is expected tohave delayed issues due its rapid release cycles. However,98% is still a considerably large percentage. Furthermore, evenfor the systems that adopt a more traditional release cycle,34% (ArgoUML) to 60% (Eclipse) of the addressed issues aredelayed by one or more releases. This result indicates that eventhough an issue is addressed, integration could be delayed byone or more releases.
Many delayed issues were addressed well before releasesfrom which they were omitted. Addressed issues could bedelayed from integration because they were addressed latein the release cycle, e.g., one day or one week before theupcoming release date. In order to compare the rapid andtraditional release cycles regarding whether delayed issuesare addressed late in the release schedule, we computed the
Fig. 5: Distribution of days between when an issue wasaddressed and the next missed release divided by the releasewindow time.
Addressing Stage metric (AS) for each issue. The AS metricis calculated using the following equation: days to next release
release window
,where days to next release is the number of days when an issueis addressed before the next release (e.g., the time between t3to t4 in Figure 3), and the release window is the time in daysbetween the next upcoming release and the respective previousrelease (e.g., t4 to t2). An AS value close to 1 means that anissue was addressed too close to the next release, whereas avalue close to 0 means that an issue was addressed at thebeginning of a release cycle. Figure 5 shows the distributionof the AS metric for each project. The smallest AS medianis observed for Eclipse, which is 0.45. For ArgoUML andFirefox, the median is 0.52 and 0.53, respectively. The ASmedians are roughly in the middle of the release. Moreover,the boxes extend to cover between 0.25 and 0.75. The resultsuggests that, in the studied projects, delayed issues are usuallyaddressed 1
4 to 34 of the way through a release. Hence, it is
unlikely that most addressed issues miss the next release solelybecause they were addressed too close to an upcoming releasedate.
The integration of 34% to 60% of the addressed issuesin the traditionally releasing systems and 98% in therapidly releasing system were delayed by one or morereleases. Furthermore, we find that many delayed issueswere addressed well before releases from which they wereomitted from.
RQ2: Can we accurately predict when an addressed issuewill be integrated?Motivation. Several studies proposed approaches to inves-tigate the time required to address an issue [2–7]. Thesestudies could help to estimate when an issue will be addressed.However, we find that integration delays when an addressedissue will be delivered to users. Even though several issues areaddressed well before the next release date, their integration isdelayed. For users and contributors, however, knowing the re-lease in which an addressed issue will be integrated is of great
Day
s be
twee
n re
leas
es
An Empirical Study of Delays in the Integration of Addressed IssuesD. A. da Costa et al.
[ICSME 2014]
Release cycles can evenvary within systems!
1.02.02.53.03.54.05.06.07.08.09.0
10.00 200 400 600 800
Fire
fox
rele
ase
Days since prior release 27
The rapid release cycle of modern software systems
28
The rapid release cycle of modern software systems
Often release several timesin one day!
28
1. All releases are equal
Harmful assumptions about release pipelines that can impact predictive modelling
29
1. All releases are equal2. All branches are equal
Harmful assumptions about release pipelines that can impact predictive modelling
29
30
Weird… The size of this project fluctuates between
50k and 45k lines!
30
Weird… The size of this project fluctuates between
50k and 45k lines!
Hmm, did you select the relevant branch? Several
are developed in parallel!
30
An example of defect prediction
Feature developmentDefect repairingMerge
Commit types
31
An example of defect prediction
Feature developmentDefect repairingMerge
Commit types
31
An example of defect prediction
v1.0
Feature developmentDefect repairingMerge
Commit types
31
An example of defect prediction
v1.0
Feature developmentDefect repairingMerge
Commit types
Pre-release
31
An example of defect prediction
v1.0
Feature developmentDefect repairingMerge
Commit types
Pre-release
Post-release
31
An example of defect prediction
v1.0
Feature developmentDefect repairingMerge
Commit types
Pre-release
Post-release
31
Handling the intricacies of a multi-branch release pipeline
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Handling the intricacies of a multi-branch release pipeline
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Handling the intricacies of a multi-branch release pipeline
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Handling the intricacies of a multi-branch release pipeline
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Handling the intricacies of a multi-branch release pipeline
v1.0
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Handling the intricacies of a multi-branch release pipeline
v1.0
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Handling the intricacies of a multi-branch release pipeline
v1.0
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Handling the intricacies of a multi-branch release pipeline
v1.0
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Handling the intricacies of a multi-branch release pipeline
v1.0
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Handling the intricacies of a multi-branch release pipeline
v1.0
Dev
Stable
Feature developmentDefect repairingMerge
Commit types
32
Branching can get really complex…
33
1. All releases are equal2. All branches are equal
Harmful assumptions about release pipelines that can impact predictive modelling
34
1. All releases are equal2. All branches are equal
3. All files are equal
Harmful assumptions about release pipelines that can impact predictive modelling
34
35
This popular web browser has an enormous codebase!
>30M lines!
35
This popular web browser has an enormous codebase!
>30M lines!
Yes, but the codebase contains several systems! The build configuration
decides which one is produced!
35
36
Many files are conditionally included in deliverables
Tracing Software Build Processes to Uncover License Compliance
InconsistenciesS. van der Berg et al.
[ASE 2014]
Aterm
Opkg
Bash
CUPS
Xalan
OpenSSL
FFmpeg
% Excluded Files
0% 20% 40% 60% 80%
36
Many files are conditionally included in deliverables
Fixes in these files may have a smaller impact (if any) on customers
Tracing Software Build Processes to Uncover License Compliance
InconsistenciesS. van der Berg et al.
[ASE 2014]
Aterm
Opkg
Bash
CUPS
Xalan
OpenSSL
FFmpeg
% Excluded Files
0% 20% 40% 60% 80%
Understanding conditionally included files using the build system
Design recovery and maintenance of build systemsB. Adams et al.
[ICSM 2007]
37
1. All releases are equal,2. All branches are equal,
Harmful assumptions about release pipelines that can impact predictive modelling
3. All files are equal
38
but some are more equal than others
1. All releases are equal,2. All branches are equal,
Harmful assumptions about release pipelines that can impact predictive modelling
3. All files are equal
38
My nightmare
Amassing and indexing a large sample of version control systems
Audris Mockus [MSR 2009]
Boa: a language and infrastructure for analyzing ultra-large-scale software repositories
R. Dyer et al. [ICSE 2013]
The GHTorrent Dataset and Tool Suite
G. Gousios [MSR 2013]
My nightmare
Amassing and indexing a large sample of version control systems
Audris Mockus [MSR 2009]
Boa: a language and infrastructure for analyzing ultra-large-scale software repositories
R. Dyer et al. [ICSE 2013]
The GHTorrent Dataset and Tool Suite
G. Gousios [MSR 2013]
We collect all of the data in the world, but it’s meaningless without context!
Release Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
Release Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
Release Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new release
http://www.informit.com/articles/article.aspx?p=1833567
reduce the risk of releasing software
if it hurts, do it more frequently, and
bring the pain forward
Jez Humble
Release Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new releasehttp://www.informit.com/articles/article.aspx?p=1833567
reduce the risk of releasing software
if it hurts, do it more frequently, and
bring the pain forward
Jez Humble
Release Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new releasehttp://www.informit.com/articles/article.aspx?p=1833567
reduce the risk of releasing software
if it hurts, do it more frequently, and
bring the pain forward
Jez Humble
master branch
How to Predict Integration Effort?
14
3rd party dependency
feature branch 1git repo
release branch
MERGE CONFLICT?
myapp v.1
feature branch 2sy
nc
Release Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new releasehttp://www.informit.com/articles/article.aspx?p=1833567
reduce the risk of releasing software
if it hurts, do it more frequently, and
bring the pain forward
Jez Humble
master branch
How to Predict Integration Effort?
14
3rd party dependency
feature branch 1git repo
release branch
MERGE CONFLICT?
myapp v.1
feature branch 2
sync
Release Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new releasehttp://www.informit.com/articles/article.aspx?p=1833567
reduce the risk of releasing software
if it hurts, do it more frequently, and
bring the pain forward
Jez Humble
master branch
How to Predict Integration Effort?
14
3rd party dependency
feature branch 1git repo
release branch
MERGE CONFLICT?
myapp v.1
feature branch 2
syncbut some are more equal than others
1. All releases are equal,2. All branches are equal,
Harmful assumptions about release pipelines that can impact predictive modelling
3. All files are equal
Release Engineering
integrating code changes building/testing (CI)
releasing to the user deploying a new releasehttp://www.informit.com/articles/article.aspx?p=1833567
reduce the risk of releasing software
if it hurts, do it more frequently, and
bring the pain forward
Jez Humble
master branch
How to Predict Integration Effort?
14
3rd party dependency
feature branch 1git repo
release branch
MERGE CONFLICT?
myapp v.1
feature branch 2
sync
but some are more equal than others
1. All releases are equal,2. All branches are equal,
Harmful assumptions about release pipelines that can impact predictive modelling
3. All files are equal