Scaling Continuous Integration Practices to Teams with Parallel Development

Scaling Continuous Integration Practices to Teams with Parallel

Development

January 19, 2012

Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.

Introduction to Today’s Speakers

Chris Lucca, Technical Evangelist, AccuRev

Chris' experience dates back to his days as a build / release specialist and software engineer for IBM-Rational. He has also filled the roles of developer, release engineer and SCM consultant for numerous organizations. In addition Chris is a certified scrum master and a frequent speaker at the AccuRev seminar series.

Now Chris brings the experience and knowledge he has acquired to help keep others in the industry up-to-date on the latest developments, and how they can impact them directly.


Introduction to Today’s Speakers

Eric Minick, Lead Consultant, UrbanCode

Eric Minick is a lead consultant at UrbanCode where he helps customers get the most out of their build, deploy and release processes. He has 8 years of automation experience throughout the application life-cycle in roles as a developer, test automation engineer, and support engineer.

Eric has been at the forefront of continuous integration for 5+ years and has worked on all three generations of UrbanCode’s Anthill. Keep up with Eric'’s latest insights on UrbanCode’s blog.


Agenda

• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A


Agenda



Jan Mar May JulFeb Apr Jun Aug Sep

Specify Design Code IntegrateWrite tests

DocPlan Test

Traditional Development


Traditionally Devs Work in Isolation


Too Much Isolation Leads to Integration Hell


CI: Minimizing Integration Pain

• Integrating frequently is less painful than infrequently• But, If we commit frequently blindly, we have a

broken code base

• Implied condition: each integration will not decrease the code quality• This is the key to CI • The rest is implementation details


CI: The Simple Implementation

• How do we determine code quality? We test it

• How do we test code? We build it and run tests against it

• This is how we get from CI to CI servers and software builds


So Simple but Very Important

• A build & unit test is a rudimentary check• Gives important feedback• The heartbeat of the dev organization


CI at Scale?

• With too many committers into the same code line, the likelihood something is broken right now is high

• We stop caring about failures

• The feedback is meaningless, and productivity suffers. CI becomes “constantly broken”


Agenda



Reasons People Work In Parallel

It’s often a business reason…not for optimization

• Maintenance Release – Patches

• Customer Specials - Just for one customer

• Development branches – Developer spikes for

experiments that will integrate at an unknown time

• Code Configurations – Different configurations of code

based on environment, such as UAT, QA

• Special Content – Images, binaries


Technical Debt Grows

As code lines diverge, integration becomes a bigger eventual problem. … But we don’t always want to merge immediately

• Maintenance Release – Don’t want new features

• Customer Specials – One directional integration

• Development branches – Dev enjoys a stable playground

• Special Content – Reused at different cadence than code


Agenda



Finding the Isolation / Integration Balance


Integration Is Easy With a Few People

Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4


It’s hard at scale

Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4


Little Hiccup

• Race condition: I could update to the latest code before the build/test notifies me it doesn’t compile

• Cost associated with this is huge

• Leads to “interesting solutions”

• Preflight• Build branches• Build Acceleration• Build Avoidance• Componentization


Cost of Integration

Per week:• # of check-ins per developer?• # of developers?• % of check-ins that break things?• # of problems encountered per developer?• Length of time per problem encountered?

200 devs, 2 commits each, 1% are bad = 4 breaks/day

If each breakage cuts productivity 50% for 15 minutes, we lose 100 dev hours / day (@$50/hr = $5000/day)


CI

CI

CI

Team A

Team B

ws

ws

ws

Multi-Stage Continuous Integration

Minutes Hours

int

Daily

ws

ws

ws

• Continuous

Integration

• Continuous Integration

• Automated Tests

• Unit Tests

• Binary Deploy

• Database Deploy

• Web Deploy

• Database Deploy

• Web Deploy

• Binary Deploy

• Customer Sign-

Off

• Database Deploy

• Web Deploy

• Binary Deploy

• IT Sign-Off

• Rollback (if needed)

x60x30x3x1

x1x4x10x100s

x1x2x5x10

0:102:003:005:00

• Team Specific CI

• Continuous

Integration

• Automated Tests

• Unit Tests

• Binary Deploy

• Database Deploy

• Web Deploy

• Database

Deploy

• Web Deploy

• Binary Deploy

• Customer Sign-

Off

• Database Deploy

• Web Deploy

• Binary Deploy

• IT Sign-Off

• Rollback (if needed)

Automated Merges


A Feature Branch Example

Urbancode’s Website Today… Next (not ready for prime time)

We need two build configurations, deploy configs, test configs, and test environments


Building and Testing Multiple Code Lines

• Build and test every active code line (continuously)

• Automatically run code scans, unit tests and functional tests


CI for a New Code Line Must Be Cheap

• Build and deployment configuration• Maximize reuse• Potentially create dynamically

• Tests• Core regression suites shared across code lines• Tests for new functionality shared as code is shared

• Test Environments• No excuse for a lack of test environments today• Deploy “what’s in production” as a baseline


Deploy Time Integration Is the Same

• In the team’s environment:• Continuously deploy and execute functional tests• Regularly (nightly / weekly) deploy other

components that have passed integration tests

• When new features work well, deliver to the integration test environment

• Releases to Stage & Prod match the full contents of an earlier test environment


In Closing

• Compile is the first test

• Unit test, deploy, automation can come after

• Get Feedback as Fast as possible

• Different teams need different types of builds

• Separate code configurations based on process

• It’s about people, but tooling helps


Best Practices For:Agile

Parallel

Global

Compliance

Change ManagementVisual Process

Automation

What Is AccuRev?

Much more than Version Control…AccuRev enables Software Process Automation

VersionControlSCM


Urbancode Products

uBuild (AnthillPro)• Continuous integration and delivery

uDeploy• Deployment and release management

uCloud (soon)• Environment provisioning from your private cloud


Q&A

Call: +1 216 858-9000Email: [email protected]: www.urbancode.comLearn: blogs.urbancode.com

Call: 1-800-383-8170 or +1-781-861-8700Email: [email protected]: www.accurev.comLearn: blog.accurev.com

http://www.urbancode.com/

http://blogs.urbancodecom/

http://www.accurev.com/

http://blog.accurev.com/

Technology

Scaling Continuous Integration Practices to Teams with Parallel Development