Scaling Continuous Integration Practices to Teams with Parallel
Development
January 19, 2012
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Introduction to Today’s Speakers
Chris Lucca, Technical Evangelist, AccuRev
Chris' experience dates back to his days as a build / release specialist and software engineer for IBM-Rational. He has also filled the roles of developer, release engineer and SCM consultant for numerous organizations. In addition Chris is a certified scrum master and a frequent speaker at the AccuRev seminar series.
Now Chris brings the experience and knowledge he has acquired to help keep others in the industry up-to-date on the latest developments, and how they can impact them directly.
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Introduction to Today’s Speakers
Eric Minick, Lead Consultant, UrbanCode
Eric Minick is a lead consultant at UrbanCode where he helps customers get the most out of their build, deploy and release processes. He has 8 years of automation experience throughout the application life-cycle in roles as a developer, test automation engineer, and support engineer.
Eric has been at the forefront of continuous integration for 5+ years and has worked on all three generations of UrbanCode’s Anthill. Keep up with Eric'’s latest insights on UrbanCode’s blog.
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Agenda
• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Agenda
• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Jan Mar May JulFeb Apr Jun Aug Sep
Specify Design Code IntegrateWrite tests
DocPlan Test
Traditional Development
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Traditionally Devs Work in Isolation
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Too Much Isolation Leads to Integration Hell
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI: Minimizing Integration Pain
• Integrating frequently is less painful than infrequently• But, If we commit frequently blindly, we have a
broken code base
• Implied condition: each integration will not decrease the code quality• This is the key to CI • The rest is implementation details
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI: The Simple Implementation
• How do we determine code quality? We test it
• How do we test code? We build it and run tests against it
• This is how we get from CI to CI servers and software builds
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
So Simple but Very Important
• A build & unit test is a rudimentary check• Gives important feedback• The heartbeat of the dev organization
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI at Scale?
• With too many committers into the same code line, the likelihood something is broken right now is high
• We stop caring about failures
• The feedback is meaningless, and productivity suffers. CI becomes “constantly broken”
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Agenda
• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Reasons People Work In Parallel
It’s often a business reason…not for optimization
• Maintenance Release – Patches
• Customer Specials - Just for one customer
• Development branches – Developer spikes for
experiments that will integrate at an unknown time
• Code Configurations – Different configurations of code
based on environment, such as UAT, QA
• Special Content – Images, binaries
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Technical Debt Grows
As code lines diverge, integration becomes a bigger eventual problem. … But we don’t always want to merge immediately
• Maintenance Release – Don’t want new features
• Customer Specials – One directional integration
• Development branches – Dev enjoys a stable playground
• Special Content – Reused at different cadence than code
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Agenda
• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Finding the Isolation / Integration Balance
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Integration Is Easy With a Few People
Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
It’s hard at scale
Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Little Hiccup
• Race condition: I could update to the latest code before the build/test notifies me it doesn’t compile
• Cost associated with this is huge
• Leads to “interesting solutions”
• Preflight• Build branches• Build Acceleration• Build Avoidance• Componentization
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Cost of Integration
Per week:• # of check-ins per developer?• # of developers?• % of check-ins that break things?• # of problems encountered per developer?• Length of time per problem encountered?
200 devs, 2 commits each, 1% are bad = 4 breaks/day
If each breakage cuts productivity 50% for 15 minutes, we lose 100 dev hours / day (@$50/hr = $5000/day)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI
CI
CI
Team A
Team B
ws
ws
ws
Multi-Stage Continuous Integration
Minutes Hours
int
Daily
ws
ws
ws
• Continuous
Integration
• Continuous Integration
• Automated Tests
• Unit Tests
• Binary Deploy
• Database Deploy
• Web Deploy
• Database Deploy
• Web Deploy
• Binary Deploy
• Customer Sign-
Off
• Database Deploy
• Web Deploy
• Binary Deploy
• IT Sign-Off
• Rollback (if needed)
x60x30x3x1
x1x4x10x100s
x1x2x5x10
0:102:003:005:00
• Team Specific CI
• Continuous
Integration
• Automated Tests
• Unit Tests
• Binary Deploy
• Database Deploy
• Web Deploy
• Database
Deploy
• Web Deploy
• Binary Deploy
• Customer Sign-
Off
• Database Deploy
• Web Deploy
• Binary Deploy
• IT Sign-Off
• Rollback (if needed)
Automated Merges
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
A Feature Branch Example
Urbancode’s Website Today… Next (not ready for prime time)
We need two build configurations, deploy configs, test configs, and test environments
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Building and Testing Multiple Code Lines
• Build and test every active code line (continuously)
• Automatically run code scans, unit tests and functional tests
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI for a New Code Line Must Be Cheap
• Build and deployment configuration• Maximize reuse• Potentially create dynamically
• Tests• Core regression suites shared across code lines• Tests for new functionality shared as code is shared
• Test Environments• No excuse for a lack of test environments today• Deploy “what’s in production” as a baseline
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Deploy Time Integration Is the Same
• In the team’s environment:• Continuously deploy and execute functional tests• Regularly (nightly / weekly) deploy other
components that have passed integration tests
• When new features work well, deliver to the integration test environment
• Releases to Stage & Prod match the full contents of an earlier test environment
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
In Closing
• Compile is the first test
• Unit test, deploy, automation can come after
• Get Feedback as Fast as possible
• Different teams need different types of builds
• Separate code configurations based on process
• It’s about people, but tooling helps
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Best Practices For:Agile
Parallel
Global
Compliance
Change ManagementVisual Process
Automation
What Is AccuRev?
Much more than Version Control…AccuRev enables Software Process Automation
VersionControlSCM
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Urbancode Products
uBuild (AnthillPro)• Continuous integration and delivery
uDeploy• Deployment and release management
uCloud (soon)• Environment provisioning from your private cloud
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Q&A
Call: +1 216 858-9000Email: [email protected]: www.urbancode.comLearn: blogs.urbancode.com
Call: 1-800-383-8170 or +1-781-861-8700Email: [email protected]: www.accurev.comLearn: blog.accurev.com