Continuous Integration
Practices and Pitfalls
David Cate and Chris Mair
May 2010
David and Chris – UnmaskedDavid Cate – 10+ years of
development from varied backgrounds. Interests include automation and pragmatic problem solving.
Chris Mair – 20+ years doing what I love. Passion for software craftsmanship, testing and automation. Wells Fargo.
AgendaIntroductionWhat is Continuous
Integration?◦Principles, practices and benefits
CI versus Build ManagementCI Anti-PatternsChallenges and Lessons LearnedWhere we are now and what’s on our
roadmapShare experiences and Q & A
“Oddly enough, it seems that when you run into a painful activity, a good tip is to do it more often.”
- Martin Fowler, in the foreword of Continuous Integration
What is Continuous Integration?
“Continuous Integration is a software development
practice where members of a team integrate their work
frequently, usually each person integrates at least daily
- leading to multiple integrations per day. Each
integration is verified by an automated build (including
test) to detect integration errors as quickly as possible.
Many teams find that this approach leads to
significantly reduced integration problems and allows a
team to develop cohesive software more rapidly.“
-- Martin Fowler
CI – PrinciplesProvide quick and continuous
feedbackContinuously produce working
softwareAddress risks earlyAutomate wherever possibleProvide transparency and visibilityBuild quality in
CI – PracticesMaintain a single source repositoryAutomate the buildStandalone build (not tied to IDE)Run tests and inspections as part of buildBuild automatically upon code commitKeep the build fastDesign builds to fail fastProvide continuous visibility and feedbackCommit code frequently (at least every day)Run private builds before check-inFix broken builds immediatelyDon’t commit broken code (fails private build)
CI – Foundational PracticesMaintain a single source repository
◦ All code and build inputs under version control
Automate the build ◦ Run from a single command
Standalone build◦ Not tied to IDE
CI – Project Practices Run tests and inspections as part of build
◦ All tests and inspections must pass
Build automatically upon code commit
Keep the build fast◦ Want fast feedback; minutes, not hours
Design builds to fail fast◦ Run tests/inspections that are most likely to fail first◦ Want to find out about failures as soon as possible
Provide continuous visibility and feedback◦ Project status dashboard◦ Continuous feedback devices
CI - Developer PracticesCommit code frequently (at least daily)
◦ Reduces magnitude of changes to be integrated with each build
◦ Reduces likelihood of stale (out-of-sync) code changes◦ Reduces effort of troubleshooting failures – less has
changed
Run private builds before check-in◦ Execute same set of tests and inspections that CI will◦ IDE may help here (e.g. PMD or Checkstyle plugin)
Fix broken builds immediately
Don’t commit broken code◦ Broken code should fail the private build
CI - Benefits
Reduced risk
Reduced manual, repetitive processes
Easier debugging / troubleshooting
Generate deployable software at any
time
Better project visibility
Greater confidence in the software
Advanced CI Practices
Staged builds
Parallel and/or distributed build
execution
Automated deployment
Dedicated integration build (CI) machine
Continuous database integration
Staged Build ExampleCompilation / Fast Introspection
Unit Tests
Integration / System Tests
Functional Tests
Overall Pass/Fail
Parallel Build Example
Compilation / Fast Introspection
Unit Tests
Integration / System Tests
Functional Tests
Overall Pass/Fail
Compilation / Fast Introspection
Unit Tests
Integration / System Tests
Functional Tests
Overall Pass/Fail
Build 1 on Machine 1 Build 2 on Machine 2
Parallel Build Example
Compilation / Fast Introspection
Unit Tests
Integration / System Tests
Functional Tests
Overall Pass/Fail
Continuous Database Integration• Rebuild your database and test data for every
build.
• Store database artifacts in version control• DDL, DML, config files, etc.
• Generate the database from build scripts • Scripts for database creation, configuration, population
• Apply continuous testing / inspection against your database(s)
• LiquiBase, Rails Migrations
Continuous Integration vs Build Management
“Build Management, essentially, aims to produce a set of artifacts that correspond to a specified snapshot of source code, and to be able to hand off those artifacts to third parties outside of development”
UrbanCode - http://www.cmcrossroads.com/pdf/DrawingtheLine.pdf
CI – BM Comparisons
Continuous Integration
Build Management
What is the purpose? To integrate the latest changes and determine current code quality
To provide a definitive set of usable artifacts based on a given code slice in time
Who is the audience? Internal External (non-developer)
What are the products?
Pass/Fail, Metrics, Code quality
Deployable artifact
What is the code base?
The latest changes Fixed time reference
Who benefits the most?
Developers Infrastructure / Support
Questions to PonderWho should own the governance for CI? How
about for build management?
Should similar, different, or exactly the same build processes be used for CI and BM?
What is needed to verify the quality of the artifact generated by the BM process?
CI Developer Anti-PatternsInfrequent check-ins
Premature check-ins (failing code)
Ignoring broken builds
Turning off tests to correct builds
Waiting until end of day to commit changes
Inconsistent developer build environments (works on my machine)
CI Tooling Anti-Patterns Insufficient feedback
Too much (spam) feedback
Underpowered (slow) build machine
Bloated build
Weak builds that never fail (Continuous Ignorance)
Using Scheduled Builds versus building software with every code change
Failing to clean out old artifacts before each build
http://www.ibm.com/developerworks/java/library/j-ap11297/index.html
CI - Challenges
“CI is not just a technical implementation; it is also an organizational and cultural implementation”
– Paul Duvall, Continuous Integration
Challenges – Cultural
Development culture◦Not familiar with continuous integration◦ Ignore failing tests or broken builds◦Not checking in code frequently (daily)◦Not running tests before check-in◦Lack of encouragement for CI practices
Challenges – Personnel/ExperienceLack of expertise
◦ Continuous Integration◦ CruiseControl / Hudson◦ Deployment (e.g., WebSphere)
No dedicated resources for CI (anymore)◦Made great progress when we did have
that.
Challenges – InfrastructureObtaining system accounts, disk
space, etc. to run CI
Insufficient and constrained environments◦ Not enough, not always available◦ Test environments, app servers,
databases, etc.
Challenges – Technical ImplementationStuck on outdated technologies
◦ Java 1.4, Tru64, PVCS, VB, VisualC++
Technology stacks◦ Grails, Java, Struts, etc – all with differing folder
structures
Database changes in separate source repo◦ Database changes in PVCS; main code in CVS
Disparate systems for CI integration◦ CVS, PVCS, PICCT, QualityCenter, VersionOne, Fortify
Cruise Control / Hudson – learning curve
Challenges – Running BuildsBrittle tests (e.g., data-dependency)
Slow tests or lots of time-consuming tests
Build “noise” ◦ CI build fails for reasons outside developer control (e.g.,
bug in Hudson or file access issues)
Interactions between builds ◦ “Access denied” and “Cannot delete …” errors on
Hudson◦ Shortcomings of Windows build server
Where have we been?Cruise Control
◦ The overall project configurations were a project unto itself – Cruise control checked itself out and handled it’s own updates
◦ Configurations were a mix of Cruise Control and raw Ant scripts.
◦ Projects were compiled, tested and provided introspection with results via email. Trending was not being utilized
Cruise Control Configuration
CRUISE CONTROL
PROJECT
ANT
ANT
CORE config.xml
common-build.xml
build-[project].xml
Project-specific build
common.properties
[project].properties
[project].properties
Properties
common.properties
base-build.xml
cc-[project].xml
Lessons Learned – Cruise Control Readability trumps DRY when working with
configurations
If the configuration is so complex that it can easily break, there’s very little advantage to having the system
The interface can make / break even a very good application. It’s about the user experience.
Getting the builds to call properly named tasks that explain what they do makes life easier (i.e. Build, BuildAndTest, Test, etc.)
Developers will become lax running local tools if there is a central place running them
The End of Cruise Control
Several reasons for leaving Cruise Control:◦Lack of community involvement
◦Extensibility / Plug-ins
◦We finally moved away from Java 1.4
◦The framework required deeper knowledge than was readily available
Where We Are NowHudson running mixture of scheduled /
change-based builds integrating a mixture of tests, introspective utilities, while providing email feedback to interested parties.
~25 projects using CI/Hudson
Whole spectrum of maturity/sophistication of CI usage across projects
Hudson Each project is capable of self-building
Builds are run after a version control check-in.
All projects build regardless of change on Thursday night. This way, broken builds are at most based on 1 week worth of change
Communication comes in the form of emails – fail (and first pass) on most projects, pass/fail on all critical projects
Groovy scripts analyze the configuration and scavenge the scripts for backups.
Hudson Technology StackJava, Groovy, Grails
CPD/PMD, CodeNarc, Fortify, JUnit, Cobertura
All builds are done via Ant
Favorite Hudson Plug-ins Used:◦ CVS Tagging Plugin◦ Email-ext Plugin◦ Sidebar-Link Plugin◦ Green Balls
Hudson Dashboard
Hudson – Project Details
Lessons LearnedCruise Control was good but happier with
Hudson
Best to gradually evolve to “full” CI◦ Start with simple build/compile; ◦ Add tests, analysis/metrics as the team can handle
and appreciate it.
Each team is different and has a different culture and personalities. ◦ No “one size fits all”◦ Varying appetites for risk, process, rules, oversight
CI – Realized Benefits
Expected and Unexpected BenefitsExposed weaknesses in processes and
code
Significantly increased transparency
Motivated and enforced consistent build (repeatable process)
Significantly fewer deployment issues
Our Roadmap / Future GoalsStreamlined build with
dependency-management (Ivy; Gradle)
Automated deployment
Streamlining and automating SDLC lifecycle (including AntHill Pro).
Thoughts on CI Start Now. Don’t let not having the entire process laid out prevent
you from doing the work that can be done now.
Make Plans and Act. Choose a path and investigate. Waiting for the perfect plan means getting nothing perfectly done.
Approach with Passionate Detachment. The result may be meaningless for the end result but the learning is the valuable asset here.
Don’t Stop. Some of the best results come from small deliberate steps. Continue making incremental steps even when there “isn’t time”.
Acknowledge Change. It’s inevitable and it’s not always bad.
Don’t Work in Isolation. Isolating your work and yourself will ensure that your system does not reflect the needs of the team.
Feedback Devices
Other notification mechanisms include: • Email• RSS feeds• Taskbar monitors such as CCTray (for CruiseControl) • X10 devices (like LavaLamps) • Instant Messages through Jabber, etc. • SMS (Text Messages)
http://www.ibm.com/developerworks/java/library/j-ap11297/index.html
Ambient Orb
Share Experiences?Successful CI?
Unsuccessful? Abandoned?
Challenges?
Involvement of Management?
Enforcement policy? Reward versus punishment for adopting CI and dealing with broken builds?
Q & A
References Continuous Integration – Paul M. Duvall, Addison Wesley, 2007
“Continuous Integration” – Martin Fowler
http://www.martinfowler.com/articles/continuousIntegration.html
Pragmatic Project Automation – Mike Clark, Pragmatic Bookshelf,
2004
“Continuous Integration: Patterns and Anti-patterns” – DZone
Refcard http://refcardz.dzone.com/refcardz/continuous-integration
“Continuous Integration: Servers and Tools” – DZone Refcard
http://refcardz.dzone.com/refcardz/continuous-integration-servers?oid=hom18527
“Drawing the Line: Continuous Integration and Build Management” http://www.cmcrossroads.com/pdf/DrawingtheLine.pdf
“Evolutionary Database Design” – Martin Fowler and Pramod Sadalage http://www.martinfowler.com/articles/evodb.html
IBM developerWorks series “Automation for the people”, by Paul
Duvall - http://www.ibm.com/developerworks/views/java/libraryview.jsp?search_by=automation+people