Continuous Integration

Practices and Pitfalls

David Cate and Chris Mair

May 2010

David and Chris – UnmaskedDavid Cate – 10+ years of

development from varied backgrounds. Interests include automation and pragmatic problem solving.

Chris Mair – 20+ years doing what I love. Passion for software craftsmanship, testing and automation. Wells Fargo.

AgendaIntroductionWhat is Continuous

Integration?◦Principles, practices and benefits

CI versus Build ManagementCI Anti-PatternsChallenges and Lessons LearnedWhere we are now and what’s on our

roadmapShare experiences and Q & A

“Oddly enough, it seems that when you run into a painful activity, a good tip is to do it more often.”

- Martin Fowler, in the foreword of Continuous Integration

What is Continuous Integration?

“Continuous Integration is a software development

practice where members of a team integrate their work

frequently, usually each person integrates at least daily

- leading to multiple integrations per day. Each

integration is verified by an automated build (including

test) to detect integration errors as quickly as possible.

Many teams find that this approach leads to

significantly reduced integration problems and allows a

team to develop cohesive software more rapidly.“

-- Martin Fowler

CI – PrinciplesProvide quick and continuous

feedbackContinuously produce working

softwareAddress risks earlyAutomate wherever possibleProvide transparency and visibilityBuild quality in

CI – PracticesMaintain a single source repositoryAutomate the buildStandalone build (not tied to IDE)Run tests and inspections as part of buildBuild automatically upon code commitKeep the build fastDesign builds to fail fastProvide continuous visibility and feedbackCommit code frequently (at least every day)Run private builds before check-inFix broken builds immediatelyDon’t commit broken code (fails private build)

CI – Foundational PracticesMaintain a single source repository

◦ All code and build inputs under version control

Automate the build ◦ Run from a single command

Standalone build◦ Not tied to IDE

CI – Project Practices Run tests and inspections as part of build

◦ All tests and inspections must pass

Build automatically upon code commit

Keep the build fast◦ Want fast feedback; minutes, not hours

Design builds to fail fast◦ Run tests/inspections that are most likely to fail first◦ Want to find out about failures as soon as possible

Provide continuous visibility and feedback◦ Project status dashboard◦ Continuous feedback devices

CI - Developer PracticesCommit code frequently (at least daily)

◦ Reduces magnitude of changes to be integrated with each build

◦ Reduces likelihood of stale (out-of-sync) code changes◦ Reduces effort of troubleshooting failures – less has

changed

Run private builds before check-in◦ Execute same set of tests and inspections that CI will◦ IDE may help here (e.g. PMD or Checkstyle plugin)

Fix broken builds immediately

Don’t commit broken code◦ Broken code should fail the private build

CI - Benefits

Reduced risk

Reduced manual, repetitive processes

Easier debugging / troubleshooting

Generate deployable software at any

time

Better project visibility

Greater confidence in the software

Advanced CI Practices

Staged builds

Parallel and/or distributed build

execution

Automated deployment

Dedicated integration build (CI) machine

Continuous database integration

Staged Build ExampleCompilation / Fast Introspection

Unit Tests

Integration / System Tests

Functional Tests

Overall Pass/Fail

Parallel Build Example

Compilation / Fast Introspection

Unit Tests

Integration / System Tests

Functional Tests

Overall Pass/Fail

Compilation / Fast Introspection

Unit Tests

Integration / System Tests

Functional Tests

Overall Pass/Fail

Build 1 on Machine 1 Build 2 on Machine 2

Parallel Build Example

Compilation / Fast Introspection

Unit Tests

Integration / System Tests

Functional Tests

Overall Pass/Fail

Continuous Database Integration• Rebuild your database and test data for every

build.

• Store database artifacts in version control• DDL, DML, config files, etc.

• Generate the database from build scripts • Scripts for database creation, configuration, population

• Apply continuous testing / inspection against your database(s)

• LiquiBase, Rails Migrations

Continuous Integration vs Build Management

“Build Management, essentially, aims to produce a set of artifacts that correspond to a specified snapshot of source code, and to be able to hand off those artifacts to third parties outside of development”

UrbanCode - http://www.cmcrossroads.com/pdf/DrawingtheLine.pdf

CI – BM Comparisons

Continuous Integration

Build Management

What is the purpose? To integrate the latest changes and determine current code quality

To provide a definitive set of usable artifacts based on a given code slice in time

Who is the audience? Internal External (non-developer)

What are the products?

Pass/Fail, Metrics, Code quality

Deployable artifact

What is the code base?

The latest changes Fixed time reference

Who benefits the most?

Developers Infrastructure / Support

Questions to PonderWho should own the governance for CI? How

about for build management?

Should similar, different, or exactly the same build processes be used for CI and BM?

What is needed to verify the quality of the artifact generated by the BM process?

CI Developer Anti-PatternsInfrequent check-ins

Premature check-ins (failing code)

Ignoring broken builds

Turning off tests to correct builds

Waiting until end of day to commit changes

Inconsistent developer build environments (works on my machine)

CI Tooling Anti-Patterns Insufficient feedback

Too much (spam) feedback

Underpowered (slow) build machine

Bloated build

Weak builds that never fail (Continuous Ignorance)

Using Scheduled Builds versus building software with every code change

Failing to clean out old artifacts before each build

http://www.ibm.com/developerworks/java/library/j-ap11297/index.html

CI - Challenges

“CI is not just a technical implementation; it is also an organizational and cultural implementation”

– Paul Duvall, Continuous Integration

Challenges – Cultural

Development culture◦Not familiar with continuous integration◦ Ignore failing tests or broken builds◦Not checking in code frequently (daily)◦Not running tests before check-in◦Lack of encouragement for CI practices

Challenges – Personnel/ExperienceLack of expertise

◦ Continuous Integration◦ CruiseControl / Hudson◦ Deployment (e.g., WebSphere)

No dedicated resources for CI (anymore)◦Made great progress when we did have

that.

Challenges – InfrastructureObtaining system accounts, disk

space, etc. to run CI

Insufficient and constrained environments◦ Not enough, not always available◦ Test environments, app servers,

databases, etc.

Challenges – Technical ImplementationStuck on outdated technologies

◦ Java 1.4, Tru64, PVCS, VB, VisualC++

Technology stacks◦ Grails, Java, Struts, etc – all with differing folder

structures

Database changes in separate source repo◦ Database changes in PVCS; main code in CVS

Disparate systems for CI integration◦ CVS, PVCS, PICCT, QualityCenter, VersionOne, Fortify

Cruise Control / Hudson – learning curve

Challenges – Running BuildsBrittle tests (e.g., data-dependency)

Slow tests or lots of time-consuming tests

Build “noise” ◦ CI build fails for reasons outside developer control (e.g.,

bug in Hudson or file access issues)

Interactions between builds ◦ “Access denied” and “Cannot delete …” errors on

Hudson◦ Shortcomings of Windows build server

Where have we been?Cruise Control

◦ The overall project configurations were a project unto itself – Cruise control checked itself out and handled it’s own updates

◦ Configurations were a mix of Cruise Control and raw Ant scripts.

◦ Projects were compiled, tested and provided introspection with results via email. Trending was not being utilized

Cruise Control Configuration

CRUISE CONTROL

PROJECT

ANT

ANT

CORE config.xml

common-build.xml

build-[project].xml

Project-specific build

common.properties

[project].properties

[project].properties

Properties

common.properties

base-build.xml

cc-[project].xml

Lessons Learned – Cruise Control Readability trumps DRY when working with

configurations

If the configuration is so complex that it can easily break, there’s very little advantage to having the system

The interface can make / break even a very good application. It’s about the user experience.

Getting the builds to call properly named tasks that explain what they do makes life easier (i.e. Build, BuildAndTest, Test, etc.)

Developers will become lax running local tools if there is a central place running them

The End of Cruise Control

Several reasons for leaving Cruise Control:◦Lack of community involvement

◦Extensibility / Plug-ins

◦We finally moved away from Java 1.4

◦The framework required deeper knowledge than was readily available

Where We Are NowHudson running mixture of scheduled /

change-based builds integrating a mixture of tests, introspective utilities, while providing email feedback to interested parties.

~25 projects using CI/Hudson

Whole spectrum of maturity/sophistication of CI usage across projects

Hudson Each project is capable of self-building

Builds are run after a version control check-in.

All projects build regardless of change on Thursday night. This way, broken builds are at most based on 1 week worth of change

Communication comes in the form of emails – fail (and first pass) on most projects, pass/fail on all critical projects

Groovy scripts analyze the configuration and scavenge the scripts for backups.

Hudson Technology StackJava, Groovy, Grails

CPD/PMD, CodeNarc, Fortify, JUnit, Cobertura

All builds are done via Ant

Favorite Hudson Plug-ins Used:◦ CVS Tagging Plugin◦ Email-ext Plugin◦ Sidebar-Link Plugin◦ Green Balls

Hudson Dashboard

Hudson – Project Details

Lessons LearnedCruise Control was good but happier with

Hudson

Best to gradually evolve to “full” CI◦ Start with simple build/compile; ◦ Add tests, analysis/metrics as the team can handle

and appreciate it.

Each team is different and has a different culture and personalities. ◦ No “one size fits all”◦ Varying appetites for risk, process, rules, oversight

CI – Realized Benefits

Expected and Unexpected BenefitsExposed weaknesses in processes and

code

Significantly increased transparency

Motivated and enforced consistent build (repeatable process)

Significantly fewer deployment issues

Our Roadmap / Future GoalsStreamlined build with

dependency-management (Ivy; Gradle)

Automated deployment

Streamlining and automating SDLC lifecycle (including AntHill Pro).

Thoughts on CI Start Now. Don’t let not having the entire process laid out prevent

you from doing the work that can be done now.

Make Plans and Act. Choose a path and investigate. Waiting for the perfect plan means getting nothing perfectly done.

Approach with Passionate Detachment. The result may be meaningless for the end result but the learning is the valuable asset here.

Don’t Stop. Some of the best results come from small deliberate steps. Continue making incremental steps even when there “isn’t time”.

Acknowledge Change. It’s inevitable and it’s not always bad.

Don’t Work in Isolation. Isolating your work and yourself will ensure that your system does not reflect the needs of the team.

Feedback Devices

Other notification mechanisms include: • Email• RSS feeds• Taskbar monitors such as CCTray (for CruiseControl) • X10 devices (like LavaLamps) • Instant Messages through Jabber, etc. • SMS (Text Messages)

http://www.ibm.com/developerworks/java/library/j-ap11297/index.html

Ambient Orb

Share Experiences?Successful CI?

Unsuccessful? Abandoned?

Challenges?

Involvement of Management?

Enforcement policy? Reward versus punishment for adopting CI and dealing with broken builds?

Q & A

References Continuous Integration – Paul M. Duvall, Addison Wesley, 2007

“Continuous Integration” – Martin Fowler

http://www.martinfowler.com/articles/continuousIntegration.html

Pragmatic Project Automation – Mike Clark, Pragmatic Bookshelf,

2004

“Continuous Integration: Patterns and Anti-patterns” – DZone

Refcard http://refcardz.dzone.com/refcardz/continuous-integration

“Continuous Integration: Servers and Tools” – DZone Refcard

http://refcardz.dzone.com/refcardz/continuous-integration-servers?oid=hom18527

“Drawing the Line: Continuous Integration and Build Management” http://www.cmcrossroads.com/pdf/DrawingtheLine.pdf

“Evolutionary Database Design” – Martin Fowler and Pramod Sadalage http://www.martinfowler.com/articles/evodb.html

IBM developerWorks series “Automation for the people”, by Paul

Duvall - http://www.ibm.com/developerworks/views/java/libraryview.jsp?search_by=automation+people