From Virtual Machines to ContainersAchieving Continuous Integration, Build Reproducibility, Isolation and Scalability

London, UK, June 2015

1

#jenkinsconf

Agenda

• Introduction• The Dark Age• The Promising Present• Lessons Learned• The Bright Future

#jenkinsconf

2014201320122011201020092008

IncorporationCamunda Services GmbHBerlin, Germany

IncorporationCamunda Inc.San Francisco, US

BPM Consulting BPM Software Vendor

● 30 Fulltime Employees ● No external funding ● Avg. 50% Turnover Growth per Year

Introduction - About Camunda@camundaBPM

Introduced Hudson

Adapted Jenkins

#jenkinsconf

Introduction - Camunda BPM

#jenkinsconf

Introduction - What is Camunda BPM

• Camunda BPM is an open source platformfor workflow and business process automation

• Integrates with:• 7 Application Server (11 different versions)• 6 Databases (17 different versions)• 1 Development & 4 Maintained Versions

#jenkinsconf

• Every Camunda BPM version is tested against:• 187 combinations of DBs and App Servers• 11 JDKs• ~ 400 Jobs per version

• Bi-Annual release of a new Camunda BPM version• Support for Enterprise Customer (24/7 + Fix Time)

Introduction - Why CI is Important

#jenkinsconf

The Dark AgeWhy our CI had to change

7

#jenkinsconf

The Dark Age - The Numbers

• 1 Jenkins Master with alot of plugins• 8 Jenkins Slaves VMs• ~1000 Jobs total configured / manually managed:

• 4 Camunda Versions• Community Projects• Websites• Maintenance

#jenkinsconf

The Dark Age - Isolation Problem

• Unit and Integration Tests need a database/application server

• Only 1 Instance per Database• All Jobs use the same Databases• Every half-year a new Version (~400 Jobs) using

same Databases

#jenkinsconf

The Dark Age - No Scalability

• Total Executors: 12• 1 Jenkins -> 4 Executors• 8 static heterogenous slave VMs, each with 1

Executor• Jobs tied to slaves through labels• Slaves “restrict” database access by allowing no

other build to run

#jenkinsconf

The Dark Age - Maintenance Problems

• Upgrading Jenkins or any plugin• Supporting a new Database vendor / version• Supporting a new App Server version• Creating jobs for new Camunda BPM version• Disaster recovery

#jenkinsconf

The Dark Age - The Other Problems

• Slow feedback cycle for developers• Developers cannot reproduce CI environments• QA engineers use and maintain separate bloated

test build setup

#jenkinsconf

So ...

#jenkinsconf

The Promising PresentHow we solved our Problem

14

#jenkinsconf

The Present - What we achieved

• Configuration & Infrastructure as Code• Isolated and Reproducible Jobs• Scalable CI Infrastructure

#jenkinsconf

The Present - Infrastructure as Code

1. Every configuration is checked into SCM2. Every application/test runs in a Docker

Container3. Every Docker image is build

automatically

#jenkinsconf

The Present - Infrastructure as Code

1. Every Configuration is checked into SCM

• Docker for• Applications• Test Environments

• JobDSL for• Jenkins Jobs

#jenkinsconf

The Present - Infrastructure as Code

2. Every application/test runs in a Docker Container

Images:• Application (Jenkins, Nexus …)• Test Env. Images (DB + SSH)• Build Env. Images

• DIND, QEMU + Packer.io

#jenkinsconf

The Present - Infrastructure as Code

3. Every Docker Container is build automatically

• Own Jenkins for Docker/KVM Images• KVM Images build in Docker Container

with Packer + QEMU• KVM Images bundled in Docker Image

#jenkinsconf

The Present - The Current Flow

camunda-ci

camunda

Camunda BPMPlatform

InfrastructureJenkins

CI Jenkins

#jenkinsconf

The Present - Isolation

One Jenkins per Concern:

• CI• Release• Infrastructure• Community and other Projects• Marketing

#jenkinsconf

The Present - Isolation & Reproducibility

• Every Jobs runs in an One-Shot Docker Container• No Interference between Jobs• The Database Settings are well documented• Every Docker Image is stored in a private registry• Developers/QA can use the Docker Images for local

testing

#jenkinsconf

The Present - Scalability

• Jenkins uses Docker-Plugin with one Docker Cloud running on Docker Swarm

• Docker images are added through Groovy scripting• Running on Commodity Hardware

• 1 Infrastructure Host (Jenkins, Nexus, …)• 6 Docker Hosts as 1 Swarm

#jenkinsconf

The Present - Scalability

#jenkinsconf

The Present - Advantages

• Easy to add new Databases/Test Environments• New Release = New Branch of JobDSL Repository• Fully parallelized Job Execution• Accountable Configuration History• Testable Infrastructure• Minimize Administration Overhead

#jenkinsconf

The Present - Conclusion

• 2 People + 3 Months of Work• A fully scalable, isolated and reproducible CI

Infrastructure• Faster Feedback• Happy Developers and Product Owner

#jenkinsconf

Lessons learned

27

Some advice from us to you

#jenkinsconf

• Automate as much as you can• Jenkins config• Jobs config• Environment creation

• Design to scale to support the business agility

Lessons learned - Architecture

#jenkinsconf

Lessons learned - Testing

Test everything:• Jenkins configuration

• required plugin features• Job generation• Docker images• Scalability• Disaster recovery

#jenkinsconf

Lessons learned - Job DSL

• Unit-test the job generation• Write JobGenerator classes to abstract the

common build logic of most jobs out of the box• Use XML diffing to compare previously generated

jobs with new ones

#jenkinsconf

Lessons learned - Docker

https://github.com/rancherio/vm/

#jenkinsconf

• Pin your plugin versions• Be prepared to contribute to plugin development or

maintain a branch yourself• Choose the right plugin for the job

Our Top 3 plugins:JobDSL, Docker-Plugin, Build-Failure-Analyzer

Lessons learned - Plugins

#jenkinsconf

Lessons learned - Control

• Control as much as possible• Third party binaries vs package manager• explicit versions• own mirrors for important packages

#jenkinsconf

The Bright FutureWhat comes next

34

#jenkinsconf

The Future

• Public Community Jenkins• Internal Webapp for Developers and QA to start

Environments• Jenkins is deploying Jenkins• Back to the Datacenter• Centralized Logging and Monitoring (ELK)

#jenkinsconf

#jenkinsconf

37

#jenkinsconf

Resources

• Camunda CI repositoryhttps://github.com/camunda-ci

• Job DSL gradle examplehttps://github.com/sheehan/job-dsl-gradle-example

• Camunda BPM Docker imagehttps://registry.hub.docker.

com/u/camunda/camunda-bpm-platform/

#jenkinsconf

We Are Hiring

Come Join Us@camundaBPM

http://camunda.com/jobs

#jenkinsconf

Please Share Your Feedback

• Did you find this session valuable?• Please share your thoughts in the

Jenkins User Conference Mobile App.• Find the session in the app and click

on the feedback area.

40