Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary.

15th February 2017

Running CI at scale with containers at PayPal

Sathiya Narayanan

© 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 2

At PayPal, we take CI seriously

©2015 PayPal Inc. Confidential and proprietary. 3

o One Ginormous Jenkins Instance, customized for PayPal

o 40,000 runnable jobso 24,000 builds run each day on Single

Instanceo Stack: Java, Node, Python, C++ o CD : Unit Test, Static analysis and code

coverage, & deployments o One Jenkins master with a pool of a

hundred Jenkins slaveso Custom build system for 50 MLOC in <1

Min

Single Jenkins instance

One Giant Jenkins Butler!


o SPOFo Scalability o Change Managemento No freedom for userso Inefficient resource managemento Plugins don’t scale

Limitations with single Jenkins instance

Too much load!


o 1:1 modelo Individual VMs running Jenkins for each

Applicationo 2,500+ VMs in useo Each Jenkins instance would have 2

executor on master to perform buildso Freedom for users

Users loved it!

Dedicated VM for each Jenkins Instance


Jenkins on Mesos

o Light weight o Long running tasko Marathon scheduler

Jenkins Master

o Provisioned on demand with Jenkins Mesos plugin

o Flavors of slaves based on the application stack

Jenkins slave


Optimized Resource Utilization with Mesos


Operating cost reduced 10X times

After Mesoso ~2400 CPU’s, o 18 TB, o 7.2 TB RAM

Before Mesoso ~ 24,000 CPU’s, o 180 TB Disk, o 54 TB RAM


After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K

2.7 million saving could buy:- A personal Jet for GPI

10X Savings

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

Recovery was slow without Mesos


10

After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K


10X Savings

Before:


~3 Million

Instant Recovery with Mesos


After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K


10X Savings

Before:


~3 Million

Our workload was too much for Marathon


o Zero outages and no customer impact.o Aurora was more stableo Aurora was able to scale to more than

5,000 jobs

Replaced Marathon with Aurora


After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K


10X Savings

Before:


~3 Million13

CIAPI

Aurora

Mesos Master

Mesos Cluster

Swift / Object Storage

Zookeeper

Jenkins Master Jenkins Slave

Nginx

Highly available setup

Devxdash


Challengeso Inconsistent build environments between user

workstation and CI build system

o Tasks stepping onto each other

o Providing conflicting tool set for different technology stacks

o Different DevOps teams/applications had their own Mesos cluster

o Slower rollout of new tools to the build environment

o Inconsistent build environments within the CI build system


Why Docker?

o Task isolation

o Cloud Native

o Eliminates the dependency on the host

o Reproducibility

o Portability

o Immutability

o Homogenous cluster


10X Savings

Before:


~3 Million

Dockerized Jenkins Masters

o Immutable containers for consistency

o Eliminates the need to have tools installed on slave

o Provides task isolation

o Other work loads can co-exist in Mesos cluster

o Easy to cater to conflicting OS / toolset requirements in a polyglot environment

o Easy to roll out updates

o Other work loads can co-exist in Mesos cluster

o Users can bake their own build environment

Dockerized Jenkins Slaves

Jenkins with Docker


10X Savings

Before:


~3 Million

Adoption to Docker containers

BC [ Before Containers ]

AD [ After Docker ]

SSH key / Tokens / Maven credentials

Stored in local disk Injected only during build time and rotating them

Toolsets like JDK, Maven etc.

Installed in host Installed within Docker container


Before:


~3 Million

What is state for Jenkins master ?

o Jenkins war

o Plugins

o Global configuration

o Job configuration

o Build historyo Console outputo Reports like code coverage, test result trend etc.o Archived artifacts


Before:


~3 Million

Backing up Jenkins state

Supervisord

Archive the

directory

Encrypt the data

Store in cloud

storage


Before:


~3 Million

When Jenkins Master container moves to another VM

Untar the files

into Jenkins home

Decrypt the

backup

Check if a backup exists

in cloud storage

Provisioning script [Entry

point for Docker]


10X SavingsBefore:


~3 Million

Use case - Maven build

Total build time - 4 minutesPersisting workspace for subsequent builds

Decreased build timeSource code [Clone]

Artifacts

Time taken 0m27.618s

2m56.000sNumber of files 10570 4456

Total Size 158 M 173 M


10X Savings

Before:


~3 Million

CIAAS with Docker ArchitecturePublic cloud

Private cloud


After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K


Before:


~3 Million

Software

Paradigm shift in CI at PayPal with Docker and Mesos