Build, Publish, Deploy and Test Docker images and containers with Jenkins Workflow

Build, Publish, Deploy and Test Docker images and containers with Jenkins Workflow.Nigel HarnimanSenior Solutions Architect, CloudBees Inc

About me

Nigel Harniman

@harnimanBuild Engineer,

Devops and Architect

DevOps, Infra as Code,Continuous Delivery

“Software is eating the world”

Marc Andreessen

4

How Do You Deliver Better Software Faster?

ProdDev

5

Automation is the Key

Photo courtesy of Steve Jurvetson via Flickr

The Docker Advantage

Docker Has PotentialAn example: Software Configuration Management Space

Docker Has PotentialAn example: Software Configuration Management Space

The New World Order: Containers Codify OS Config

9

ProdDev QA Staging

DEV Server/VM QA Server/VM STG Server/VMPROD Server/VM

<PROD OS config><STG OS config><QA OS config><DEV OS config>

App<code>

<APP OS config>

App<code>

<APP OS config>

App<code>

<APP OS config>

App<code>

<APP OS config>

So is this how I build a Docker Image?

10

Jenkins & Docker

How Can You Use Jenkins & Docker Together?

+

How Can You Use Jenkins & Docker Together?

1. Run Jenkins Masters & Slaves in Docker

2. Build, Test, & Deploy Docker Images from Jenkins

1. Run Jenkins Masters & Slaves in DockerDocker (Cloud) – use Docker images as standardized build environments to improve isolation and elasticity

Docker Custom Build Environment – specify customized build environments as Docker containers

CloudBees Docker Shared Config – manage Docker (or Swarm) host configuration centrally in CloudBees Jenkins Operations Center

2. Build, Test, & Deploy Docker Images from JenkinsBuild and Publish – build projects that have a Dockerfile and push the resultant tagged image to Docker Hub

Docker Traceability – identify which build pushed a particular container and displays the build / image details in Jenkins

Docker Hub Notification – trigger downstream jobs when a tagged container is pushed to Docker Hub

Jenkins Workflow & Docker

Jenkins Workflow PrimerJenkins powered CD pipelines

Jenkins Workflow

ProdDevPerf Test

BuildCommit Selenium Test Stage Deploy

Sonar Test

Pipelines Need: Branching Looping Restarts

Checkpoints Manual Input

??

Key Workflow Features

18

Entire flow is one concise Groovy script using Workflow DSL• For loops, try-finally, fork-join …

Can restart Jenkins while flow is running

Allocate slave nodes and workspaces• As many as you want, when you want

Stages throttle concurrency of builds

Human input/approval integrated into flow

Standard project concepts: SCM, artifacts, plugins

Jenkins Workflow + Docker

20

Pipeline Stages

Build and Unit Test App

Test Docker Image

Publish Docker Image

SCM Checkoutmvn package

mvn sonar:sonarmvn verify

docker build

docker tag

docker run

notifycucumber

war

img

Sonar Analysi

s

Prepare Release

Build Docker Image

Int Test

docker push

image.inside withServer

21

Build, unit test and package

Build and Unit Test App

Test Docker Image

Publish Docker Image

SCM Checkoutmvn package

mvn sonar:sonarmvn verify

docker build

docker Tag

docker run

notifycucumber

war

img

Sonar Analysi

s

Prepare Release

Build Docker Image

Int Test

docker push

image.inside withServer

Build, unit test and package

stage 'Build App’

node('docker') {

docker.image(‘maven:3.3.3-jdk-8’).inside(‘-v /data:/data’ {

mkdir –p /data/mvn

writeFile file: 'settings.xml', text: ”(………)"

git 'https://github.com/cloudbees/mobile-deposit-api.git’

sh 'mvn –s settings.xml clean package’

…

Specify the Stage Name

Specify the slave labelCustom Build Env Mount volume from

slave

.m2 repo locationco and build

Defining a Docker SlaveSpecify Image as

templateAssign labels

24

Test the app

Build and Unit Test App

Test Docker Image

Publish Docker Image

SCM Checkoutmvn package

mvn sonar:sonarmvn verify

docker build

docker Tag

docker run

notifycucumber

war

img

Sonar Analysi

s

Prepare Release

Build Docker Image

Int Test

docker push

image.inside withServer

Test the app

node('docker') {

docker.image(‘maven:3.3.3-jdk-8’).inside(‘-v /data:/data’ {

…

stage 'Sonar analysis’

sh 'mvn -s settings.xml sonar:sonar’

stage 'Integration-test’

sh 'mvn -s settings.xml verify’

step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml'])

}

…

In same env as build

Sonar tests

Run API Tests

26

Build, test and publish Docker image

Build and Unit Test App

Test Docker Image

Publish Docker Image

SCM Checkoutmvn package

mvn sonar:sonarmvn verify

docker build

docker Tag

docker run

notifycucumber

war

img

Sonar Analysi

s

Prepare Release

Build Docker Image

Int Test

docker push

image.inside withServer

Build, test and publish Docker image

docker.withServer('tcp://192.168.99.100:2376', 'slave-docker-us-east-1-tls'){stage 'Build Docker image’

def mobileDepositApiImage

dir('.docker') {sh "mv ../target/*-SNAPSHOT.jar mobile-deposit-api.jar”

mobileDepositApiImage = docker.build "harniman/mobile-deposit-api:$

{buildVersion}”

} stage 'Test Docker image’

container=mobileDepositApiImage.run("--name mobile-deposit-api -p 8080:8080”)

sh "curl

http://<user>:<token>@<host>:8080/docker-traceability/submitContainerStatus \......// insert cucumber tests here

stage 'Publish Docker image’

withDockerRegistry(registry: [credentialsId: 'dockerhub-harniman']) {mobileDepositApiImage.push()

}

}

Bind to docker host

Change directory

Launch container

Build docker image

Bind to registryPush image

Submit traceability report

28

Tagged Version

aTagged Image in Docker Huba

Traceability

30

Traceability

Builds on existing Jenkins artifact traceabilityAllows the tracking of the creation and use of Docker containers in Jenkins and their future use.Combine with artifact fingerprinting for a comprehensive solutionEach Build shows the image fingerprints created

Identify which build pushed a particular container and display the build / image details in Jenkins

Image fingerprints

31

Traceability – registering eventsJenkins can track actions against this image such as:

• Creating a container• Container events such as start/stop

To achieve this, it is necessary to call the Traceability API – see $(JENKINS_URL)/docker-traceability/api/There are two endpoints to submit events to:/docker-traceability/submitContainerStatus

Allows to submit the current container status snapshot with a minimal set of parameters. Outputs of docker inspect $(containerId) can be directly submitted to Jenkins server using this command.

/docker-traceability/submitReport

Submits a report using the extended JSON API. This endpoint can be used by scripts to submit the full available info about the container and its environment in a single command.

32

Traceability – registering events - exampleWorkflow usage example:

container = mobileDepositApiImage.run("--name mobile-deposit-api -p 8080:8080")

sh "curl http://<user>:<token>@<host>:8080/docker-traceability/submitContainerStatus \

--data-urlencode status=deployed \--data-urlencode inspectData=\"\$(docker inspect $container.id)\" \--data-urlencode environment=test \--data-urlencode hostName=mymac \--data-urlencode imageName=harniman/mobile-deposit-api"

Spin up container

Notify Jenkins

33

Docker Traceability View

Docker TraceabilityContainer

Container Use View

34

Deployment Events

Link to Build

Dockerhub Notifications

36

Docker Hub NotificationTrigger downstream jobs when a tagged container is pushed to Docker Hub

The Docker Hub Notification Trigger plugin lets you configure Jenkins to trigger builds when an image is pushed to Docker Hub. E.g. to run verification for the container.

What are the steps

Set up a WebHook Account for Notification

Set up your Docker Registry to make callbacks on Image events

Set up your builds

37

Docker Hub Notification – Docker Registry WebhookIn the format:http://<user>:<token>@<jenkins_url>/dockerhub-webhook/notify

38

Docker Hub Notification – Job Set up

Configure Trigger

In Conclusion

40

Docker and Jenkins with Workflow is the proven CD Platform

+

TESTING

STAGING

PRODUCTIONWorkflow CD Pipeline Triggers:• New application code (i.e. feature, bug, etc.)• Updated certified stack (security fix in Linux, etc.)

… will lead to a new gold image being built and available for…… TESTING

… STAGING… PRODUCTION

All taking place in a standardized/similar/consistent environment

<OS config>

Company“Gold”

Docker Img(~per app)

App<code>

(git, etc.)<OS config>

Certified DockerImages

(Ubuntu, etc.)

Jenkins Workflow

CloudBees: Leading the Way for Docker and CD

Docker Workflow – Provides first-class support for Jenkins Workflow to build real world CD pipelines for containerized applications using Jenkins and Docker

Build and Publish – Builds projects that have a Dockerfile and pushes the resultant tagged image to Docker Hub

Docker Hub Notification – Triggers downstream jobs when a tagged container is pushed to Docker Hub

Docker Traceability – Identifies which build pushed a particular container that is running in production and displays that on the Jenkins builds page

Docker – Uses Docker containers as standardized build environments to improve isolation and elasticity – Dockerized Build Slaves

Docker Custom Build Environment – Specifies customized build environments as Docker containers

Getting started

Docker plugin documentation:http://documentation.cloudbees.com/docs/cje-user-guide/docker-workflow.html

Workflow tutorial:https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md

Example Source Codehttps://github.com/harniman/mobile-deposit-api/blob/master/flow.groovy

How Do You Manage CD at Enterprise Scale?

43

CloudBees Jenkins PlatformJenkins at Enterprise Scale for CI and CD

Thank you!Nigel Harniman@harnimannharniman@cloudbees.com