Scaling Jenkins with Docker and KubernetesCarlos
Sanchez@csanchez
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
FooterContainers & micro services
#jenkinsconf
#jenkinsconf
Footer
Scale
high availability distributed apps
Google starts 20billion/week
#jenkinsconf
Footer
But it is not trivial
#jenkinsconf
Footer
But it is not trivial
#jenkinsconf
Footer
But it is not trivial
#jenkinsconf
Footer
Service orientedthink about services, not machines
#jenkinsconf
Footer
Unix processes - lightweight VMs
#jenkinsconf
#jenkinsconf
FooterDockerLinux containersUnion File SystemFile
SystemUsersProcessesNetwork
@YourTwitterHandle#jenkinsconf
#jenkinsconf
FooterDockerLinux requiredbutDocker Machine (formerly
Boot2Docker) to the rescueOS XWindows
#jenkinsconf
#jenkinsconf
FooterDockerBuild once, run anywhere (kind of)Bare metalVirtual
MachinesCloudDocker
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
Footerdeveloper orientedDependency hellinstallation nightmaresit
ran on my machine
#jenkinsconf
#jenkinsconf
Footerops orientedno need to know internals of appsfocus on OPs
problems(scale, monitoring,)clearer deliverables from dev
#jenkinsconf
#jenkinsconf
FooterDocker deliveryDocker imageDocker
infrastructureAppSystem
@YourTwitterHandle#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
FooterDockerdocker run -t -i ubuntu /bin/bashdocker ps -adocker
build -t csanchez/test .
#jenkinsconf
#jenkinsconf
FooterRocketApp Container ImageApp Container RuntimeApp
Container Discovery
#jenkinsconf
#jenkinsconf
FooterContainer Optimized Operating Systems
#jenkinsconf
#jenkinsconf
FooterContainer OSMinimal OSContainer optimizedNo package
managerAutomatic atomic updatesEasy rollbacks
#jenkinsconf
#jenkinsconf
Footer
Kubernetes
#jenkinsconf
#jenkinsconf
FooterHow would you design your infrastructure if you couldn't
login? Ever.
Kelsey HightowerCoreOS
#jenkinsconf
Footer
#jenkinsconf
FooterKubernetesContainer cluster orchestrationDocker containers
across multiple hosts(nodes or minions)Higher level APIEnforced
stateMonitoring of endpoints
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
FooterMasterKubernetes API Serverscheduling and
synchronizationetcdKubernetes Controller Manager Serverimplements
replication algorithm watching etcd
#jenkinsconf
#jenkinsconf
FooterNodeDockerKubeletensures state of PodsKubernetes
Proxysimple network proxyetcdSkyDNSElasticSearch + Kibana
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
FooterProvidersGKEAzureVmwareRackspaceoVirtVagrantCloudStackUbuntu
#jenkinsconf
#jenkinsconf
FooterClusterexport KUBERNETES_PROVIDER=gceexport
KUBERNETES_NUM_MINIONS=2cluster/kube-up.sh
#jenkinsconf
#jenkinsconf
Footer
Google Container Engine
#jenkinsconf
FooterGoogle Container Enginegcloud beta container --project
my-project clusters create cluster-1 --machine-type g1-small
--num-nodes 2
#jenkinsconf
#jenkinsconf
Footer
Tectonic by CoreOS
#jenkinsconf
#jenkinsconf
FooterNode--- kind: "Node" apiVersion: "v1" metadata: name:
"127.0.0.1" selfLink: "/api/v1/nodes/127.0.0.1" uid:
"8c3192d2-48ef-11e5-8d27-bae1092286ff" resourceVersion: "55118"
creationTimestamp: "2015-08-22T17:02:19Z" labels:
kubernetes.io/hostname: "127.0.0.1" spec: externalID: "127.0.0.1"
status: capacity: cpu: "0" memory: "0" pods: "40" conditions: -
type: "Ready" status: "True" lastHeartbeatTime:
"2015-08-26T19:38:20Z" lastTransitionTime: "2015-08-26T12:18:48Z"
reason: "kubelet is posting ready status" addresses: - type:
"LegacyHostIP" address: "127.0.0.1" nodeInfo: ...
#jenkinsconf
FooterPodGroup of colocated containersSame network
namespace/IPEnvironment variablesShared volumeshost mountedempty
volumesGCE data disksAWS EBS volumes
#jenkinsconf
#jenkinsconf
FooterPodsNodeclientkubeletpodsMaster
NodekubeletpodsNodekubeletpods
@YourTwitterHandle#jenkinsconf
#jenkinsconf
FooterPod--- kind: "Pod" apiVersion: "v1" metadata: name:
"jenkins" labels: name: "jenkins" spec: containers: - name:
"jenkins" image: "csanchez/jenkins-swarm:1.609.2" ports: -
containerPort: 8080 hostPort: 8090 - containerPort: 50000 hostPort:
50000 volumeMounts: - name: "jenkins-data" mountPath:
"/var/jenkins_home" volumes: - name: "jenkins-data" hostPath: path:
"/home/docker/jenkins"
#jenkinsconf
FooterReplication controllerEnsure a number of pods are
runningPod templatesRolling update
podpodpodcontainer 1container 2container 3
#jenkinsconf
#jenkinsconf
FooterReplication controllersNodeclientpodAMaster
NodepodANodepodB
podApodB
@YourTwitterHandle#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
FooterReplication controller--- apiVersion: "v1" kind:
"ReplicationController" metadata: name: "jenkins-slave" labels:
name: "jenkins-slave" spec: replicas: 1 template: metadata: name:
"jenkins-slave" labels: name: "jenkins-slave" spec: containers: -
name: "jenkins-slave" image: "csanchez/jenkins-swarm-slave:2.0"
env: command:
#jenkinsconf
#jenkinsconf
FooterReplication controllercommand: -
"/usr/local/bin/jenkins-slave.sh" - "-master" -
"http://$(JENKINS_SERVICE_HOST):$(JENKINS_SERVICE_PORT)" -
"-tunnel" -
"$(JENKINS_SLAVE_SERVICE_HOST):$(JENKINS_SLAVE_SERVICE_PORT)" -
"-username" - "jenkins" - "-password" - "jenkins" - "-executors" -
"1"
#jenkinsconf
#jenkinsconf
FooterServicesPod discoveryIP per serviceRoute to pods selected
with labelsCan create a load balancer in GCE and AWS
#jenkinsconf
#jenkinsconf
FooterServicesclientpodService
podpod
@YourTwitterHandle#jenkinsconf
#jenkinsconf
FooterServices--- apiVersion: "v1" kind: "Service" metadata:
name: "jenkins" spec: type: "NodePort" selector: name: "jenkins"
ports: - name: "http" port: 8090 nodePort: 32080 protocol:
"TCP"
--- apiVersion: "v1" kind: "Service" metadata: name:
"jenkins-slave" spec: type: "NodePort" selector: name: "jenkins"
ports: - name: "http" port: 50000 nodePort: 32050 protocol:
"TCP"
#jenkinsconf
FooterNetworkingall containers can communicate with all other
containerswithout NATall nodes can communicate with all containers
(and vice-versa) without NATthe IP that a container sees itself as
is the same IPthat others see it as
Containers in a Pod can talk using localhost
#jenkinsconf
#jenkinsconf
FooterNetworkingEvery machine in the cluster is assigned a full
subnetie. node A 10.0.1.0/24 and node B 10.0.2.0/24Simpler port
mappingOnly supported by GCE
CoreOS flannelCreates an overlay network in other providers
#jenkinsconf
#jenkinsconf
FooterRelated projects
#jenkinsconf
#jenkinsconf
FooterDocker MachineProvision Docker enginesVirtualBox, replaces
boot2docker !Amazon EC2Microsoft AzureGoogle Compute
EngineOpenStackRackspaceVMware
#jenkinsconf
#jenkinsconf
FooterDocker SwarmClustering for Docker containersUsing the same
APIIntegrates with Mesos / MesosphereAnd plannedAmazon EC2
Container Service (ECS)Google KubernetesIBM Bluemix Container
ServiceJoyent Smart Data CenterMicrosoft Azure
#jenkinsconf
#jenkinsconf
FooterDocker ComposeOrchestration of multi-container appsBased
on FigDefined by:containersconfigurationlinksvolumes
#jenkinsconf
#jenkinsconf
FooterApache MesosA distributed systems kernel
Docker ContainerizerMarathon & Chronos
#jenkinsconf
#jenkinsconf
FooterFlockerdata volume managermulti-host Docker cluster
management toolZFS volumeshttps://github.com/ClusterHQ/flocker
#jenkinsconf
#jenkinsconf
Footer
Weave
#jenkinsconf
#jenkinsconf
FooterKubernetes and Jenkins
#jenkinsconf
#jenkinsconf
FooterKubernetes cluster with docker-compose# Docker Compose
definition for a one node Kubernetes cluster# Based on Docker
Cookbook example#
https://github.com/how2dock/docbook/ch05/dockeretcd: image:
kubernetes/etcd:2.0.5.1 net: "host" command: /usr/local/bin/etcd
--addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001
--data-dir=/var/etcd/datamaster: image:
gcr.io/google_containers/hyperkube:v1.0.1 net: "host" volumes: -
/var/run/docker.sock:/var/run/docker.sock command: /hyperkube
kubelet --api_servers=http://localhost:8080 --v=2 --address=0.0.0.0
--enable_server --hostname_override=127.0.0.1
--config=/etc/kubernetes/manifestsproxy: image:
gcr.io/google_containers/hyperkube:v1.0.1 net: "host" privileged:
true command: /hyperkube proxy --master=http://127.0.0.1:8080 --v=2
https://github.com/carlossg/kubernetes-jenkins
#jenkinsconf
#jenkinsconf
FooterJenkins master pod kind: "Pod" apiVersion: "v1" metadata:
name: "jenkins" labels: name: "jenkins" spec: containers: - name:
"jenkins" image: "csanchez/jenkins-swarm:1.609.2" ports: -
containerPort: 8080 hostPort: 8090 - containerPort: 50000 hostPort:
50000https://github.com/carlossg/kubernetes-jenkins
#jenkinsconf
#jenkinsconf
Footerstorage options volumeMounts: - name: "jenkins-data"
mountPath: "/var/jenkins_home" volumes: - name: "jenkins-data"
hostPath: path: "/home/docker/jenkins"# gcePersistentDisk:# pdName:
my-data-disk# fsType: ext4# awsElasticBlockStore:# volumeID:
aws:///# fsType:
ext4https://github.com/carlossg/kubernetes-jenkins
#jenkinsconf
#jenkinsconf
FooterJenkins master services--- apiVersion: v1" kind: "Service"
metadata: name: "jenkins" spec: type: "NodePort" selector: name:
"jenkins" ports: - name: "http" port: 8090 nodePort: 32080
protocol: TCP"
--- apiVersion: "v1" kind: "Service" metadata: name: "jenkins"
spec: type: "NodePort" selector: name: "jenkins" ports: - name:
"http" port: 8090 nodePort: 32080 protocol:
"TCP"https://github.com/carlossg/kubernetes-jenkins
#jenkinsconf
#jenkinsconf
FooterJenkins slaves replication pool--- apiVersion: "v1" kind:
"ReplicationController" metadata: name: "jenkins-slave" labels:
name: "jenkins-slave" spec: replicas: 1 template: metadata: name:
"jenkins-slave" labels: name: "jenkins-slave" spec: containers:
https://github.com/carlossg/kubernetes-jenkins
#jenkinsconf
#jenkinsconf
FooterJenkins slaves replication pool - name: "jenkins-slave"
image: "csanchez/jenkins-swarm-slave:2.0" command: -
"/usr/local/bin/jenkins-slave.sh" - "-master" -
"http://$(JENKINS_SERVICE_HOST):$(JENKINS_SERVICE_PORT)" -
"-tunnel" -
"$(JENKINS_SLAVE_SERVICE_HOST):$(JENKINS_SLAVE_SERVICE_PORT)" -
"-username" - "jenkins" - "-password" - "jenkins" - "-executors" -
"1"https://github.com/carlossg/kubernetes-jenkins
#jenkinsconf
#jenkinsconf
FooterJenkins cluster in Kuberneteskubectl get nodeskubectl
create --validate -f pod.ymlkubectl get podskubectl create
--validate -f service-http.ymlkubectl create --validate -f
service-slave.ymlkubectl get serviceskubectl create --validate -f
replication.ymlkubectl get podskubectl scale replicationcontrollers
--replicas=20 jenkins-slave
https://github.com/carlossg/kubernetes-jenkins
#jenkinsconf
#jenkinsconf
FooterKubernetes Jenkins plugin
#jenkinsconf
#jenkinsconf
FooterKubernetes Jenkins pluginAs a pluginon demand slaves
https://github.com/jenkinsci/kubernetes-plugin
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
FooterKubernetes Jenkins pluginpods, not replication
controllersJenkins Cloud APIFabric8 Java APIWorkflow support
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
Footer
#jenkinsconf
#jenkinsconf
FooterRoadmapStable APIWhen Kubernetes Java lib is stableUsing
new Jenkins Cloud/Containers APIs
#jenkinsconf
#jenkinsconf
FooterExample code and slidesAvailable
athttp://slideshare.csanchez.orghttps://github.com/carlossg/kubernetes-jenkins
http://blog.csanchez.org
#jenkinsconf
#jenkinsconf
Footer
Thanks!
#jenkinsconf
FooterThanks to our Sponsors!
#jenkinsconf
Footer
