View
273
Download
12
Category
Preview:
Citation preview
An Open-SourceChef Cookbook CI/CD ImplementationUsing Jenkins Pipelines
Steffen Gebert (@StGebert)
Config Management Camp, Gent, 06.02.2017
2
Agenda
• Context
• Motivation
• Jenkins Pipelines (in general)
• Chef CI/CD using Jenkins Pipelines
About MeResearcher / PhD Student(Software-based Networks)2011 - 2016
Core Team Member2010 - 2013
Server Admin Team Membersince 2011
3
Co-Organizer DevOps Meetup Würzburgsince 2016
4
PHP-based Content Management System
Picture by Benjamin Kott
5
TYPO3 Open Source Project
• Established open-source community since 1997
• TYPO3 Association as non-profit organization
• Server Admin Team responsible for project infrastructure *
Thanks for partiallyfunding my work andcovering travel costs!
*listedonopensourceinfra.org
Picture by Daniel Pötzinger: http://www.typo3-media.com/blog/article/typo3-wallpaper-for-wide-screen.htmlBased on image by Mike Swanson: http://interfacelift.com/wallpaper/details.php?id=1402 Pi
ctur
e by
Ste
fan
Buse
man
n
6
Chef at TYPO3
• Started with Chef in 2011• Started with librarian-chef (monolithic repo)• Took years to get rid of monolithic repo
• Currently operating 45 Debian nodes• Usual tooling: berkshelf, test-kitchen• berkshelf-api-based /universe (connecting to Chef Server 11*)
• Small team of volunteers, varying degree of Chef knowledge
* because of laziness
7
Demo Time!
Picture by annca/ pixabay:https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/
Working with Chef(and knife, and berks, and..)
9
We haveworkflow problems!
Picture by tpsdave / pixabay:https://pixabay.com/en/firefighters-fire-flames-outside-115800/
10
11
12https://blog.twitter.com/2017/the-infrastructure-behind-twitter-scale
13
Related Work
• zts/cooking-with-jenkins• Provides resource to create single Jenkins freestyle jobs per cookbook• Executes pre-defined rake tasks
• chef-solutions/pipeline• Reads cookbooks from a Berksfile enclosed in main chef-repo• Creates single Jenkins freestyle job per cookbook
• Chef Automate• Commercial• Little known to me, what's going on under the hood
14
Survey
• Who’s Chef cookbooks (Puppet modules/etc) are tested in CI?
• Who has an automated release process?
15
Jenkins Pipelines
16
Jenkins Pipelines
17
18
Jenkins PipelinesIntroduction to
20
Jenkins Pipeline DSL
• Groovy DSL• Provides steps, like sh• Defined in the Jenkinsfile• Can use Groovy/Java logic (Chef users like that idea, right?)
• Pipeline plugin (workflow-aggregator)• Formerly called "Workflow"• Open-sourced one year ago• Whole bunch of plugins (10+)
sh "make"sh "make install"
21
Jenkins Jobs as Code?
• Jenkins Job Builder• Python / YAML based• From OpenStack
• Job DSL plugin (job-dsl)• Also Groovy DSL• Supports many, many plugins• Creates only single jobs, not pipelines
# Job DSL examplejob('my-project-main') {scm {git('https://github.com/...')
}triggers {scm('H/15 * * * *')
}publishers {downstream('my-project-unit')
}}
22
Cookbook Pipeline?
sh 'berks install'sh 'foodcritic'sh 'kitchen test'sh 'berks upload'
Picture by BarnImages / pixabay:https://pixabay.com/en/sushi-food-japanese-fish-seafood-789820/
23
Stages
• Allow visual groupingstage('lint') {sh 'foodcritic'sh 'cookstyle'
}stage('resolve dependencies') {sh 'berks install'
}stage('test-kitchen') {sh 'kitchen test'
}stage('upload') {sh 'berks upload'
}
24
Nodes
• Allocates a Jenkins executor (master/slave)
• Optionally with label
node {stage('lint') {sh '..'
}stage('resolve') {sh '..'
}}
node('chefdk') {stage('lint') {..}stage('resolve') {..}
}
Master Agent2Agent1
Executors(node):
Pipeline
Stage
Step
Step
Stage
Step
Step
Stage
Step
Step
25
Ready, Set, Go!
26
Multibranch Jobs
• Scans repo for branches containingJenkinsfile• Automatically creates (deletes) jobs
• Works with plain Git• Works better™ with GitHub / Bitbucket (via API calls)
27
More Steps to Come..
• Jenkins Plugins can contribute DSL steps
28
Global Variables
• Environment variables:
• Information about current build:
• Parameterized build*:
env.BRANCH_NAMEenv.BUILD_URL
params.myparamparams.deployEnv
* well-hidden featuresee https://st-g.de/2016/12/parametrized-jenkins-pipelines
currentBuild.displayName = "v1.2.3"CurrentBuild.rawBuild.getCause()currentBuild.result = 'FAILED'
29
Now Copy & Paste?
Picture by aitroff / pixabay:https://pixabay.com/en/stormtrooper-star-wars-lego-storm-1343772/
30
Pipeline Shared Libraries
• Additional code to be used by Jenkinsfiles• Loaded from SCM repo
• Inclusion via• @Library annotation in Jenkinsfile• Configuration of enclosing folder• Jenkins global configuration*
* then library code is "trusted"
31
Configuring Global Pipeline Library
• Jenkinsfile:
• Load implicitly would load it without @Library• Use non-default branch/tag
@Library('chefci')org.example.Pipeline ...
@Library('chefci@testing')
Don't playwell together
32
Pipeline Shared Libraries
(root)+- src # Groovy source files| +- org| +- foo| +- Bar.groovy # for org.foo.Bar class|+- vars| +- foo.groovy # for global 'foo' variable| +- foo.txt # help for 'foo' variable|+- resources # resource files| +- org| +- foo| +- bar.json # static helper data for org.foo.Bar
https://jenkins.io/doc/book/pipeline/shared-libraries/
static files
feel like functions
the magic –actually code
33
Global Variables
• Can store global state (for the current build)• Can behave like steps• Useful to simplify pipeline code• Hide implementation details from users
# vars/deployTo.groovydef call(def environment) {echo "Starting deployment to ${environment}"withCredentials(..) {sh "rsync –avzh . ${environment}.example.com"
}}
# Jenkinsfilenode {sh "make test"deployTo "production"
}
34
More Magic: Global Library Classes (src/)
• Groovy classes implementing arbitrary logic • Make use of pipeline DSL steps• Use other Jenkins functions• Import and use Java libraries
à How we implement our pipeline
35
# Declarative Jenkinsfilepipeline {agent label:'has-docker', dockerfile: trueenvironment {GIT_COMMITTER_NAME = "jenkins"
}stages {stage("Build") {steps { sh 'mvn clean install' }
}stage("Archive"){// ..
}}post {always {deleteDir()
}success {mail to:"me@example.com", subject:"SUCCESS// ..
Scripted vs. Declarative Pipelines
• Scripted pipelines• Just (imperative) Groovy code• The original implementation• The approach used here
• Declarative pipelines• Hit the 1.0 release last Friday• Can be validated prior to execution• Ease some tasks, i.e., failure handling• Visual Pipeline Editor plugin
36
Jenkins Pipeline Summary
• Groovy DSL for specifying pipelines as code
• Ideally stored within the tested repo
• Extract shared functionality to library repos
• Pipeline execution triggered on push event
jenkins-chefciAn Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
38
Forbidden Commands$ berks upload$ knife cookbook upload$ knife data bag from file$ knife data bag delete$ knife environment from file$ knife environment delete$ knife role from file$ knife role delete
Allowed Commands
$ git pull$ git commit$ git push
Pictures by PublicDomainPictures & serrano / pixabay:https://pixabay.com/en/costume-demon-devil-board-female-15747/https://pixabay.com/en/bebe-girl-child-child-portrait-1237704/
39
Meet the jenkins-chefci cookbook
• Sets up a Chef CI/CD infrastructure using Jenkins• Sets up Jenkins master• Installs ChefDK, configures credentials• Configures job that scans a GitHub organization• Configures our shared pipeline library
https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/
• Add this Jenkinsfile to your cookbooks:# Jenkinsfile@Library('chefci') _org.typo3.chefci.v2.cookbook.CookbookPipeline.builder(this, steps)
.buildDefaultPipeline().execute()
40
Meet the Shared Pipeline Library
• Implements pipelines for• Cookbook testing, versioning & upload• [WIP] Chef-repo's data bags, environments, roles
• Contains couple of Groovy classes• Some helper classes• *Pipeline as entry points from Jenkinsfile• AbstractStage as base class for.. Stages
• Yes, we're at v2 already J
41
Executed Stages & Steps
• Linting• Foodcritic• Cookstyle
• Build• Berkshelf install (endpoints for Supermarket & Chef Server)Berksfile.lock in Git for top-level cookbooks
• Acceptance• Test-Kitchen (using kitchen-docker)
• Publish• Version Bump (thor-scmversion, based on user feedback)• Berkshelf upload (to Chef Server)
Picture by Kaz/ pixabay:https://pixabay.com/en/hands-hand-raised-hands-raised-220163/
42
43
node {createKitchenYaml() stash "cookbook"
}
parallel("essentials-debian-86": { node {
unstash "cookbook"sh "kitchen test --destroy always essentials-debian-86" }},
"essentials-ubuntu-1604": { node { .. } },"full-debian-86": { node { .. } },"full-ubuntu-1604": { node { .. } }
)
Test-Kitchen
• If there is no .kitchen.docker.yml, put default one• Use Jenkins' parallel step
expected result, not hard-coded
44
Test-Kitchen (2)
• Parallel instances automatically derived from kitchen status
• Log files of failed instances are archived• Less chaos than complete Jenkins log• Traditional UI does not clearly show failed parallel branch
• Not limited to kitchen-docker
45* cookbook configures this via API
46
47
48
49
The Art of Cookbook Versioning
• Let's agree on SemVer• I do not agree with manual changes to metadata.rb
50
Demo Time!
Picture by annca/ pixabay:https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/
51
Versioning
• Based on thor-scmversion• Made by RiotGames for their cookbooks• Uses Git tags to derive current version• Version treated as "label", not part of the source code
• Jenkins pushes only Git tags, no code changes
# metadata.rbname "example-cookbook"version IO.read(File.join(File.dirname(__FILE__), 'VERSION')) rescue '0.0.1'
52
Versioning (2)
• Publish stage notifies via Slack
• Except commit message subject indicates version increase• Do this #patch• Add this #minor• Break this #major
53
"Main Chef-Repo"
• Not covered here, but in site-chefcitypo3org cookbook• Freestyle job, defined via JobDSL• Uses script to parse git diff and
upload changes using knife
54
Additional Notes
• Manually configured J organization-level webhook triggersimmediate builds• Everything else is done automatically, no "manual handholding"
Using jenkins-chefci
56
Ease of Use
• Checkout cookbook from github.com/TYPO3-cookbooks/jenkins-chefci
• Increase GitHub API rate limit
• Allow to steal your Chef credentials
• Run test-kitchen
export JENKINS_GITHUB_LOGIN=johndoeexport JENKINS_GITHUB_TOKEN=123456supersecure
export JENKINS_COPY_CHEF_CREDENTIALS=1
kitchen converge full-ubuntu-1604
WARN: allows commitstatus update
WARN: allows knife/berksto access Chef Server
57
(Potential) TODOs for you
• Write your wrapper cookbook around jenkins-chefci• Point it to your organization• Add authentication, e.g., GitHub OAuth?
• Fork the global library• Adjust pipeline to your needs• You don't have to agree with our (current) implementation
58
Outlook / TODOs
• Documentation / blog posts• Move more stuff from site-chefcitypo3org to jenkins-chefci• chefdk() global function to run in chef/chefdk Docker container• Store Chef private key as Jenkins credential• Test using multiple chef-client version• Use JobDSL for organization-folder setup (#966)• Trigger downstream cookbooks
• Read out dependencies and subscribe to successful pipeline runs• Paves the road to Policyfiles
• Use Jenkins slaves• Collect chef-client deprecation warnings
59
Conclusion
• Jenkins Pipelines allow to define pipelines as code• Groovy-based DSL allows programming of pipelines• Can definitively become complex, i.e., debugging
• Jenkins Pipelines for Chef cookbook CI/CD• Running at TYPO3 since May 2016 (site-chefcitypo3org cookbook)• Public instance at https://chef-ci.typo3.org, open source from day 1• Warning: Many cookbooks still use v1 pipeline (git-flow)
• jenkins-chefci cookbook as reusable implementation• Makes setup accessible to broader audience• No dependencies to other TYPO3 cookbooks
60
Further Reading• TYPO3's Chef CI:
https://chef-ci.typo3.org• TYPO3-cookbooks on GitHub:
https://github.com/TYPO3-cookbooks
• TYPO3's Shared Global Library:https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/
• Pipeline Tutorial:https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md
• Getting started with pipelines:https://jenkins.io/pipeline/getting-started-pipelines/
• Step documentation:https://jenkins.io/doc/pipeline/steps/
• Pipeline shared libraries:https://jenkins.io/doc/book/pipeline/shared-libraries/
• Notifications (Mail, Slack, etc.) in Scripted Pipelines:https://jenkins.io/blog/2016/07/18/pipline-notifications/
• Declarative Pipelineshttps://jenkins.io/blog/2016/12/19/declarative-pipeline-beta/https://jenkins.io/blog/2017/02/03/declarative-pipeline-ga/
Recommended