Using the Pipeline Plugin to Accelerate CD (1)

7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)

1/21

Using the Pipeline Pluginto Accelerate ContinuousDelivery


2/21

Table of Contents!"#$%&'(#!%") #+, -!-,.!", -.'/!" 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 1203 !"4#5..!"/ #+, -!-,.!", -.'/!" 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 6

703 ($,5#!"/ 5 -!-,.!", 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 6"#$%$&' ()*+ ,$,"-$&" ................................................................................................................................................. /*0$&' %1" 0&$,,"% '"&"+2%)+ ..................................................................................................................................... /-)2#$&' "3%"+&2- ,$,"-$&" 04+$,%0............................................................................................................................... 5

103 8'!.&!"/ 9%'$ -!-,.!", 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 :41"46$&' )*% 4)#" .................................................................................................................................................... 5+*&&$&' ()*+ ,$,"-$&" ............................................................................................................................................... 72##$&' 0%2'"0 2 0%",0 ............................................................................................................................................ 7'"&"+2- 8*$-# 0%",0 ................................................................................................................................................... 7

04+$,%$&'

................................................................................................................................................................... 9603 !"#,/$5#!"/ 9%'$ #%%.4 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 ;

%))-0 ........................................................................................................................................................................ 9'-)82- :2+$28-"0 ....................................................................................................................................................... 9"3$0%$&' ;)80 .............................................................................................................................................................. ?12-$&' "++)+0 ..................................................................................................................................................... >?

:03 4($!-# 4,('$!#9 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 22

@03 5((,44!"/ =!.,4 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 27

0%201$&' @$-"0 .......................................................................................................................................................... >A2+41$:$&' ................................................................................................................................................................ >A

;03 4(5.!"/ 9%'$ -!-,.!", 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 2141"46,)$&%0 ............................................................................................................................................................. >B,$,"-$&" %"=,-2%"0 ................................................................................................................................................... >B

?03 #9!"/ !# #%/,#+,$) 45A-., -!-,.!", 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 21

2303 &%(B,$ >!#+ -!-,.!", 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 26

2203 ,C#,"&!"/ -!-,.!", 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 2


3/21

3

USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE

Introduction: The Pipeline PluginJenkins is a powerful, open source automation tool with an impressive plugin architecture

that helps development teams automate their software lifecycle. Jenkins is used to powermany industry-leading companies' software development pipelines.

Jenkins Pipeline is a powerful, first-class feature for managing complex, multi-step pipelines.

Jenkins Pipeline, a set of open source plugins and integrations, brings the power of Jenkins

and the plugin ecosystem into a scriptable Domain Specific Language (DSL). Best of all, like

Jenkins core, Pipeline is extensible by third-party developers, supporting custom extensions

to the Pipeline DSL and various options for plugin integration.

The Pipeline plugin was formerly known as the Workflow plugin prior to version 1.13 of the

plugin.

Image 1: Pipeline Stage View UI

In this whitepaper, we provide an introduction and step-by-step guide on how to use the

Pipeline plugin. We conclude with a full syntex reference card and a real-world delivery

pipeline example, that is built on the more basic Pipeline snippets provided earlier in the

whitepaper.


4/21

4


1.0 Installing the Pipeline PluginIt is assumed that you have already installed Jenkins either via the CloudBees Jenkins

Platform or from the Jenkins open source project website. Jenkins Version 1.609.1+ isrequired.

Open Jenkins in your web browser

Navigate to Manage Jenkins > Manage Plugins

Navigate to the Available tab, filter by Pipeline

Select the Pipeline plugin and install

Restart Jenkins

This whitepaper was written using Pipeline version 1.13. Installing the Pipeline plugin installs

all necessary Pipeline dependencies and a new job type called Pipeline.

2.0 Creating a PipelineNow that you have Jenkins running and have installed the Pipeline plugin, you are ready

to create your first pipeline. Create a new pipeline by selecting New Item from the Jenkins

home page.

First, give your pipeline a name, e.g.: "hello-world-flow." Pipelines are simple Groovy scripts,

so let's add the obligatory Hello World. Add a pipeline to the Pipeline script textarea:

!"#$ &'!(($ )$*(+&

Now save your pipeline, ensuring the Use Groovy Sandbox option is checked (more details

to follow on this setting). Click Build Now to run your pipeline.


5/21

5


Editing Your PipelineBecause pipelines are simple text scripts, they are easy to edit. As you've seen, pipelines can

be edited directly in the Jenkins UI when configuring your pipeline.

Using the Snippet GeneratorTo make editing your pipelines easier, use the Snippet Generator. The Snippet Generator is

dynamically populated with the latest Pipeline steps. Depending on the plugins installed in

your environment, you may see more available steps.


6/21

6


Loading External Pipeline ScriptsBecause pipelines are text assets, they are ideal to store in a source control system. Pipelines

can be edited in your external IDE then loaded into Jenkins using the Pipeline Script fromSCM option.

3.0 Building Your PipelineNow that you've created a pipeline, let's continue to build on it. For a complex flow, you

should leverage Jenkins' job scheduling queue:

,$+!- .# &/,01!&2

The concept of a node should be familiar to Jenkins users: node is a special step that

schedules the contained steps to run by adding them to Jenkins' build queue. Even better,requesting a node leverages Jenkins' distributed build system. Of course, to select the right

kind of node for your build, the node element takes a label expression:

,$+!3&/,45 66 789:4;&


7/21

7


Running Your Pipeline

Because pipelines are built as Jenkins jobs, they can be built like other jobs. You can use theBuild Now feature to manually trigger your build on-demand or set up triggers to execute

your pipeline based on certain events.

Adding Stages and StepsStages are usually the top most element of Pipeline syntax. Stages allow you to group your

build step into its component parts. By default, multiple builds of the same pipeline can run

concurrently. The stage element also allows you to control this concurrency:

.;0K! &:/4(+&

,$+!- N 2.;0K! ,01!@ &;!.;&E "$,"/**!,"O@ P

,$+!- N 2

.;0K! ,01!@ &+!L($O&E "$,"/**!,"O@ Q

,$+!- N 2

In this example, we have set a limit of three concurrent executions of the test stage and only

one execution of the deploy stage. You will likely want to control concurrency to prevent

collisions (for example, deployments).

Newer builds are always given priority when entering a throttled stage; older builds will

simply exit early if they are preempted.

General Build StepsWithin your stages, you will add build steps. Just like with Freestyle Jenkins jobs, build steps

make up the core logic of your pipeline. Jenkins Pipeline supports any compatible Build Step

and populates the snippet generator with all available Build Steps in your Jenkins

environment.

.;!L3>?"(0..@ &R0S0+$"T*"#4S!*&E U0S0+$"V4*@ &;0*K!;G*!.$/*"!.GU0S0+$"&E =!!LT((@

J0(.!H?"(0..@ &W4,K!*L*4,;!*&E ;0*K!;.@ &;0*K!;G0L4M)0*&H


8/21

8


ScriptingJenkins Pipeline supports executing shell (*nix) or batch scripts (Windows) just like freestyle

jobs:

.# &.(!!L QX&

:0; &;41!$/; G; QX&

Scripts can integrate with various other tools and frameworks in your environment - more to

come on tools in the next section.

4.0 Integrating Your ToolsFor a real-life pipeline, Jenkins needs to integrate with other tools, jobs and the underlyingenvironment.

ToolsJenkins has a core capability to integrate with tools. Tools can be added and even

automatically installed on your build nodes. From Pipeline, you can simply use the tool DSL

syntax:

+!J1S,'$1! Y;$$( &DP&

.# Z?-1S,'$1!2G:4,G1S, 9[ S!*4JOZ

In addition to returning the path where the tool is installed, the tool command ensures thenamed tool is installed on the current node.

Global VariablesThe env global variable allows accessing environment variables available on your nodes:

!"#$ !,SM\T]'

Because the env variable is global, changing it directly is discouraged as it changes the

environment globally, so the withEnv syntax is preferred (see example in Full Syntax

Reference Card at the end of this whitepaper).

The currentBuild global variable can retrieve and update the following properties:

"/**!,;[/4(+M*!./(;

"/**!,;[/4(+M+4.L(0O^01!

"/**!,;[/4(+M+!."*4L;4$,


9/21

9


Existing JobsExisting jobs can be triggered from your pipeline via the build command (e.g.: build 'existing-

freestyle-job'). You can also pass parameters to your external jobs as follows:

+!JU$: Y:/4(+ U$:@ &.0O9#!(($&E L0*01!;!*.@ >>?"(0..@ &B;*4,K\0*01!;!*_0(/!&E ,01!@

&)#$&E S0(/!@ &`#4;!L0L!* I!0+!*.&HH

5.0 Controlling FlowBecause the Pipeline plugin is based on the Groovy language, there are many powerful flow

control mechanisms familiar to developers and operations teams, alike. In addition to

standard Groovy flow control mechanisms like if statements, try/catch and closures there are

several flow control elements specific to Pipeline.

Handling ApprovalsPipeline supports approvals, manual or automated, through the input step:

4,L/; &T*! O$/ ./*!a&

With the submitter parameter, the input step integrates the Jenkins security system to restrict

the allowed approvers.

The input step in Jenkins Pipeline Stage View UI:


10/21

10


TimingTimeouts allow pipeline creators to set an amount of time to wait before aborting a build:

;41!$/;3;41!@ PXE /,4;@ &BbCc^VB&< - N 2

Parallel stages add a ton of horsepower to Pipeline, allowing simultaneous execution of build

steps on the current node or across multiple nodes, thus increasing build speed:

L0*0((!( &d/0(4;O ."0,&@ -

,$+! -.# &1S, .$,0*@.$,0*&2

2E &4,;!K*0;4$, ;!.;&@ -

,$+! -.# &1S, S!*4JO&2

2

Jenkins can also wait for a specific condition to be true:

)04;e,;4( - N 2

Handling ErrorsJenkins Pipeline has several features for controlling flow by managing error conditions in your

pipeline. Of course, because Pipeline is based on Groovy, standard try/catch semantics apply:

;*O-

2 "0;"#3!< -

2

Pipeline creators can also create error conditions if needed based on custom logic:

4J3f.$/*"!.< -

!**$* &^$ .$/*"!.&

2

Jenkins can also retry specific Pipeline steps if there is variability in the steps for some reason:

*!;*O3g< - N 2


11/21

11


6.0 Script SecurityAs you've seen, Pipeline is quite powerful. Of course, with power comes risk, so Pipeline has a

robust security and approval framework that integrates with Jenkins core security.

By default, when creating pipelines as a regular user (that is, without the Overall/RunScripts

permission), the Groovy Sandbox is enabled. When the Sandbox is enabled, Pipeline creators

will only be allowed to use pre-approved methods in their flow.

When adding pre-approved methods to a pipeline, script changes do not require approval.When adding a new method (such as a Java API), users will see a RejectedAccessException

and an administrator will be prompted to approve usage of the specific new API or method.

Deselecting the Use Groovy Sandbox option changes this behavior. When the Sandbox

is disabled, pipeline edits require administrator approval. Each change or update by a

non-administrator user requires approval by an administrator. Users will see an

UnapprovedUsageException until their script is approved. Approving individual edits may

not scale well, so the Groovy Sandbox is recommended for larger environments.


12/21

12


7.0 Accessing FilesDuring your pipeline development, you will very likely need to read and write files in your

workspace.

Stashing FilesStashing files between stages is a convenient way to keep files from your workspace to share

them between different nodes:

.;0K! &:/4(+&,$+!-

K4; ;L.@GGK4;#/:M"$1G"($/+:!!.G;$+$90L4MK4;&

.;0.# 4,"(/+!.@ &L$1M51(&E ,01!@ &L$1&

2

.;0K! ,01!@ &;!.;&E "$,"/**!,"O@ P

,$+! -

/,.;0.# &L$1&

.# &"0; L$1M51(&

2

Stash can be used to prevent cloning the same files from source control during different

stages, while also ensuring the same exact files are used during compilation and tested in

later pipeline stages.

ArchivingLike other Jenkins job types, pipelines can archive their artifacts:

0*"#4S! 4,"(/+!.@ &FMU0*&E !5"(/+!.@ &F9.$/*"!.MU0*&

Archives allow you to maintain binaries from your build in Jenkins for easy access later. Unlike

stash, archive keeps artifacts around after a pipeline execution is complete (where stash is

temporary).

Beyond stashing and archiving files, the following Pipeline elements also work with the file

system (more details at the end of this whitepaper):

L)+3

Documents

Using the Pipeline Plugin to Accelerate CD (1)