Going literate in Amadeus JUC Berlin June 25th 2014

Jenkins User Conference Europe #jenkinsconf

Vincent LatombeAmadeus IT Grouphttp://www.amadeus.com

June 25, 2014

Going Literate in Amadeus

#jenkinsconf

Jenkins User Conference Europe #jenkinsconf

About me

Vincent Latombe a.k.a vlatombeDeveloper advocate at Amadeus

Used to build for Java/JEE env., now also C++/Python/RubyHeavy Jenkins user since 2010Maintainer of the Clearcase pluginContributes to core (windows fixes), git and literate

Jenkins User Conference Europe #jenkinsconf

Agenda

What is Literate ?Literate in PracticeLiterate in AmadeusUnder the hood

Jenkins User Conference Europe #jenkinsconf

WHAT IS LITERATE?Part I

WTF IS LITERATE???

Jenkins User Conference Europe #jenkinsconf

Literate Programming

• Introduced by Donald Knuth in 1983

• Explain program logic using natural language with snippets of code and macros

Jenkins User Conference Europe #jenkinsconf

Literate Builds

• Applies Literate paradigm to builds

• Describes the build steps in a marker file or simply in README.md

• Build definition stored in the same SCM as the code it is building

Jenkins User Conference Europe #jenkinsconf

Literate Builds

• Marker file within the SCM

• May contain the build definition

• Fallback to README.md if empty

Jenkins User Conference Europe #jenkinsconf

Markdown Cheatsheet

A section=========This is text inside section.And `this is inline code`.

some code

Jenkins User Conference Europe #jenkinsconf

Hello world exampleHello world literate project============================

Build-----

Let's say hello

echo "Hello world"

Jenkins User Conference Europe #jenkinsconf

Ant example

Environments===========Notice how we specify the environment to build with by providing Jenkins node labels or tool installer names in code snippet sections attached to bullet points in an "environment" section?

* `ant-1.8`, `java-1.7`

Build=====

ant clean dist

Jenkins User Conference Europe #jenkinsconf

Matrix build? No problem

Complex project===============

Environments===========

We have two different environments that the build must be run on:

* `linux`, `gcc-4.2`, `ant-1.8`, `maven-3.0.5`, `java-1.7`* `windows`, `vs-pro-2012`, `ant-1.8`, `maven-3.0.5`, `java-1.7`

Jenkins User Conference Europe #jenkinsconf

Build commands per environment

Build=====

We have two different sets of build instructions, one for building with visual studio and the other for building with GCC

* On `gcc-4.2`, we start by building the native code

./configure make

* On `vs-pro-2012`, we have a batch file to do the native steps

call build-native.bat

Jenkins User Conference Europe #jenkinsconf

Additional tasks

• Defined in the marker file• Enabled per branch using the job

configuration• Lightweight promotion– Self-promotion– Manual promotion (with optional parameters)

Jenkins User Conference Europe #jenkinsconf

Extensions

• Files stored in SCM under .jenkins folder• Basic: <extensionName>.xml, contains

the config snippet• Implement Agent to expose higher-level

configuration files

Jenkins User Conference Europe #jenkinsconf

LITERATE IN PRACTICEPart II

Jenkins User Conference Europe #jenkinsconf

DEMO TIME !

Jenkins User Conference Europe #jenkinsconf

LITERATE IN AMADEUSPart III

Jenkins User Conference Europe #jenkinsconf

Our context

• About 3000 developers• Commercial products– ~1000 Java/JEE– ~2000 C++

• For internal tooling, add– Python– Ruby– Perl

Jenkins User Conference Europe #jenkinsconf

It’s about control!

• Fully open• Locked down• Balanced– Template-based solutions: only defined

attributes can be customized– Literate: commands can be customized, post-

build actions can be whitelisted

Jenkins User Conference Europe #jenkinsconf

Infrastructure

Stash Jenkins

1.Enable CI

2.Creates

3.Index branches

4.Parses marker

Literate Multi-branch project

Literate branch project

Project/Repository

Literate env.

project

Literate env.

project

5.Build status

Jenkins User Conference Europe #jenkinsconf

YAML Parser

• Markdown considered too textual, too much space for syntax errors, even if using README.md was tempting

• We have some YAML lovers internally ;-)• Easy to implement thanks to existing

Literate architecture

Jenkins User Conference Europe #jenkinsconf

Markdown vs YAMLEnvironments============

* `windows`, `java-1.6`* `linux`, `java-1.7`

Build=====

mvn –B clean verify

environments: - [windows, java-1.6] - [linux, java-1.7]build: mvn –B clean verify

Jenkins User Conference Europe #jenkinsconf

UNDER THE HOODPart IV

Jenkins User Conference Europe #jenkinsconf

Architecture

Literate-plugin

Literate-api Branch-api

Scm-api

Jenkins User Conference Europe #jenkinsconf

Literate-api

• Doesn’t depend on Jenkins• Pluggable model builders (Markdown,

YAML, …)

Jenkins User Conference Europe #jenkinsconf

Scm-API

• jenkins.scm.api.SCMSource : provider for hudson.scm.SCM instances

• Able to comprehend multiple heads on a SCM

Jenkins User Conference Europe #jenkinsconf

Branch-api

• Toolkit to handle multi-branch– Dead branch policy– Untrusted branches– Build retention– Throttling

• Foundation for– Multi-branch freestyle project– Multi-branch template project

Jenkins User Conference Europe #jenkinsconf

Literate-plugin

• New extension points– LiterateBranchProperty : decorate branch and

environments with anything (buildwrapper, properties…)

– Agent : Builds Publisher instances from files in the SCM.

Jenkins User Conference Europe #jenkinsconf

Towards a release

• Develop the ecosystem– More plugins integration– Github/Bitbucket/Stash pull request support

• Integration with isolation features– Sandbox commands within a container

Jenkins User Conference Europe #jenkinsconf

Corporate Community

Thank You To Our Sponsors

Platinum Gold

Silver