How OpenStack is Built - Anton Weiss - OpenStack Day Israel 2016

How Openstack is Built

Ant(on) Weiss - Otomatohttp://otomato.link

How Big is OpenStack?265,850 commits

5,744 developers

54 project teams

585 git repos

OpenStack CI System

• Developed and managed by the OpenStack Infra team

Ansible+Puppet - two invisible hands.■ All configuration of CI infra nodes is managed by Puppet manifests ■ The ‘Puppetmaster’ is actually an Ansible control machine■ Puppetmaster holds playbook + hiera data■ Ansible playbooks trigger ‘puppet apply’ on the nodes

+

Git - the mighty archivist

■ the most widely used distributed open-source VCS

■ Served by ‘cgit’■ Changes submitted with the

help of ‘git-review’ tool

Gerrit – the gatekeeper

■ Originally developed by Shawn Pearce at Google for Android development.

■ Manages git repositories■ Provides code review and

approval rules■ Collaborates with human and

non-human gatekeepers

Zuul – the dispatcher

■ Defines and enforces the gating process

■ Listens to Gerrit events■ Manages the queue and

dependencies■ Dispatches the verification

process to Jenkins (through Gearman)

Gearman - the outsourcer■ a framework to farm out work

to other machines or processes

■ sends jobs to Jenkins through jenkins gearman plugin

Jenkins - the butler :)

■ Continuous Integration Server■ Executes testing jobs, records

executions results■ Jobs performed on slaves■ Jobs created and managed by

Jenkins Job Builder■ Slaves provisioned by

Nodepool

Jenkins Job Builder■ Configures Jenkins jobs using YAML files stored in git■ Defaults, macros and job configs

- job: name: example-docs node: node-label

triggers: - zuul

builders: - git-prep - docs

publishers: - scp: site: 'scp-server' files: - target: 'dir/ectory' source: 'build/html/foo' keep-hierarchy: true - console-log

Nodepool - the provisioner

■ Manages a pool of Devstack images to use in project testing.■ Provisions nodes on HP and Rackspace clouds.■ Makes sure there are always available nodes for each provider type.■ Tears down old instances when tests are completed.

DevStack Gate■ The integration test that verifies that all projects still work together

after the changes■ Provides a full OpenStack installation on a single machine with

Devstack■ Installs Devstack and runs Tempest

Logstash

■ All jenkins job logs are indexed and stored by Logstash

■ Logs pushing also handled by Gearman jobs

■ Elastic Search helps analyze the test results

Wrap Up

■ Git■ Gerrit■ Zuul■ Gearman■ Jenkins Job Builder■ Jenkins■ Devstack■ Logstash

Only 9 out of 32 projects/tools managed by the Openstack InfraTeam