Applying software engineering

to configuration management

Bart Vanbrabant, PhD

iMinds-Distrinet / Impera

● Research started with PoDIM by Thomas Delaet

● LISA 07: Best paper award with PoDIM

● Master thesis 2008: apply PoDIM concepts to Puppet

● Security in cfg mgmt side step

● LISA 10: A survey of system configuration tools

● Summer 2010: First prototype of Impera tool

● June 2014: My PhD “A framework for integrated configuration

management of distributed systems”

● Late 2014: Open Source release on Github and University

spin-off: https://github.com/impera-io

About

Why configuration management?

● Automate configuration (deployment,

configuration and re-configuration) of software

● Keep all configuration parameters consistent

to avoid errors

● Cloud computing has increased scale and

dynamics

● Continuous deployment increases dynamics

even more

● Application in cloud are distributed systems

(SOA)

Managed applications

● Applications are business driver of why the

operations are there in the first place

● Application architecture:o components

o services that interact

o logical architecture

o abstract deployment architecture

Configuration management

● Current tools focus on managing individual

components on machineso files

o packages

o services

● Abstraction level mismatch betweeno how the application is architected and developed

o how it is operated

● Both configuration management and monitoring

are in function of system resources

Abstraction blend

● Operations already a blend of abstraction

levels

● Mixing very high level with very low levelo e.g. large website on Amazon

o own web servers (managed in function of low-level

resources)

o combined with ELB (services, managed through

API)

● Blend will only increaseo mash up of as-a-service

o missing components self-hosted on virtual machines

One tool (model) to rule them all

● Inconsistencies in configuration cause errors

and downtime

● Many interdependencies between services

and components

● One integrated configuration model

● Highest and lowest abstraction level in a

single model (and everything in between)

Software engineering (SE)

● SE has been raising the level of abstraction

● Our approach: apply concepts used in SE to

cfg mgmt:o object-oriented programming

o encapsulation

o interfaces

o component-based software development

o refinement

o …

Impera tool

● Declarative configuration model

● Object-oriented modelling

● Relations as first class citizen

● Separation of type (interface) and implementation

(refinement) to build up abstractions

● Re-usable modules

● Plug-in mechanism:o transformation in configuration model

o export to 3rd party

o resource handlers

● Transparent orchestration

Entities and relations

entity Server:

number listen_port

end

Entities and relations

entity Server:

number listen_port

end

entity WebServer extends ip::services::Server,

web::Container

string document_root

end

Entities and relations

entity Server:

number listen_port

end

entity WebServer extends ip::services::Server,

web::Container

string document_root

end

std::Host host [1] -- [0:] Server servers

Client clients [0:] -- [1:] Server server

Entities and relations

entity Server:

number listen_port

end

entity WebServer extends ip::services::Server,

web::Container

string document_root

end

std::Host host [1] -- [0:] Server servers

Client clients [0:] -- [1:] Server server

index WebServer(host, listen_port)

Entities and relations

...

entity WebServer extends ip::services::Server,

web::Container

string document_root

end

std::Host host [1] -- [0:] Server servers

Client clients [0:] -- [1:] Server server

index WebServer(host, listen_port)

typedef HttpWebServer as WebServer(listen_port=80)

Instantiating a model

srv1 = std::Host(name=”server-1”, os=fedora::fedora21)

webserver = WebServer(host=serv1, document_root=”/var/www”)

Refinement

implementation apacheFedoraWebServer for WebServer:

p_apache = std::Package(host=host, name=”httpd”, state=”installed”)

f_cfg = std::ConfigFile(host=host, path=”/etc/httpd/conf/httpd.conf”,

content=std::template(“apache/httpd.conf”),

reload=true,

requires=p_apache)

f_svc = std::Service(host=host, name=”httpd”, state=”running”,

onboot=true, requires=[f_cfg, p_apache])

end

implementation apacheFedoraWebServer for WebServer:

p_apache = std::Package(host=host, name=”httpd”, state=”installed”)

f_cfg = std::ConfigFile(host=host, path=”/etc/httpd/conf/httpd.conf”,

content=std::template(“apache/httpd.conf”),

reload=true,

requires=p_apache)

f_svc = std::Service(host=host, name=”httpd”, state=”running”,

onboot=true, requires=[f_cfg, p_apache])

end

implement WebServer using apacheFedoraWebServer

when std::familyof(host.os, "fedora")

Refinement

Refinement

implementation apacheFedoraWebServer for WebServer:

p_apache = std::Package(host=host, name=”httpd”, state=”installed”)

f_cfg = std::ConfigFile(host=host, path=”/etc/httpd/conf/httpd.conf”,

content=std::template(“apache/httpd.conf”),

reload=true,

requires=p_apache)

f_svc = std::Service(host=host, name=”httpd”, state=”running”,

onboot=true, requires=[f_cfg, p_apache])

end

implement WebServer using apacheFedoraWebServer

when std::familyof(host.os, "fedora")

Instantiating a model

srv1 = std::Host(name=”server-1”, os=fedora::fedora21)

webserver = WebServer(host=serv1, document_root=”/var/www”)

Result:

● ensure that the httpd package is installed

● ensure that the httpd.conf file has the content

generated using a template

● ensure that the httpd server is running and starts at

boot

Building a model

● Configuration modules with:o model:

types: entities, relations, …

refinement

o templates

o files

o plugins: model, resource handlers, dependency

managers, …

● Initial model instantiates types

● Refine model until ready

Refinement

Refinement

Derive configuration

● Implicit configuration

● Derive configuration from integrated

configuration modelo firewall rules based on network topology, network

connection and security policy

o monitoring configuration

o backup

o …

● Get this configuration for “free”

Derive configuration

● Use types from generic modules to model

required informationo e.g. logging::LogFile(host=host,

path=”/var/log/httpd/access_log”,

type=”apache”)

o Can be used to configure:

logstash

syslog

awstats

backup

logrotate

...

Plug-ins

● Declarative configuration model

● Integration of imperative code with plug-ins

● Python 3 plug-ins:o typed parameters:

accept parameters from configuration model

navigate configuration model

read-only view

o typed return value: results from imperative code

● Impera schedules plug-in invocation and

supports backtracking

Plug-ins

● Model type system is meta-programmed into

Python

● Each instance is represented by a Python

object

● Access to configuration model by native

code is mediated with proxy wrappers

● Template is a special type of plug-in with

implicit arguments

Plug-in example

demo/plugins/__init__.py:

@plugin

def get_ipaddress(host: “ip::Host”) -> “ip::address”:

# lookup IP in a database

return db.lookup_ip(host.name)

Plug-in example

demo/plugins/__init__.py:

@plugin

def get_ipaddress(host: “ip::Host”) -> “ip::address”:

# lookup IP in a database

return db.lookup_ip(host.name)

main.cf

implementation lookupIP for ip::Host:

self.ipaddres = demo::get_ipaddress(self))

end

implement ip::Host using lookupIP

server1 = ip::Host(name=”server-1”, os=fedora::fedora21)

Orchestration

● Incremental roll-out of configuration changes

● Prune incomplete parts of the model

● Plug-ins can indicate that a parameter is

unknown

● Orchestration is derived automatically based

on availability of configuration parameters

Plug-in example

demo/plugins/__init__.py:

@plugin

def get_ipaddress(host: “ip::Host”) -> “ip::address”:

# lookup IP in a database

ip = db.lookup_ip(host.name)

if ip is None:

return Unknown(source=host)

return ip

Orchestration examples

● Cloudo IP address of VM only available when VM is

provisioned at cloud provider

o Deploy service updates only when VM is available

● OpenStack deploymento Complex service dependencies

o e.g. network service requires uuid allocated by

identity service in its configuration files

Deployment architecture

Deploying changes

● two modes available

● client, server, agento push and pull

o fact renewal

o enforcing dependencies over machine boundaries

● deployo deploy directly, machine per machine

o local or using ssh

o runs embedded agent and server

Demo

● Deploy a Drupal website on Amazon

● Use varnish as loadbalancer for 2 web

servers

● Add a web server

● Switch to ELB

Demo

Configuration demo project:

https://github.com/bartv/cfgmgmt15-demo

Demo

lb-1

db-1

webhost-1 webhost-2

Demo

ELB @ AWS

db-1

webhost-1 webhost-2

Demo

ELB @ AWS

db-1

webhost-1 webhost-2 webhost-3

Status

● Started in 2010

● 5 master students

● Large hybrid cloud demonstrator

(OpenStack and Amazon)

● Manage our own OpenStack:o deployment

o accounts, tenants, networks, routers, …

● Open source on Github

● Basic documentation + tutorial

Roadmap

● Integrate monitoring (management)

● State management and migration

● Increase compilation speed

● Improve documentation

● Automated testing for core and modules

● …

Contact

Bart Vanbrabant bart@impera.io

@bartvanbrabant

https://distrinet.cs.kuleuven.be

Impera tool:

● https://github.com/impera-io/impera

● modules https://github.com/impera-io/*

● http://impera.readthedocs.org/en/latest/

● https://impera.io | @impera_io