Orchestrated Functional Testing with Puppet-spec and Mspectator

ORCHESTRATED FUNCTIONALTESTING WITH PUPPET-SPECAND MSPECTATOR

RAPHAËL PINSON

Who am I?Raphaël Pinson (@raphink)

■ Infrastructure Developer & Trainer @ Camptocamp

■ Augeas & Augeasproviders developer

■ Various contributions to Puppet & ecosystem

www.camptocamp.com / 2/32

Monitoring vs. Functional TestsComplementary or redundant?


Conformity Tests

■ Check if machines comply to standards

■ Avoid permanent heavy monitoring checks

■ Tests must be inter-dependent

■ Focus on getting sysadmins to fix one thing at a time to convergetoward standards


Treetester■ Back in 2008

■ Written in Perl

■ Orchestrate conformity tests on a 4k+ server fleet


Treetester: modules output■ For all hosts/modules

■ Number of hosts filtered per module

■ Modules dependency tree

■ Colors by priority


Treetester: host output■ For each host

■ Failed steps in the module tree

■ Green: OK, Red: KO, Purple: Ignored


Treetester architecture■ All data in a database (MySQL)

■ Tests scripts output YAML

■ Tests scripts can be local (hosts as STDIN) or remote (ssh or http)

■ Tests are inter-dependent

■ Generate filtered data as a tree

■ Generate graphs (graphviz)


Treetester filters■ For each test/module

■ Based on data in MySQL (joins and additional SQL conditions)

■ Allows to link tests to each other

■ Like multiple sieves


Treetester: future?■ Not open-sourced :'-(

■ Too monolithic/not flexible enough

■ Heavily linked to specific architecture

■ Needed a rewrite


Adding specs to Puppet runs■ Testing the catalog before it gets applied

■ Testing the node after the catalog is applied

Enter the Puppet-spec module


https://github.com/raphink/puppet-spec

Rspec-puppet■ http://rspec-puppet.com

■ Now the standard to unit test Puppet manifests

■ Generates catalogs in clean environments

■ Asserts catalogs for resources/classes

requirerequire 'spec_helper'

describe 'logrotate::rule' dodo

let(:title) { 'nginx' }

it { should compile.with_all_deps }

it { should contain_class('logrotate::setup') }

endend


Puppet-spec■ Runs tests from within Puppet runs

■ Test catalogs using rspec-puppet

■ Test hosts using serverspec


Puppet-spec: Unit testing■ Catalog exposed by PuppetSpec::Catalog.instance.catalog

■ Uses rspec-puppet matchers

■ Asserts real catalogs

■ Runs on the master or agent side (as catalog indirection terminii)

describe 'puppet' dodo

subject { PuppetSpec::Catalog.instance.catalog }

it { should contain_package('puppet') }

it { should contain_package('ppet') }

it { should include_class('puppet') }

it { should include_class('puppet::client::base') }

endend


Puppet-spec: Unit tests output# puppet agent -tinfo: Retrieving pluginerr: Could not retrieve catalog from remote server: Unit tests failed:F..

Failures:

1) packageFailure/Error: it { should contain_package('augeas') }

expected that the catalogue would contain Package[augeas]# /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3# /var/lib/puppet/lib/puppet/indirector/catalog/rest_spec.rb:31:in `find'

Finished in 0.00092 seconds3 examples, 1 failure

Failed examples:

rspec /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3 # package

info: Not using expired catalog for foo.example.com from cache; expired at Tue Apr 02 17:40:21 +0200 2013notice: Using cached catalog


Puppet-spec: Deploying unit tests■ On the master side:○ Tests are located in the spec/catalog/class directory of the

environment○ Only the directories named after classes declared in the catalog

are tested

■ On the agent side:○ Deploy tests using pluginsync○ Tests are located in the lib/spec/catalog/class directory of each

module○ Only the directories named after classes declared in the catalog

are tested


Puppet-spec: Unit tests limits■ When to apply the tests (currently based on class names)

■ Tests on master, or need to deploy all tests with pluginsync

■ Redundant with existing unit tests, or additional security?


Puppet-spec: Setting up Unit testing■ Tests achieved from catalog indirection terminii

■ Plugins (terminii) deployed with pluginsync

■ Setup done in routes.yaml:

agent:

catalog:

# Either on the agent side

terminus: rest_spec

cache: yaml

master:

catalog:

# Or on the master side

terminus: compiler_spec


Serverspec■ http://serverspec.org

■ Provides RSpec matchers for local functional tests (packages,users, services, ports, etc.)

■ Independant from configuration management tools


describe service('httpd') dodo

it { should be_enabled }

it { should be_running }

endend

describe port(80) dodo

it { should be_listening }

endend

describe file('/etc/httpd/conf/httpd.conf') dodo

it { should be_file }

its(:content) { should match /ServerName www.example.jp/ }

endend


Serverspec backendsAllows to use various means of launching tests:

■ SSH (default)

■ Exec

■ Puppet (RAL, removed from core)

$ serverspec-init

Select OS type:

1) UN*X

2) Windows

Select number: 1

Select a backend type:

1) SSH

2) Exec (local)

Select number: 1


Puppet-spec: Functional testing■ Uses serverspec/specinfra matchers

■ Tests the machine state (not the catalog)


describe service('httpd') dodo

it { should be_enabled }

it { should be_running }

endend

describe port(80) dodo

it { should be_listening }

endend

describe file('/etc/httpd/conf/httpd.conf') dodo

it { should be_file }

its(:content) { should match /ServerName www.example.jp/ }

endend


Puppet-spec: Function tests output# puppet agent -tinfo: Retrieving plugininfo: Caching catalog for foo.example.cominfo: Applying configuration version 'raphink/a2c8e0f [+]'... Applying changes ...notice: Finished catalog run in 59.19 secondserr: Could not send report: Unit tests failed:FF

Failures:

1) augeasFailure/Error: it { should be_installed }

expected "augeas" to be installed# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'

2) /usr/share/augeas/lenses/distFailure/Error: it { should be_file }

expected "/usr/share/augeas/lenses/dist" to be file# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'

Finished in 0.06033 seconds2 examples, 2 failures

Failed examples:

rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2 # augeasrspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6 # /usr/share/augeas/lenses/dist


Puppet-spec: Deploying functionaltests

■ Tests are run after catalog application

■ Tests can be distributed via pluginsync (in the spec/server/class)directory of each module

■ Tests can be distributed with file Puppet resources, optionallyusing the spec::serverspec defined resource type


Puppet-spec: MCollective agent■ Communicates with distant nodes

■ Sends action and values to specinfra check commands

■ Does not implement serverspec syntax

■ Returns true/false

■ Uses MCollective as transport (instead of SSH)

Examples:

$ mco rpc spec check action=user values=rpinson$ mco rpc spec check action=file values=/etc/passwd$ mco rpc spec check action=resolvable values=google.fr,A$ mco rpc spec check action=listening values=80$ mco rpc spec check action=process values=mcollectived$ mco rpc spec check action=file_contain values=/etc/passwd,rpinson


Mspectator■ https://github.com/raphink/mspectator

■ RSpec matchers

■ Calls MCollective to achieve tests

■ Uses MCollective spec agent (among others)


Mspectator architecture■ Client runs RSpec

■ RSpec calls MCollective

■ MCollective calls distant spec agent

■ spec agent calls specinfra backend


Mspectator syntaxOwn matchers, mapping to specinfra backend methods:

requirerequire 'mspectator'

describe 'apache' dodoit { should find_nodes(100).or_less } # Counts discovered nodesit { should pass_puppet_spec } # Runs the `spec` agentit { should have_certificate.signed } # Uses the `puppetca` agent

context 'when on Debian',:facts => { :operatingsystem => 'Debian' } dodo # Filter by facts

it { should find_nodes(5).with_agent('spec') }it { should have_package('apache2.2-common') }it { should_not have_package('httpd') }it { should have_service('apache2').with(

:ensureensure => 'running') }it { should have_file('/etc/apache2/apache2.conf') }it { should have_directory('/etc/apache2/conf.d') }it { should have_user('www-data') }

endend

context 'when using SSL', :classes => ['apache::ssl'] dodo # Filter by classesit { should find_nodes(50).or_more }it { should have_package('ca-certificates') }

endendendend


Mspectator output$ rake spec SPEC=apache_spec.rb/home/rpinson/.rvm/rubies/ruby-1.8.7-p371/bin/ruby -S rspec apache_spec.rb

apacheshould find nodes 100should pass puppet spec (FAILED - 1)should have certificatewhen on Debianshould find nodes 5 (FAILED - 2)...

when using SSLshould find nodes 50 (FAILED - 3)

No request sent, we did not discover any nodes. should have package "ca-certificates"

Failures:

1) apacheFailure/Error: it { should pass_puppet_spec }

expected that all hosts would pass tests, the following didn't:soekris01.wrk.cby.camptocamp.com:soekris02.wrk.cby.camptocamp.com:

# ./apache_spec.rb:5

...


Mspectator demo


ContributeOn GitHub:

■ puppet-spec: https://github.com/raphink/puppet-spec

■ mspectator: https://github.com/raphink/mspectator


Thank you!■ [email protected]

■ @raphink on Twitter/Github

■ raphink on Freenode

■ Slides: slideshare.net/raphink


Engineering

Orchestrated Functional Testing with Puppet-spec and Mspectator