Test-driven Infrastructure

Jon Topper

What is Infrastructure?★ Physical Servers

★ Virtual / Cloud Servers

★ Switches

★ Firewalls

★ Routers

★ Load Balancers

3 Year Infrastructure Lifecycle

Operate86%

Build14%

ChangeRisk!

The Rise of DevOps

★ Increased collaboration between developers and operations staff

★ Improved tooling for automation

★ “Dev” solutions to “Ops” problems

Infrastructure as Code

Puppetnode 'webserver' { package { 'httpd': ensure => latest }

file { '/etc/httpd/httpd.conf': require => Package['httpd'], owner => root, mode => 644, content => template('httpd.conf') }

service { 'httpd': ensure => running, enable => true, require => File['/etc/httpd/httpd.conf'] }

}

“Dev” Tooling★ IDEs, text editors, refactoring tools

★ Version Control Systems

★ Automated documentation generation

★ ... Testing?

Automated Infrastructure Testing★ cucumber-puppet / rspec-puppet

★ cucumber-nagios

★ puppet-lint

★ cucumber-chef

★ vagrant-guard-demo

Cucumber ExampleScenario: Basic install of Apache Given there is a running VM called "server" When I apply a puppet manifest containing: """ include cucumber_defaults class { sf_apache: 'Port' => '80', 'Children' => '10' } """ Then a second manifest application should do nothing

And there should be 11 processes called “httpd” running And the Apache module "core_module" should be loaded And a process called “httpd” should be listening on TCP port 80 And a GET request to http://localhost/server-status/ should return an http status of 200

Given there is a running VM called "server"

★ Template (“Box”) based virtual environment

★ Shared filesystem between host and guest

★ Snapshot support via “Sahara” plugin

★ API for scripted interaction

http://vagrantup.com/

Vagrant

Given there is a running VM called "server"

Given /^there is a running VM called "([^"]*)"$/ do |vm_name|

vm_platform.vm( vm_name ).start vm_platform.vm( vm_name ).snapshot

end

attr_reader :last_vm

def initialize @name_map = {}end

def vm(name)

if @name_map.has_key?(name) @last_vm = @name_map[name] return @name_map[name] end

vm = create_vm_object_by_name( name )

@name_map[name] = vm @last_vm = vm

return vm

end

def clean_tainted @name_map.each { |name,vm| vm.rollback @name_map.delete(name) }end

★ Fragment uploaded with SCP

★ Puppet tasks run over Vagrant SSH link

★ Included manifests read from Vagrant shared folder

When I apply a puppet manifest containing: """ include cucumber_defaults class { sf_apache: 'Port' => '80', 'Children' => '10' } """ Then a second manifest application should do nothing

When /^I apply a puppet manifest(#{VMRE}) containing:$/ do |vmre, manifest_content|

vm = identified_vm( vmre )

file = Tempfile.new('cucumber-puppet') begin file.write(manifest_content) file.fsync vm.upload(file.path,'/tmp/cucumber-puppet.pp')

@puppet_command ="puppet apply --verbose --modulepath=#{$puppet_modulepath} " + "--manifestdir=#{$puppet_manifestdir} --detailed-exitcodes --color=false " + "/tmp/cucumber-puppet.pp"

exit_status = vm.sudo( @puppet_command ) do |type,data| data.chomp! puts data if data != “” end

Test::Unit::assert( exit_status == 0 || exit_status == 2, 'Exit code of puppet run not 0 or 2 - errors' )

ensure file.close file.unlink endend

VMRE

VMRE ||= /(?: on the last VM| on the VM(?: called|) "(?:[^"]+)"|)/

def identified_vm( str ) case str when /^( on the last VM|)$/ return vm_platform.last_vm when /^ on the VM(?: called|) "([^"]+)"$/ return @vm_platform.vm( $1 ) endend

★ ‘freeman’ is an XML-RPC service

★ First call starts a new XML-RPC server in the guest

★ Code shared over Vagrant’s folder system

And a GET request to http://localhost/server-status/ should return an http status of 200

Then /^a GET request to (.+)(#{VMRE}) should return an http status of (\d+)$/ do |url,vmre,status|

vm = identified_vm( vmre )

response = vm.freeman.call('http.GET',url)

assert( response['code'].to_i == status.to_i, "Response code 200 expected from #{url}, " + "received #{response['code']}" )

end

Full Stack

Features

Step Definitions

Cumberbatch Freeman Client

Host Filesystem

Vagrant Library

Vagrant VM

sshdShared Folder

Freeman Server

Benefits★ Cleaner interfaces

★ Improved separation of concerns

★ Increased reusability

★ Rapid troubleshooting

★ Empowering for junior engineers

Challenges★ Scenarios slow to run

★ Difficult to debug when snapshot rolled back

★ Multi-VM VirtualBox unstable on OS X

★ Good use of Cucumber not always obvious to the sysadmin-minded

Jon Topper

jon@scalefactory.com

http://www.scalefactory.com/

Twitter: @jtopper