London Hashicorp Meetup #8 - Testing Programmable Infrastructure By Matt Long

MATT LONG

TESTING PROGRAMMABLE INFRASTRUCTURE

PROGRAMMABLE INFRASTRUCTURE IS GREAT, BUT WE'RE MISSING SOMETHING.

TESTING.

I'M A TESTERHELLO, I'M MATT

I WORK HERE ↑

I AM NOT A SYSADMIN

WHAT IS PROGRAMMABLE INFRASTRUCTURE?


THE APPLICATION OF METHODS AND TOOLING FROM SOFTWARE DEVELOPMENT TO MANAGEMENT OF IT INFRASTRUCTURE

PROGRAMMABLE INFRASTRUCTURE IS..

THE INTERNET


EXAMPLES OF PROGRAMMABLE INFRASTRUCTURE

▸ Automated provisioning & configuration

▸ Configuration as code

▸ Version / source controlled


TOOLING EXAMPLES

PROGRAMMABLE INFRASTRUCTURE IS AWESOME!

Credit: Vault Boy, Bethesda Softworks

IT'S FAST!

IT'S AUTOMATIC!

IT'S ALL CODE!

BUT IT GETS COMPLEX

TESTING IS USED TO MITIGATE COMPLEXITY & RISK

BUT TESTING IS RARE

Credit: Gunshow, KC Green


WHAT I'M GOING TO TALK ABOUT

▸PART 1: Testing a cloud broker

▸PART 2: Building a Kubernetes cluster

▸CONCLUSIONS

TESTING A CLOUD BROKER

AN INFRASTRUCTURE HEAVY PRODUCT

THE PROBLEM


WE WANT TO MOVE TO THE CLOUD...

BUT WE'RE WARY OF LOCK IN

Large organisation


USERS

USE MULTIPLE CLOUD PROVIDERS


PROBLEMS

▸ Different interfaces, feature sets & lingo

▸ Can't switch easily

▸ Spending difficult to track

▸ Temptation to fall back on most popular


USERS

CLOUD BROKER


BENEFITS

▸Quick, easy provisioning

▸ one team previously took 3 months

▸ Common interface to cloud features

▸ Templates for common dev environments

▸ Built in best practice: monitoring, security

▸ Track spending

THIS IS A REALLY COMPLICATED APPLICATION



WORKFLOW

▸ Log into Web UI

▸ Fill in information about environment

▸ Broker creates and bootstraps resources

▸ SSH into resources


WEB TESTING

▸ Log into Web UI

▸ Fill in information about environment


???

▸ Broker creates and bootstraps resources

▸ SSH into resources

HOW DO YOU TEST INFRASTRUCTURE?


WHAT TO TEST?

Do our deployment scripts work?

Does the VPN server work?

Can instances access one another?

Are services running?

Can I SSH into a server?

THIS SEEMS FAMILIAR..


Does the VPN box work?Can I SSH into a server?

Do our deployment scripts work?

Are services running?

ANOTHER TESTING PYRAMID?

credit: Ubuntu dev quality guidehttps://developer.ubuntu.com/en/phone/platform/quality/

Can instances access one another?

TOOLING


TOOLS AVAILABLE

▸ Bats

▸ ShUnit2

▸ Goss

▸ ServerSpec / Inspec / TestInfra

▸ Test Kitchen

UNIT TESTING


BATS

▸ "Bash Automated Testing System"

▸ Unit testing for bash

▸ Like JUnit


SH UNIT 2

▸ Shell unit testing framework

▸ Runs on all Bourne shells

▸ sh, BASH, DASH, ksh, zsh

▸ No activity or support?

INTEGRATION TESTINGOR: SERVER VALIDATION


GOSS

▸ Go based

▸ Specs in YAML

▸ Minimal, fast, and simple

▸ Some neat features

▸ .. have to run on the server

▸ .. no Windows support


SERVERSPEC

▸ Server based assertions

▸ Ruby/RSpec based

▸ Probably the most famous

▸ Can SSH into instances


INSPEC

▸ Written & maintained by Chef

▸ Very similar to ServerSpec

▸ Different feature set

▸ More focused on compliance


TESTINFRA

▸ ServerSpec, but in Python

TEST HARNESS


TEST KITCHEN

▸ Orchestrates setup, test, teardown

▸ Runs BATS, shUnit2, RSpec, Serverspec

▸ Popular in the Chef community

▸ Not suitable for our cloud broker

OUR SOLUTION


USERS

CLOUD BROKER


USERS

WEB TEST FRAMEWORK


USERS

INFRASTRUCTURE TEST FRAMEWORK


USERS

WEB TESTS

https://github.com/opencredo/test-automation-quickstart


INFRASTRUCTURE TESTS

Serverspec


INFRASTRUCTURE TESTING STACK

/ Serverspec

???


WHY RUBY?

▸ Fantastic testing community

▸ More suitable for SSHing into boxes

▸ "Win RM" gem

▸ Ops already familiar with it

▸ Reduces tech stack


SERVERSPEC SMOKE TESTS

▸ Run before everything else

▸ Really quick

▸ Catches obvious errors

▸ Not complex tasks


SERVERSPEC EXAMPLE

describe package('jenkins') do it { should be_installed } end

describe service('jenkins') do it { should be_enabled } it { should be_running } end

describe port(8080) do it { should be_listening } end


Background: Given environment has been created And the following user details: | user_alias | username | public_key | | userA | envoy | test |

Scenario: IPA - Login via SSH Key authentication succeeds Given user "userA" is authorised to access environment vms When user "userA" starts ssh session in host "env"

Then I should be able to echo "hello world"

CUCUMBER FOR ACCEPTANCE TESTING






Cloud broker APIs






Standard Ruby






RSpec assertions


UNDER THE CUCUMBER, PLAIN RUBY

Then(/^I should be able to echo "([^"]*)"$/) do |text| cmd = "echo #{text}" output = @session.exec!(cmd) close_ssh(@session) expect(output.to_s.strip).to eql(text) end

THOUGHTS


THE GOOD

▸ Specialised tests for each layer

▸ Really quick, expressive ServerSpec tests

▸ Power of a full programming language for user tests


THE BAD

▸ Over reliance on acceptance tests

▸ Awkward switching between two suites

▸ Out of my comfort zone


THE UGLY

▸ Starting infrastructure is SLOW.

▸ It's expensive...

IT WAS WORTH ITDESPITE ALL THAT

BUILDING A KUBERNETES CLUSTER

APPLYING TDD TO INFRASTRUCTURE

INTERNAL DEVOPS TRAINING COURSE

I LEARNED A LOT!

Credit: The Simpsons, Fox


BUILD THIS:

WITH THESE:


NOT A STRAIGHTFORWARD TASK


BUT HOW TO TEST IT?

▸ This is a dev activity

▸ Want fast feedback

▸ Complexity is mitigated by testing!


▸ Provisions cloud infrastructure

▸ Declarative files

▸ Some support for variables

TERRAFORM


TERRAFORM COMMANDS

▸ terraform plan

▸ Tells you what will change

▸ terraform apply

▸ Applies changes

▸ terraform validate

▸ Lints terraform syntax


TERRAFORM FILE EXAMPLE

resource "aws_instance" "etcd-node" { count = 3

ami = "ami-7abd0209" # centos availability_zone = "eu-west-1a" # ireland instance_type = "t2.micro" subnet_id = .... private_ip = ....

key_name = "${aws_key_pair.my-key.key_name}" }


LINT WITH 'TERRAFORM VALIDATE' COMMAND

Omitting a variable:


BUT IT DOESN'T CATCH ALL PROBLEMS

Duplicate subnet CIDRS:


LINTING ISN'T ENOUGH

▸ Devs don’t just rely on compilers

▸ We need something more powerful

Credit: Nick Cave, "Soundsuit"

UNIT TESTING


TERRAFORM_VALIDATE

▸ Python based unit testing

▸ NOT to be confused with 'validate' command

▸ Builds map of resources & properties

▸ Totally offline

▸ New and incomplete

https://github.com/elmundio87/terraform_validate

https://github.com/elmundio87/terraform_validate


TERRAFORM_VALIDATE FORK

OC has forked the terraform validate repohttps://github.com/opencredo/terraform_validate

https://github.com/opencredo/terraform_validate

INTEGRATION TESTING


GOSS

▸ Easy to get up and running

▸ Doesn’t support remote

# example usage: ./goss-test.sh 34.248.91.167

TARGET='centos@'$1 SSH_KEY_PATH=~/.ssh/aws

ssh -t -t -i $SSH_KEY_PATH $TARGET 'curl -fsSL https://goss.rocks/install | sudo sh' scp ./goss.json $TARGET:~/goss.yaml

ssh -t -t -i $SSH_KEY_PATH $TARGET 'goss validate'

https://gist.github.com/burythehammer/081d6ee11cc33c2f4c4729ae67622f5b


▸ Terraform compatibility

▸ Already a talk about this

▸ “Untangling Infrastructure Code” by Nell Shamrell-Harrington

TEST KITCHEN + INSPEC


TestCreate Config Destroy ]TEST KITCHEN MANAGES YOUR TEST LIFECYCLE


TestCreate Config Destroy

TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS


TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS

https://github.com/test-kitchen/test-kitchen/issues/329

https://github.com/test-kitchen/test-kitchen/issues/329


TERRAFORM 'NULL RESOURCE'

resource "null_resource" "ansible" { triggers { instance_ids = "${join(",", aws_instance.etcd-node.*.id)}"

}

provisioner "local-exec" { command = "sleep 20 && cd ../ansible/ && ansible-playbook etcd.yaml" }

}


TestCreate

Config

Destroy

THOUGHTS


THE GOOD

▸ Tooling exists!

▸ You can totally get a test suite working

Credit: Overwatch, Blizzard Entertainment


THE BAD

▸ Unit testing extremely immature

▸ Tools immature in general



THE HACKY

▸ Be prepared to hack

▸ It might not even be possible


THIS IS BRAND NEW GROUND

REMEMBER:

TESTING TOOLS DEPEND ON YOUR STACK

CONCLUSIONS

TESTING IS IMPORTANT

BUT OFTEN IGNORED

TESTERS AND OPS SHOULD WORK TOGETHER

WE NEED TO GET OUT OF OUR COMFORT ZONES

TOOLS EXISTBUT BE PREPARED

TO HACK

FINALLY...


THE APPLICATION OF METHODS AND TOOLING FROM SOFTWARE DEVELOPMENT TO MANAGEMENT OF IT INFRASTRUCTURE

PROGRAMMABLE INFRASTRUCTURE IS..

TESTING IS A SOFTWARE DEVELOPMENT METHOD

WE SHOULD APPLY IT TO INFRASTRUCTURE

THANKSQUESTIONS?

@burythehammer

[email protected]

mailto:[email protected]

Software

London Hashicorp Meetup #8 - Testing Programmable Infrastructure By Matt Long