Scaling DevOps

Scaling DevOpsJeffrey HultenWhitepages, Inc.

Wednesday, December 4, 13

Who Am I?

Jeffrey Hulten - @jhulten

Software Engineer, Architecture Improvement at WhitePages, Inc.

Organizer of Seattle DevOps MeetUp

Too Ops for Dev, Too Dev for Ops

Wednesday, December 4, 13

Scaling DevOps

Your Tools Aren’t Enough

Your First 100 Machines

Watching the Horizon

Wednesday, December 4, 13

Your Tools Aren’t Enough

Culture

Deploy a Package

Manage the Metal

Minimize the Surface Area

Growing Pains

Wednesday, December 4, 13

CultureNot Just for Bacteria Anymore

Starts with one person

Infect your friends

Try changing things

Mutation must allow for failure

Requirement for survival

Wednesday, December 4, 13

Deploy a Package

TGZ

OS Package (.deb / .rpm)

Machine Image (AMI)

LXC Image (Docker)

Wednesday, December 4, 13

OS Packages Made Easier

FPM

Package Config Separately

Package Repository

Wednesday, December 4, 13

FPM: Effing Package Management

https://github.com/jordansissel/fpm

$ gem install fpm

Make RPM, DEB, and Solaris packages and Puppet module

Create from directories, RPM, DEB, Gem, PyPi, Tarball and more

Wednesday, December 4, 13

Example: Jenkins

NAME=jenkinsVERSION=1.396

.PHONY: packagepackage: rm -f jenkins.war wget http://ftp.osuosl.org/pub/hudson/war/$(VERSION)/jenkins.war fpm -s dir -t deb -n $(NAME) -v $(VERSION) \ --prefix /opt/jenkins jenkins.war

Wednesday, December 4, 13

Machine Images & Containers

Machine Image

Entire OS

All Services Enabled

Containers

Minimal OS Image

Run One Process (Replace `init`)

Wednesday, December 4, 13

Manage the Metal

Application as Inventory...

Wednesday, December 4, 13

Minimize the Surface Area

What is Surface Area?

Libraries?

Versions?

Components?

Data Stores?

Wednesday, December 4, 13

Minimize the Surface Area

What is Surface Area?

Libraries?

Versions?

Components?

Data Stores?

Deployables...

Wednesday, December 4, 13

Growing Pains

Don’t bottle knowledge

There are no specialists

Think about the smallest solution

Delay your decisions

Wednesday, December 4, 13

Your First 20 Machines

DEMO TIME!

Source at: https://github.com/whitepages/scaledemo-20-machines

AWS CloudFormation, cloud-init...

Wednesday, December 4, 13

Application: Random Numbers as a Service

Wednesday, December 4, 13

CloudFormation

Deploy Stacks

Templates in JSON

Parameters for Reuse

Wednesday, December 4, 13

Template: Parameters"Parameters": { "OperatorEmail": { "Type": "String", "Description": "Email address to notify if there are any scaling operations" }, "InstanceType": { "Description": "WebServer EC2 instance type", "Type": "String", "AllowedValues": [ "t1.micro", "m1.small", ...], "Default": "m1.small", "ConstraintDescription": "must be a valid EC2 instance type." },

... "SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", "Description": "The IP address range that can be used to SSH to the EC2 instances", "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x." } },

Wednesday, December 4, 13

Template: Parameters

"SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3}).../(\\d{1,2})", "Description": "...", "ConstraintDescription": "..."}

Wednesday, December 4, 13

Template: Mappings"Mappings": { "AWSRegionArch2AMI": { "us-east-1": { "64": "ami-955b79fc" }, "us-west-1": { "64": "ami-6ca89929" }, "us-west-2": { "64": "ami-a8395c98" } }, ...},

Wednesday, December 4, 13

Template: Resources"Resources": { "NotificationTopic": { "Type": "AWS::SNS::Topic", "Properties": { "Subscription": [ { "Protocol": "email", "Endpoint": { "Ref": "OperatorEmail" } } ] } }, ...}

Wednesday, December 4, 13

AutoScaling Group"RandServerGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": { "Fn::GetAZs": "" }, "NotificationConfiguration": { "NotificationTypes": [ "autoscaling:EC2_INSTANCE_LAUNCH", "autoscaling:EC2_INSTANCE_LAUNCH_ERROR", "autoscaling:EC2_INSTANCE_TERMINATE", "autoscaling:EC2_INSTANCE_TERMINATE_ERROR" ], "TopicARN": { "Ref": "NotificationTopic" } }, "MinSize": "1", "MaxSize": "100", "LaunchConfigurationName": { "Ref": “..." }}}

Wednesday, December 4, 13

Launch Config"RandLaunchConfig": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Properties": { "SecurityGroups": [{ "Ref": InstanceSecurityGroup" }], "InstanceType": { "Ref": "InstanceType" }, "KeyName": { "Ref": "KeyName" }, "UserData": { "Fn::Base64": { "Fn::Join" : ["",[ "#include\n", "https://.../cloud-config.txt\n" ]]}}, "ImageId": {"Fn::FindInMap": [...] } }},

Wednesday, December 4, 13

Cloud Init

#cloud-configpackages: - ruby1.9.3 - git - build-essential

runcmd: - /usr/bin/gem install bundler - /usr/bin/git clone "https://github.com/whitepages/scaledemo-20-machines.git" /opt/rand - cd /opt/rand && /usr/local/bin/bundle install - cd /opt/rand &&/usr/local/bin/bundle exec randserver &

Wednesday, December 4, 13

Watching the Horizon

Docker / LXC

SmartStack

Akka Cluster / Riak Core

Wednesday, December 4, 13

Questions?jhulten@whitepages.com

Twitter: @jhulten

Github: jhulten

whitepages

whitepages.com/careers

Wednesday, December 4, 13