Towards FutureOps: Stable, repeatable, environments from dev to prod.

I’m Mitchell HashimotoAlso known as @mitchellh

I build toolshttp://hashicorp.com

Vagranthttp://www.vagrantup.com

Packerhttp://www.packer.io

SERFhttp://www.serfdom.io

History of OpsOr, the history of pretending that anything works.

Single Dedicated Server

Single Dedicated Server• Manually setup

Single Dedicated Server• Manually setup • Expensive to replace

(time and money)

Single Dedicated Server• Manually setup • Expensive to replace

(time and money) • Professionals, usually

What is hard/slow/expensive?• Acquiring hardware • Manual setup • Expertise • Downtime, Failures

Multiple Dedicated Servers

Multiple Dedicated Servers• Golden images

Multiple Dedicated Servers• Golden images • Manually made

Multiple Dedicated Servers• Golden images • Manually made • Slow to change,

expensive to grow

Multiple Dedicated Servers• Eventually: config

management (CFEngine, Puppet, Chef)

What is hard/slow/expensive?• Acquiring hardware • Making images • Expertise • Downtime, Failures

The Low-Cost VPS Server

The VPS Server• Manually made (cargo

cult, usually)

The VPS Server• Manually made (cargo

cult, usually) • Cheaper to replace

The VPS Server• Manually made (cargo

cult, usually) • Cheaper to replace • Hobbyist, startup

What is hard/slow/expensive?• Expertise • Downtime, Failures • Rebuilding it

The Cloud

The Cloud• Many cheap servers

The Cloud• Many cheap servers • Seconds to create

The Cloud• Many cheap servers • Seconds to create • Configuration

management

• Configuration mgmt • Portability • Repeatability • Can be expensive

What is hard/slow/expensive?

The Ops of TomorrowBetter, faster, stronger

Immutable Infrastructure

Immutable InfrastructureInstead of modifying or maintaining a server, replace it with a new one.

Immutable InfrastructureServers don’t change anymore, they’re pre-built, static images that are tested/verified.

Immutable Wins• Super fast server deployment • Failure expected, not a big deal • Testable, more stable

Automation Critical• Machine Images • Software install/configure • Failure detection, recovery • etc…

Machine ImagesEasily build machine images for multiple platforms with Packer.

Machine image: A single deployable unit that contains a pre-configured OS and software.

Ew... Images.The anti-image stigma

DevOps has historically been against machine images.

Important to understand why DevOps has historically been against machine images.

Golden images used to be the way.

Quarterly, unchanged, blessed image.

Getting any changes (ops or dev) in was slow and frustrating.

Necessary evil while tooling wasn’t as good as it is today.

With modern config management, easy to simply ignore images.

Machine images also have a ton of benefits.

Super fast infrastructure deployment.

Multi-cloud portability

Stability and Testability

Analogy: Ops without machine images is like applications without binaries

Put it another way: There is no promise of reproducible builds in ops, and no binary-like snapshots

Source Code Binary

Source Code Binary

libA 1.0 libB 1.0 libC 1.0

Source Code Binary

libA 2.0 libB 1.0 libC 1.0

Source Code COMPILE FAILED

libA 2.0 libB 1.0 libC 1.0

New Server Ready Server

Chef, Puppet, Scripts, etc.

New Server Ready Server

Packages NetworkChef/Puppet

Changes

Chef, Puppet, Scripts, etc.

New Server

NetworkChef/Puppet

Changes

Chef, Puppet, Scripts, etc.

Package Changes

SERVER SETUP FAILED

New Server

PackagesChef/Puppet

Changes

Chef, Puppet, Scripts, etc.

Network Unreliable

SERVER SETUP FAILED

New Server

Packages Network

Chef, Puppet, Scripts, etc.

Chef/Puppet Changed

SERVER SETUP FAILED

New Server Ready Server

Machine Image

New Server Ready Server

Machine Image

PackerEmbraces modern best practices and automates building images

Takes template input and repeatably produces image output.

$ packer build \! -var ‘aws_access_key=YOUR KEY’ \! -var ‘aws_secret_key=YOUR SECRET’ \! template.json!...

Automated, repeatable, fast, uses Chef/Puppet/Shell/etc. Gets rid of old downsides.

Can also produce Vagrant boxes for development, at the same time.

WorkflowsWhat does this look like in practice?

Ops Push

Packer Build

Machine Image(s)

Ops

Ops Push

Packer Build

Machine Image(s)

Ops

Setup users, install software, configure software, configure

services. No orchestration.

Request Server

Running Server

Server Deploy

Request Server

Running Server

Server Deploy• Server started with Packer-

built image

• Running server has all pre-installed software

• Service orchestration runs now

“vagrant up”

Ready to work

Development

“vagrant up”

Ready to work

Development• Vagrant downloads the latest

machine image created by Packer

• “vagrant up” takes seconds to minutes and runs, nearly identical to production

Packer Wins• Automated, repeatable images • Fast deployment from dev to prod • Testable, very stable • Embraces configuration mgmt

SERFService orchestration, discovery, failure detection.

Remember, Chef/Puppet/etc. aren’t doing service orchestration for the image.

Service Discovery: Web servers in a load balancer, Memcache in a cluster, MySQL master/slave, etc.

Service Orchestration: Add this node to your cluster, deploy this application, restart, etc.

Serf is a decentralized, highly available, fault tolerant, and lightweight solution to service orchestration and discovery.

ExampleWeb servers in a load balancer

Run the Agent$ serf agent \!! -event-handler=“handle.sh”!…

Join a Cluster$ serf join 10.0.1.5!…

Serf Agent

Handle.sh

Serf Events

Add or Remove Node from LB

Serf Agent

handle.sh

Serf Events

Add or Remove Node from LB

Serf invokes event handlers for any event: member-join, member-leave, member-failed, user.

How?How does Serf do what it does?

GOSSIP-BASED MEMBERSHIP

FAILURE DETECTION

CUSTOM EVENTS

Chef/Puppet Run

Configure Serf Agent Services

Config Management

Don’t start Serf. Serf starts on boot.

Chef/Puppet Run

Configure Serf agent init script

Config Management

Don’t start Serf. Serf starts on boot.

Chef and Puppet just configure the Serf init script to start the agent.

Serf Wins• No orchestration in image build • Fast membership detection, don’t

wait for Chef/Puppet • Easily and infinitely extensible

The Future is ImmutableBig wins, little downsides.

As servers become cheaper, more on-demand, and infrastructures become more distributed, immutable infrastructure will win.

Important to remember:You don’t have to go all in on immutable. You can start with certain servers. Example: databases are hard, just ignore that for awhile.

Mostly ImmutableThere are still big benefits for going mostly immutable. Things like quick config changes and deploys can still be mutable. Serf is good for this.

The Wins• Super fast deployment • Repeatability • High failure tolerance • Improved stability and testability • Versioning

The Downsides• Requires proper mindset • Tooling is new • Deploys, simple config changes

Thank you!

Vagranthttp://www.vagrantup.com

Packerhttp://www.packer.io

SERFhttp://www.serfdom.io