Vagrant to AWS FlowDelivered by Colin Johnson, BlueChipTek (@bluechiptek)

Who am I?• DevOps Engineer and Occasional Developer• Certifications:

• AWS Certified Solutions Architect – Associate Level (Number 224)• AWS Certified SysOps Administrator – Associate Level (Number 2564)

• Worked with a number of Bay Area companies: inPowered, Jive Software, Lyft, NetShelter, Wingz

• Contributed VPC and Testing Support for boto_* Modules in SaltStack Project

Goals for this Presentation• Understand benefits and pain points of cloud, local and vagrant based

development• Describe a development flow that combines vagrant and AWS to

create a:• consistent environments for all developers• consistent environment from development to production• help organizations move fast – if they are not already doing this • integrate nearly flawlessly with AWS

• Ease Development <-> Production Software Deployment

Why Vagrant to AWS Flow?• Benefits for Everyone:

• Really Fast. Minutes to get a development environment setup.• Really Fast. No lag to test code changes.• Really Consistent. More similar to production than traditional “local” development.• Easy. Developers just type “vagrant ssh”

• Developers• Develop Offline• Provision Infrastructure on Own

• DevOps / Operations• No need to be involved in development or environment setup.

• QA• Test Offline.• Automate Testing.

What does Vagrant do?• Simple Use Case:

• Allows local development.• Offline? No Problem.

• Ensures consistency.• You use a Mac, your co-worker uses Windows but you want to run your app on Ubuntu? No

Problem.

• Advanced Use Cases:• Allows local development of an entire “dev” or “qa” application.

• Your application depends on MySQL, redis and an application developed by another group as well? No problem. Run them all locally.

• Run tests.• Need to run tests quickly but don’t want to automate infrastructure? No problem. Vagrant up.

Test. Vagrant destroy.

Experts at inPowered• Problem - developers loved AWS, but wanted to develop locally.

• Developers couldn’t develop offline• Developers had to “share” a single Development environment• Developers were developing on OS X whereas production was Amazon Linux• Developers were struggling to access Redis/MySQL and other dependencies

• Solution:• Vagrant + AWS Flow:

• Develop in Vagrant, Ship to AWS!• Ubuntu 15.04 for both Vagrant and AWS• Same configuration management for both Vagrant and AWS

• Vagrant:• offline development• per developer environments• all dependencies were run or “mocked” locally

Salt boto_* Module Development at Lyft• Project:

• adding VPC support and testing to salt’s boto_* modules (https://github.com/saltstack/salt/)

• Problems:• Needed to do testing of AWS automation (approximately 200+ tests), but building test

environment load balancers, security groups, etc was taking too long as did calls• Preferred testing locally• Preferred testing with multiple Operating Systems

• Solution:• Two Vagrant instances for Running Salt – Ubuntu 12.04 and 14.04 – fresh environment for

every test• Moto* for mocking AWS calls (https://github.com/spulec/moto)• Tests could be run by simply typing “vagrant up” and results could be generated and reviewed

Vagrant Development vs Cloud Development• One Environment for each Developer on Cloud Platform

• Administration burden – DevOps either has to Build or Automate• I used to spend 10 hours/week setting up environments for new Developers

• Inefficient Spend• Connectivity and Security requirements extends to uncontrolled “Development”

environments

• Shared Development on Cloud Platform• Sharing? You must be kidding.

• Who “owns” the server? Your feature, my configuration change and someone else’s change are all mixed together. Testing is going to be so fun!

• AWS / Cloud Development does typically offer all dependencies and services.

Vagrant Development vs Cloud Development:Act I – Internet Access Required

• Developer 1:• Going to be on vacation for the next week. Probably limited Internet access.

• Manager:• No problem. Can you get the new “every metric for every user action feature”

shipped on Monday?

• Developer 1:• I’ll try, but I can’t really test my code in without Internet.

Vagrant Development vs Cloud Development:Act II – Access and Management Headaches

• DevOps Engineer:• It looks like port 22 and port 80 in all our development environments are open. Google just

crawled all our development sites.

• Manager:• Do you know why all the ports are open?

• DevOps Engineer:• The docs for the webserver project say “to ssh in or do web testing, go to the security group

and open ports 22 and 80 to 0.0.0.0/0”• Manager:

• Do you know why?• DevOps Engineer:

• I think the Developers get tired of waiting for VPN credentials to be issued.

Vagrant Development vs Local Development• Match local and production environments• Local environment (OS X, Windows or *nix) may not match production

• Faster Environment Standup• Run same configuration management as production!

• Develop Offline / Complete infrastructure available• Run your app and all dependencies

Vagrant Dev vs Local Dev:Act I – Dev/Prod Inconsistency• Developer:• It worked on my machine.

• DevOps:• So, it worked on OS X 10.9.2, nginx preview release 1.9.11 and MariaDB.

• Developer:• Yes. I also had to install SciPy and NumPy.

• DevOps: • Anything else?

• Developer:• I think that is it. Make sure to use SciPy 0.17.

Vagrant Development vs Local DevelopmentAct II – Local Development Inconsistency

• Developer 1:• It worked on my machine. OS X 10.9.2 with nginx 1.9.11 preview release…

• Developer 2:• I’m running Windows. 10.

• Developer 1:• Damn. Get a Mac. Everyone else uses them.

• Developer 3:• You should use Linux. Same as production. Except for MariaDB instead of

MySQL. But I don’t think that’s a problem.

Vagrant Development vs Local DevelopmentAct III – Running the Application• Developer 1:

• I’ve been trying to get nginx 1.9.11 preview installed on my Mac for the last four hours

• Developer 2:• Use homebrew. I found an article online somewhere.

• Developer 2:• Here’s a link:

http://www.robertmulley.com/tutorial/nginx-install-and-setup-mac-os-x-yosemite/

• Developer 1:• Link me for MongoDB, InfluxDB and Redis installs as well :-)

Vagrant to AWS Flow:• Standup Development in Minutes• Easy ssh Access• Easy file access• Run local mocks• No sharing environments!

Vagrant + AWS: How To• Use same OS in Vagrant as AWS• Separate and Make Modular Each “Concern”• Infrastructure Provisioning• Configuration Management• Software Deployment

• Avoid or Be Aware of Platform Specific Technologies• Vagrant synced_folder = not available in AWS – you can still use for software

deployment• AWS IAM Profile = not available in Vagrant/Virtualbox – you can still use for

software deployment

Vagrant Setup • Infrastructure Provisioning:

• Run “vagrant up”• Uses a Vagrantfile in root directory of project• Vagrantfile defines:

• CPU, Memory• Operating System – use same as AWS!• A “Provisioner” – which runs Configuration Management

• Configuration Management (aka “Provisioner”)• I prefer shell as “glue code”• Use same Configuration Management as AWS - Ansible, Chef, Puppet, Salt, etc.

• Software Deployment• In Vagrant, use a “synced directory”– then changes made by developers are manifest in real-

time near real-time

AWS Setup• Infrastructure Provisioning:

• Run “cfn-create-stack”• Creates an Auto Scaling Group and Launch Configuration with user-data• Use same OS as Vagrant!• Launch Configuration calls Configuration Management

• Configuration Management (aka “Provisioner”)• I prefer shell as “glue code”• Use same Configuration Management as Vagrant - Ansible, Chef, Puppet, Salt, etc.

• Software Deployment• Use whatever you’d like…

Vagrant to AWS Flow - Diagram

DemoIn the next 15 minutes we will:

1. Standup local development infrastructure on Ubuntu.1. And look at a Vagrantfile.

2. Standup identical* production infrastructure in AWS.1. And look at a CloudFormation file.

3. Deploy code to AWS and run the same code in Dev/Prod.1. The deployment tool is a bit of a hack, btw…