Continuous Integration & Continuous Deployment

Brett Hollman – Solutions Architect October 17th, 2013

Con$nuous    Integra$on  

Cloud-powered Continuous Integration

Goal: to have a working state of the code at any given time Benefit: Fix bugs earlier when they are cheaper to fix Metric: New guy can check out and compile at first day at job

Cloud-powered Continuous Deployment

Goal: 1-click deploy and 1-click rollback Benefit: Release early, release often and iterate quickly Metric: fast feedback of your feature

Con$nuous    Deployment  

Let’s start by taking a look at a traditional IT Software Development Model…

Operations Release QA Dev

Feedback Loop

SCM & CI Testing Deploy Manage

Ticketing Ticketing

Now let’s contrast this with a more modern software development model using Continuous

Integration and Continuous Deployment…

Testing SCM Deploy Manage

Operations

Feedback Loop

Dev Release QA

Ticketing

Common Automation Platform

Ticketing Ticketing

Tenets for Continuous Integration & Deployment

Treat infrastructure as code Automate the testing/deploy process end to end Make sure environments mimic each other as closely as possible Use repeatable patterns between environments at a different scale Simplify and streamline the deploy process

Since Infrastructure is code, let’s treat it like code!

§  Not JUST Revision control! §  Make use of bug tracking/ticketing systems §  Peer reviews of changes before they happen §  Establish infrastructure code patterns/designs §  Test infrastructure changes like code changes

Let’s talk about the journey our code is

going to take to production

First stop on our journey: Continuous Integration / Automation Tools

•  Help improve code quality and function repeatedly with predefined results

•  Lots of options; self hosted, open source, closed source, and SaaS

Version Control

CI Server

Package Builder

Deploy Server Commit to

Git/master Dev

Get / Pull Code

AMIs

Send Build Report to Dev Stop everything if build failed

Distributed Builds Run Tests in parallel

Staging Env

Test Env Code Config Tests

Prod Env

Push

Config Install Create

Repo

CloudFormation Templates for Env

Generate

Cloud Continuous Integration – AMI Deployment Model

This all seems like a lot of work,

and potentially costly.

But it doesn’t need to be!

Infrastructure Automation

We want to be able to use tools to rapidly stand up environments as we need to. What are some automation tools we might use?

§  CloudFormation §  Elastic Beanstalk §  OpsWorks §  Chef §  Puppet

AWS Application Management Services

Elastic Beanstalk OpsWorks CloudFormation

Application Container Application & Resource Management

Provision and manage AWS Resources

You want to quickly get code running scalable and reliable manner without worrying about the underlying resources

You want to run an application with the ability to customize and control software or resources

You want a simple, yet finely controllable way to model and version collections of AWS resources

AWS CloudFormation "WebServer" : { "Type" : "AWS::EC2::Instance", "Properties" : { "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, "64" ]}, "SpotPrice" : { "Ref" : "SpotPrice" }, "InstanceType" : m1.large", "SecurityGroups" : [{ "Ref" : "InstanceSecurityGroup" }], "KeyName" : { "Ref" : "KeyName" }, "UserData": { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash -v\n", "yum update -y aws-cfn-bootstrap\n”,

"curl -L http://www.opscode.com/chef/install.sh | bash\n", "cd /etc/chef\n", "/usr/bin/wget http://",{ "Ref" : "ChefServerIP" },"/chef/validation.pem\n", "/usr/bin/wget http://",{ "Ref" : "ChefServerIP" },"/chef/client.rb\n", "/bin/chown -R chef:chef /etc/chef\n",

AWS Elastic Beanstalk

Application container framework similar to a PaaS Deploy your application into Elastic Beanstalk and it takes care of building a self healing, auto-scaling, multi-AZ infrastructure Allows you to turn some of the knobs under the hood to tweak Considered one of the easiest places to start with hosting an application on AWS No additional cost

What is Elastic Beanstalk?

Instance

Amazon  RDSElastic  LoadBalancer

Instance

Auto  scaling  GroupCloudWatch

deploy! WAR

•  Automatically prepares scalable and highly available environments •  Deploy your web app and it runs on AWS best practice architecture

Logical View of an Application

Application

　Environment Version

WAR/ZIP URL Environment Configuration

Configuration Template

　Environment URL Environment Configuration

WAR/ZIP

WAR/ZIP

WAR/ZIP

WAR/ZIP 　Environment

URL Environment Configuration

AWS OpsWorks •  Integrated application management solution for

ops-minded developers and IT admins

•  Model, control and automate applications of nearly any scale and complexity

•  Management Console, SDKs, or CLI

•  No additional cost

OpsWorks Benefits

Simple. Start with built-in templates for common technologies or build your own using Chef. Flexible. Model and control your entire application, from load balancers to databases. Productive. Specify how to deploy, scale, and maintain your application and automate common tasks.

AWS OpsWorks Primer

A stack represents your application. One stack might be used for staging and another for production.

A layer defines how to setup and configure a set of instances and related resources such as volumes and software.

Simply tell OpsWorks where it can find your code and define any additional deployment tasks. Then OpsWorks will take care of deploying your app in your desired configuration.

Scale your stack based on time or load. Clone your production stack to a different region. Set up user permissions and access. Automate workflows for common tasks.

What is Chef and how does OpsWorks use it

•  Chef is an open-source framework that automates software deployment and configuration.

•  Whenever a change happens on your stack, or upon request, all instances are notified and recipes are run.

Recipes Lifecycle Events

Metadata

Already said this, but all of these tools can be called via APIs to trigger resource creation and

deployment…

Region

Amazon Route 53

Amazon CloudFront

Customer Traffic

Amazon S3

Availability Zone

Availability Zone

Availability Zone

Internet Gateway

VPC Subnet VPC Subnet VPC Subnet VPC Subnet

VPC Subnet VPC Subnet VPC Subnet VPC Subnet

VPC Subnet VPC Subnet VPC Subnet VPC Subnet

Instance Instance

Instance

NAT

Bastion/Chef

ELB ELB

RDS DB Instance

RDS DB Instance Standby (Multi-

AZ) Instance

AWS CloudFormation

Amazon CloudWatch

Amazon SNS

Potential RDS DB Instance

Read Replica

Instance Instance

Our Development &Test Infrastructure

Developers &

Operations

Internal Git CI

Server

Pre-commit Hook

Testing Environment Subnet

CI Workers

Dev Environment VPC Subnet

DEV WEB ELB Dev Stack

Tier 1 Dev Stack

Tier 2

Dev MySQL

DB Instance

DEV APP ELB

VPN TUNNEL

VPN facing VPC Subnet

Internet Gateway

VPN Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoD

B

Amazon SQS

Amazon CloudFro

nt

Amazon Route

53

Our Development/Test Infrastructure

Developers &

Operations

Internal Git CI

Server

Pre-commit Hook

Testing Environment Subnet

CI Workers

Dev Environment VPC Subnet

DEV WEB ELB Dev Stack

Tier 1 Dev Stack

Tier 2

Dev MySQL

DB Instance

DEV APP ELB

VPN TUNNEL

VPN facing VPC Subnet

Internet Gateway

VPN Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoD

B

Amazon SQS

Amazon CloudFro

nt

Amazon Route

53

CloudFor

mation

Stacks

Region

Amazon Route 53

Amazon CloudFront

Customer

Traffic

Amazon S3

Availability Zone

Availability Zone

Availability Zone

Internet Gatewa

y

VPC Subnet VPC Subnet VPC Subnet VPC Subnet

VPC Subnet VPC Subnet VPC Subnet VPC Subnet

VPC Subnet VPC Subnet VPC Subnet VPC Subnet

Instance

Instance

Instance

NAT

Bastion/Chef

ELB ELB

RDS DB Instance

RDS DB Instance Standby (Multi-

AZ) Instanc

e

AWS CloudFormation

Amazon CloudWatch

Amazon SNS

Potential RDS DB Instance

Read Replica

Instance

Instance

Elastic Beanstalk or OpsWorks

CloudFormation Stacks

Example

Deploy Code Build Test Provision Monitor

OpsWorks Git Automation Tools

Check in – Build & Test Tests pass – Deploy to Staging

Now that we know where our code is going,

how is it getting there?

Ship that code! How are we going to deploy our code?

§  File shipping: •  Just text files •  Binaries

§  As an AMI: •  Package bundling into a

Tarball / RPM •  Bundle one of the above into

an AMI How fast do we need to do this? Across how many instances? How do we roll back (or forward)?

Yes, you can technically ship your code to AWS in a box. See AWS Import/Export. Probably not for CI&CD though…

File Shipping Deploy Method

Leave existing hosts in place Push new code to the instances Rolling restarts for web/application servers Have to worry about tracking what version is live for building new hosts Can be easier to work with than AMI method and package bundling

File Shipping Deploy Method

Elastic Beanstalk and Opsworks use the File Shipping methodology Code is deployed on all live servers in a layer in a parallel directory structure Symlink is moved for “current” directory structure to point to new deployed code Nginx / Apache services are restarted

AMI Deployment Method

Code gets bundled into an AMI, we then deploy that AMI

§  Pluses •  New shouldn’t effect older versions •  Can deploy alongside current prod env •  Easy tools to automate (Automation and AMI baking tools i.e. Netflix

Aminator) §  Cons

•  Bit more work involved

Leverage configuration management tools in automation process

AMI Deployment Method - Building

AMI Deployment Method - Building

Fully Functional AMI OS-Only AMI

Partially Configured AMI

AMI Deployment Method - Building

Fully Functional AMI OS-Only AMI

Partially Configured AMI

Least flexible to maintain

AMI Deployment Method - Building

Fully Functional AMI OS-Only AMI

Partially Configured AMI

Most amount of post-boot

work

Least flexible to maintain

AMI Deployment Method - Building

Fully Functional AMI OS-Only AMI

Partially Configured AMI

Most amount of post-boot

work

Least flexible to maintain

Try and find a happy medium

here

AMI Deployment Method - Deploying Blue/Green Deploys

§  We stand up a duplicate part of our infrastructure and slowly cut traffic over to it

•  Shift load via DNS •  Makes it easy to do testing

of new features •  Makes it easy to roll back

§  As we shift more traffic over, let auto-scaling grow/shrink our instances of the new or old application

•  Shut down the old when no traffic there

Amazon Route

53

EC2 Instances

ELB

100%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

AMI Deployment Method - Deploying

Amazon Route

53

EC2 Instances

ELB

EC2 Instances

ELB

90%

10%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

Blue/Green Deploys §  We stand up a duplicate part

of our infrastructure and slowly cut traffic over to it

•  Shift load via DNS •  Makes it easy to do testing

of new features •  Makes it easy to roll back

§  As we shift more traffic over, let auto-scaling grow/shrink our instances of the new or old application

•  Shut down the old when no traffic there

AMI Deployment Method - Deploying

Amazon Route

53

EC2 Instances

ELB

EC2 Instances

ELB

50%

50%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

Blue/Green Deploys §  We stand up a duplicate part

of our infrastructure and slowly cut traffic over to it

•  Shift load via DNS •  Makes it easy to do testing

of new features •  Makes it easy to roll back

§  As we shift more traffic over, let auto-scaling grow/shrink our instances of the new or old application

•  Shut down the old when no traffic there

AMI Deployment Method - Deploying

Amazon Route

53

EC2 Instances

ELB

EC2 Instances

ELB

0% 100%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

Blue/Green Deploys §  We stand up a duplicate part

of our infrastructure and slowly cut traffic over to it

•  Shift load via DNS •  Makes it easy to do testing

of new features •  Makes it easy to roll back

§  As we shift more traffic over, let auto-scaling grow/shrink our instances of the new or old application

•  Shut down the old when no traffic there

AMI Deployment Method - Deploying

Amazon Route

53

EC2 Instances

ELB

EC2 Instances

ELB

0% 100%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

Blue/Green Deploys §  We stand up a duplicate part

of our infrastructure and slowly cut traffic over to it

•  Shift load via DNS •  Makes it easy to do testing

of new features •  Makes it easy to roll back

§  As we shift more traffic over, let auto-scaling grow/shrink our instances of the new or old application

•  Shut down the old when no traffic there

AMI Deployment Method - Deploying

Amazon Route

53

EC2 Instances

ELB

100%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

Blue/Green Deploys §  We stand up a duplicate part

of our infrastructure and slowly cut traffic over to it

•  Shift load via DNS •  Makes it easy to do testing

of new features •  Makes it easy to roll back

§  As we shift more traffic over, let auto-scaling grow/shrink our instances of the new or old application

•  Shut down the old when no traffic there

AMI Deployment Method - Deploying

Amazon Route

53

EC2 Instances

ELB

100%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

Blue/Green Deploys §  We stand up a duplicate part

of our infrastructure and slowly cut traffic over to it

•  Shift load via DNS •  Makes it easy to do testing

of new features •  Makes it easy to roll back

§  As we shift more traffic over, let auto-scaling grow/shrink our instances of the new or old application

•  Shut down the old when no traffic there

Our code has arrived at its destination

Thanks