©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

DevOps, Continuous Integration &

Deployment on AWS

Leo Zhadanovsky, Senior Solutions Architect, AWS@leozh

Dave Beck, Technical Director, National Novel Writing Month@DaveB_NaNoWriMo

DevOps

What is DevOps?

• « DevOps is the practice of operations and

development engineers participating together in

the entire service lifecycle, from design through

the development process to production support

»

- theagileadmin.com

Continuous Integration

What is Continuous Integration?

• Changes to code automatically deployed to mainline branch– After passing unit and mock tests

• Makes changes to code, and deployments iterative, not monolithic

• Bugs are detected quickly

• Helps automate deployments

• Allows rapid development and deployment

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

DEVELOPER

PICK

TASKS

SUBMIT

CODE

SCHEDULE

BUILD

RECURRENT

BUILDS

CODE

FETCHCODE QUALITY

TESTS

TEST

RESULTS

BUILD OUTPUT

DOCS

BINARIES

& PACKAGES

DEV FACING

NOTIFICATIONS

CLOUDFORMATION

AMIS or CONTAINERS

SOURCE CODE

REPOSITORY

DNS

CONTINUOUS

INTEGRATION SERVER

PROJECT

MANAGEMENT SERVER

BUILDS

AWS code services

AWS CodeCommit

Coming soon

AWS CodePipeline

Coming soon

AWS CodeDeploy

Launched Nov 2014

Cloud software development lifecycle

10/13/14 10

MonitorProvisionDeployTestBuildCode

AWS Elastic Beanstalk

AWS OpsWorks

Amazon

CloudWatch

AWS

CloudFormation

?

Source ControlPrivate Git repositories hosted on Amazon S3

• Full Git compatibility (use with existing tools)

• All the benefits of the cloud

(scalable, durable, reliable, low pay as you go

pricing)

• No size limits on repositories (store binary files)

• Online code tools with browse, edit, diff

10/13/14 11

Same Git experience

$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli

Cloning into 'aws-cli'...

Receiving objects: 100% (16032/16032), 5.55 MiB | 1.25 MiB/s, done.

Resolving deltas: 100% (9900/9900), done.

Checking connectivity... done.

$ nano README.rst

$ git commit -am 'updated README'

[master 4fa0318] updated README

1 file changed, 1 insertion(+)

$ git push

Counting objects: 3, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (3/3), done.

Writing objects: 100% (3/3), 297 bytes | 0 bytes/s, done.

Total 3 (delta 2), reused 0 (delta 0)

remote:

To https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli

4dacd6d..4fa0318 master -> master

Continuous DeliveryCustomizable release automation, with integrated build and test

• Model and visualize custom release workflow

(source build beta gamma prod)

• Automate builds, tests, and deployments

• Enforce custom rules, approvals, and gates

• Integrate with third-party and custom tools

10/13/14 13

DeploymentCoordinate software updates to fleets of EC2 instances

• Rolling updates for no downtime

• Deployment health checks and easy rollback

• Auto Scaling integration

• Works with any application

• Reuse existing setup tools

(Bash, Powershell, Chef, Puppet…)

10/13/14 15

9/25/14 Slides not intended for

redistribution.

16

Cloud software development lifecycle

10/13/14 17

MonitorProvisionDeployTestBuildCode

AWS Elastic Beanstalk

AWS OpsWorks

CloudWatchCloudFormationCodeDeploy

CodeCommit CodePipeline

CODECOMMIT

DNS

CODEPIPELINE

PROJECT

MANAGEMENT SERVER

BUILDS

PAIN POINTS• UNIT TESTS INCOMPLETE

• MOCK TESTS MAINTENANCE

• EXPENSIVE TEST ENVIRONMENT

• TEST ENVIRONMENT ≠

PRODUCTION

• DEPLOYMENT CYCLES

ON-DEMAND

PAY AS YOU GO

ELASTIC

=PROGRAMMABLE PLATFORM

IF YOU CAN PROGRAM IT

YOU CAN AUTOMATE IT

A lot of options…

• Configuration Management Systems– Puppet

– Chef

– Saltstack

• Deployment Frameworks– CodeDeploy

– AWS Elastic Beanstalk

– AWS OpsWorks

– Ansible

– Fabric

– Capistrano

• Infrastructure Management– CloudFormation

• Containers– Amazon EC2 Container Service

Bake an Image Configure dynamically

Time consuming configuration (startup time)

Static configurations (less change management)

Bootstrapping

Bake an Image Configure dynamically

Continuous deployment (latest code)

Environment specific (dev-test-prod)

Bootstrapping

©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

AWS CLOUDFORMATIONSTACK-BASED DEPLOYMENT SERVICE

CLOUDFORMATION

TEMPLATE

Headers

Parameters

Mappings

Resources

Outputs

{"Description" : "Create RDS with username and password","Resources" : {

"MyDB" : {"Type" : "AWS::RDS::DBInstance","Properties" : {

"AllocatedStorage" : "500","DBInstanceClass" : "db.m1.small","Engine" : "MySQL","EngineVersion" : "5.5","MasterUsername" : "MyName","MasterUserPassword" : "MyPassword"

}}

}}

"AWS::CloudFormation::Init" : { "config" : {

"packages" : {"yum" : {

"mysql" : [],"mysql-server" : [],"httpd" : [],"php" : [],"php-mysql" : []

}},"sources" : {"/var/www/html" :

"https://s3.amazonaws.com/my-builds/build-v4.zip"}

}

{"Parameters" : {"KeyName" : {

"Description" : "Name of an existing EC2KeyPair to enable SSHaccess to the instance",

"Type" : "String"}

},}

CLOUDFORMATION

TEMPLATE

PROCEDURAL

DEFINITIONCreate it programmatically

KNOWN

CONFIGURATIONStore stack configuration in

source control

PARAMETER

DRIVENDynamic and user-driven

templates

COLLABORATIONShare templates with ease

as just files

APPLICATION VERSIONS

+INFRASTRUCTURE VERSIONS

CLOUDFORMATION

TEMPLATE

CONTINUOUS

DEPLOYMENTSMALL, FREQUENT CHANGES CONSTANTLY

INTEGRATING INTO PRODUCTION.

KEY = ITERATION

ITERATION

=MODIFY THE SYSTEM TO BETTER MEET

THE EXPECTATIONS OF YOUR USERS

11.6s

Mean time

between

deployments

(weekday)

1,079

Max number of

deployments in a

single hour

10,000

Mean number of

hosts

simultaneously

receiving a

deployment

30,000

Max number of

hosts

simultaneously

receiving a

deployment

DEPLOYMENTS AT

AMAZON.COM

(in 2011)

SOFTWARE DEPLOY

≠PRODUCT LAUNCH

DATA-DRIVEN

ARCHITECTURES

METRICS @ETSY

METRICS @OBAMA FOR AMERICA

Metrics and Monitoring Options

CloudWatch

… and many more

CONTINUOUS

INTEGRATION

CONTINUOUS

DEPLOYMENT

CONTINUOUS DEPLOYMENT

=

CONTINUOUS EXPERIMENTATION

CONTINUOUS DEPLOYMENT

=

CONTINUOUS IMPROVEMENT

INNOVATE

SPEED AND AGILITY

Experiment

Often

Fail quickly at

a low cost

More

Innovation

Experiment

Infrequently

Failure is

expensive

Less

Innovation

“ON-PREMISE”

At a glanceWhat is NaNoWriMo?

• National

• Novel

• Writing

• Month

We are a community of novelists.

Every November, hundreds of

thousands of writers gather virtually

and in-person to write 50,000 words

each. (The Great Gatsby, by

comparison, is 47,094 words.)

Last year, 180,000 started new

novels, and 40,000 finished.

It all started in 1999

Our founder, Chris Baty,

challenged 20 over-caffeinated

friends to write 50,000 words each

in a single month.

Six of them finished.

Explosive growth, and no plan to manage it

NaNoWriMo now

NaNoWriMo Main: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6

Camp NaNoWriMo: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6

The Young Writers Program: Drupal 5, MySQL 5.6

NaNoWriMo Store: Drupal 5, MySQL 5.6

November in detail

• Nov 1: ~500k sessions as people begin• All month: ~50k forum posts per day• Nov 30: Last chance to post word counts

Annual Site Crash

• Every year from 2003 to 2011 the site would crash on

November 1.

• In 2012 we migrated the sites to Amazon Web Services.

• The result: No more outages!

• Naturally, this is how our participants reacted…

Outrage!

They wanted the site to crash; it was

considered a rite of passage. Oh well.

Keeping NaNoWriMo Online

• Proxy servers (Varnish) on Debian 7.5 EC2 instances

• Ruby on Rails on EC2 (Nginx)

• Worker servers running Resque on EC2

• Support servers running Resque, Redis, Sphinx

• MySQL on RDS instances

• Domains and subdomains on Route 53

• Image assets on S3 instances via EBS

• Deployment via Capistrano, Puppet

The Year in Instances

Next Steps: Goals

To increase uptime for all services

To make server management less time-consuming

To reliably and quickly respond to user-driven events

To reduce EC2 costs through autoscaling

To create a faster and more dynamic email system

To improve site search

Next Steps: Goodbye Drupal

What: Porting Drupal sites to Ruby 2.2, Rails 4.2

Why: The old Drupal-based sites are slow and use more

than their share of server resources.

Next Steps: Emailing via Amazon SES

What: Migrating outgoing emails from PHPlist/Sendmail to

SES

Why: Event-driven emails on the sites are backgrounded to

support servers, which has proved slow and unreliable.

Mass emails rely on an old and very sluggish application

which has limited layout tools and metrics.

Next Steps: Load Balancing

What: Converting proxy servers to Elastic Load Balancers

Why: We will benefit from better AWS dashboard tools,

metrics, and notifications. It is part of our autoscaling

strategy.

Next Steps: CodeDeploy

What: Using CodeDeploy for deployment process

Why: Currently deployment relies on many custom shell

scripts and tends to be unreliable. We intend to move to

stateless instance spin-ups as part of our autoscaling

strategy.

Next Steps: Autoscaling

What: Implement an Auto Scaling process

Why: In the past we’ve had to predict our server needs

ahead of time. If we underestimated, the sites would

crash; if we overestimated, we weren’t using our small

budget effectively. And the planning process consumed

too much staff time.

Next Steps: Amazon SNS Notifications

What: Increased use of Amazon SNS service

Why: Last year SNS event messaging helped us respond

to emerging crises very quickly. With our expanded AWS

presence, we will link event notifications to all of the

services we use.

Next Steps: On the Horizon

What: Amazon CloudSearch; AWS Lambda

Why: We are interested in easy-to-manage alternatives to

Sphinx – CloudSearch is one option. Lambda looks

intriguing for a variety of possible applications.

Potential future architecture

Internet

Load Balancer

ELB EC2 instances

Code servers

EC2 instances

Databases

RDS DB instances

Search

CloudSearch instance

Outgoing email

Amazon SES service

SAN FRANCISCO