AWS - Managing Your Cloud Assets 2013

Journey through the cloud: Managing your cloud assets

Ryan Shuttleworth – Technical Evangelist @ryanAWS

Common use cases & stepping stones into the AWS cloud Learning from customer journeys

Best practices to bootstrap your projects

Journey through the cloud

Getting to grips with system sprawl in an unconstrained environment Leverage management frameworks to take control

Choose systems that suite your technology delivery style

Managing cloud assets

Setting the scene A shrink wrapped approach – Elastic Beanstalk Fine grained control – CloudFormation A middle way - OpsWorks

Agenda

Cloud assets are growing quickly

Defined architectural patterns More applications

More data More services

aws.amazon.com/architecture

Greater agility = faster moving environments

How do we maintain control over what is deployed in the cloud?

Two ends of the spectrum…

Controlled chaos

Meticulous management

Controlled chaos


Prototype an app Deploy to an instance

Test and develop cycles Go to production

Deploy more instances Deploy RDS

Add load balancers Add autoscaling

Audit what you have?

Quickly becomes DevOps nightmare

Controlled chaos








Prototype an app Plan infrastructure

Build standards Deploy environment

Manage through change control Formal processes

Formal tools Embed configuration in

chef/puppet

Tightly controlled, but lots of effort to achieve

Controlled chaos


People Processes Controls

Tools







Prototype an app Plan infrastructure

Build standards Deploy environment

Manage through change control Formal processes

Formal tools Embed configuration in

chef/puppet

Tightly controlled, but lots of effort to achieve

High level, shrink

wrapped

Fine grained, declarative

control

Tools

“I just want my app to run, and I only want

access to machines if I really need to”

“I want to source control my entire datacenter

deployment like I would my apache config”

High level, shrink

wrapped


control

Elastic Beanstalk

CloudFormation

Tools





High level, shrink

wrapped


control

Elastic Beanstalk

CloudFormation

Middle ground

AWS OpsWorks





High level, shrink

wrapped


control

Elastic Beanstalk

CloudFormation

Middle ground

AWS OpsWorks





Quickly deploy and manage apps in AWS…

Elastic Beanstalk

CloudFormation

Tools

…into a range of containers

Elastic Beanstalk

CloudFormation

Tools

.Net PHP

Java Python

Ruby Node.js

…into a range of containers

Elastic Beanstalk

CloudFormation

Tools

http://aws.typepad.com

Keep updated on new containers:

Application package is deployed into Beanstalk

Elastic Beanstalk

CloudFormation

Tools

User Application

Application Service

HTTP Service

Language Interpreter

Operating System

Host

Which creates the container in EC2

Elastic Beanstalk

CloudFormation

Tools

Beanstalk takes care of the environment…

Elastic Beanstalk

CloudFormation

Tools

…adding Elastic Load Balancer…

Elastic Beanstalk

CloudFormation

Tools

…Auto Scaling groups

Elastic Beanstalk

CloudFormation

Tools

…and launching instances

Elastic Beanstalk

CloudFormation

Tools

…all wired into the Beanstalk framework

Elastic Beanstalk

CloudFormation

Tools

…and published under a CNAME

Elastic Beanstalk

CloudFormation

Tools

…with logs and app versions held in S3

Elastic Beanstalk

CloudFormation

Tools

User Application

Application Service

Apache

Python

Operating System

Host

mod_wsgi

…Python mod-wsgi & apache

Elastic Beanstalk

User Application

Application Service

Nginx

Ruby

Operating System

Host

Passenger & Nginx…

Elastic Beanstalk

passenger

What about the data?

Rails

Django

Git Beanstalk

rake

syncdb

RDS

How do you create an app for AWS ElasticBeanstalk?

Like any other app

No dependencies on AWS inherent for Beanstalk deployment

Elastic Beanstalk

CloudFormation

Tools

Source uploaded and executed in AWS

Source downloaded

and executed locally

Package up as normal…

Elastic Beanstalk

CloudFormation

Tools

Java .war file

Microsoft Web

Deploy package

PHP .zip file

Python .zip file

Git integration IDE plugins

Package up as normal…

Elastic Beanstalk

CloudFormation

Tools

Java .war file

Microsoft Web

Deploy package

PHP .zip file

Python .zip file

Git integration IDE plugins

Console deployments and version updates

Elastic Beanstalk

CloudFormation

Tools

Command line tools and wizards

Elastic Beanstalk

CloudFormation

Tools

CheckDNSAvailability elastic-beanstalk-check-dns-availability

CreateApplication elastic-beanstalk-create-application

CreateApplicationVersion

elastic-beanstalk-create-application-version

CreateEnvironment

elastic-beanstalk-create-environment

eb init wizard to initialize an application

eb start/stop start/stop an application

eb update

update application version

eb status

get status of a running application

Eclipse integration

Elastic Beanstalk

CloudFormation

Tools

Eclipse integration

Elastic Beanstalk

CloudFormation

Tools

Eclipse integration

Elastic Beanstalk

CloudFormation

Tools

Eclipse integration

Elastic Beanstalk

CloudFormation

Tools

Visual Studio integration

Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools

Git integration

Elastic Beanstalk

CloudFormation

Tools

Container configuration

Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools


Elastic Beanstalk

CloudFormation

Tools

Full admin access on underlying resources EC2 instances, AMIs, AutoScaling groups…

High level, shrink

wrapped


control

Elastic Beanstalk

CloudFormation

Middle ground

AWS OpsWorks





What happens under the covers of Beanstalk?

Elastic Beanstalk

CloudFormation

Tools

Elastic Beanstalk

AWS

Create security group

Create auto-scaling policy

Create elastic load balancer

Add instance

Install software

Consolidate logs

What happens under the covers of Beanstalk?

Elastic Beanstalk

CloudFormation

Tools

AWS

Create security group

Create auto-scaling policy

Create elastic load balancer

Add instance

Install software

Consolidate logs

A series of calls into AWS to marshal resource

creation/configuration

CloudFormation – underlying automation framework

Elastic Beanstalk

CloudFormation

Tools

Security Scaling

Database

Networking Monitoring

Messaging

Workflow

DNS

Load Balancing

Backup CDN

Compute

Storage

Elastic Beanstalk

CloudFormation

Tools

CloudFormation

A consistent scripting interface for creating/configuring AWS

resources

A framework for lifecycle management of resources

created using scripts

Security Scaling

Database

Networking Monitoring

Messaging

Workflow

DNS

Load Balancing

Backup CDN

Compute

Storage

CloudFormation – underlying automation framework

CloudFormation components & terminology

Elastic Beanstalk

CloudFormation

Tools

Template CloudFormation

Stack

JSON formatted file

Parameter definition

Resource creation

Configuration actions

Configured AWS services

Comprehensive service support

Service event aware

Customisable

Framework

Stack creation

Stack updates

Error detection and rollback

Template examples

Elastic Beanstalk

CloudFormation

Tools

Template

JSON formatted file


Resource creation


{

"Parameters" : {

"KeyName" : {

"Description" : "Name of an existing EC2 KeyPair to enable SSH

access to the instance",

"Type" : "String"

}

},

}

Snippet: Taking user defined parameters

Template examples

Elastic Beanstalk

CloudFormation

Tools

Template

JSON formatted file


Resource creation


{

"AWSTemplateFormatVersion" : "2010-09-09",

"Description" : "Create RDS with fixed username and password",

"Resources" : {

"MyDB" : {

"Type" : "AWS::RDS::DBInstance",

"Properties" : {

"AllocatedStorage" : "5",

"DBInstanceClass" : "db.m1.small",

"Engine" : "MySQL",

"EngineVersion" : "5.5",

"MasterUsername" : "MyName",

"MasterUserPassword" : "MyPassword"

}

}

}

}

Snippet: Creating RDS instance

Template examples

Elastic Beanstalk

CloudFormation

Tools

Template

JSON formatted file


Resource creation


"AWS::CloudFormation::Init" : {

"config" : {

"packages" : {

"yum" : {

"mysql" : [],

"mysql-server" : [],

"httpd" : [],

"php" : [],

"php-mysql" : []

}

},

"sources" : {

"/var/www/html" : "https://s3.amazonaws.com/

cloudformation-examples/

CloudFormationPHPSample.zip"

},

Snippet: Installing software

The benefit of templated environments

Elastic Beanstalk

CloudFormation

Tools

Template

Procedural definition Create it programmatically


Elastic Beanstalk

CloudFormation

Tools

Template


Parameter driven Dynamic and user-driven

templates


Elastic Beanstalk

CloudFormation

Tools

Template



templates

Collaboration Share templates with ease as just files


Elastic Beanstalk

CloudFormation

Tools

Template


Known configuration Store stack configuration in source control


templates

Collaboration Share templates with ease as just files

NASA TV & Curiosity Route 53 DNS Multi-region & AZ

Elastic Loadbalancing Amazon Live Streaming

Cloud Front Cloud Formation




Template ELBs to front secondary cache

~100 Nginx secondary cache servers

2-3 Nginx mid-tier cache servers

Stack

CloudFormation template defined video caching

infrastructure




Availability Zone

As demand increased new stacks added to an availability

zone using template

One-click complex

rollout

Where to get more information?

Elastic Beanstalk

CloudFormation

Tools

http://aws.amazon.com/cloudformation/aws-cloudformation-templates/

Server templates High availability configurations

RDS configurations Open source stacks

Snippets

+ Cloud Former (prototype)

Where to get more information?

Elastic Beanstalk

CloudFormation

Tools

http://aws.amazon.com/cloudformation/aws-cloudformation-templates/

Server templates High availability configurations

RDS configurations Open source stacks

Snippets

+ Cloud Former (prototype)


Elastic Beanstalk

CloudFormation

Tools

Amazon EC2 Instances (On-Demand Instances, Spot Instances, and Reserved Instances) Amazon Elastic Block Store (EBS) Volumes Elastic Load Balancers Elastic IP Addresses Amazon EC2 Security Groups Auto Scaling Groups Amazon Relational Database Service (RDS) Instances Amazon RDS Security Groups Amazon ElastiCache Clusters Amazon ElastiCache Security Groups Amazon ElastiCache Parameter Groups

Amazon CloudWatch Alarms Amazon Simple Storage Service (S3) Buckets

Amazon Simple DB Domains Amazon Simple Queue Service (SQS) Queues

Amazon Simple Notification Service (SNS) Topics Amazon SNS Subscriptions

Amazon Route 53 DNS Records Amazon CloudFront Distributions and Multi-Origin support

AWS Identity and Access Management users and groups AWS Identify and Access Management policies

Amazon Virtual Private Cloud VPCs, Subnets, Gateways Amazon DynamoDB Tables


Elastic Beanstalk

CloudFormation

Tools

Amazon EC2 Instances (On-Demand Instances, Spot Instances, and Reserved Instances) Amazon Elastic Block Store (EBS) Volumes Elastic Load Balancers Elastic IP Addresses Amazon EC2 Security Groups Auto Scaling Groups Amazon Relational Database Service (RDS) Instances Amazon RDS Security Groups Amazon ElastiCache Clusters Amazon ElastiCache Security Groups Amazon ElastiCache Parameter Groups

Amazon CloudWatch Alarms Amazon Simple Storage Service (S3) Buckets

Amazon Simple DB Domains Amazon Simple Queue Service (SQS) Queues

Amazon Simple Notification Service (SNS) Topics Amazon SNS Subscriptions

Amazon Route 53 DNS Records Amazon CloudFront Distributions and Multi-Origin support

AWS Identity and Access Management users and groups AWS Identify and Access Management policies

Amazon Virtual Private Cloud VPCs, Subnets, Gateways Amazon DynamoDB Tables

+ AWS Elastic Beanstalk

High level, shrink

wrapped


control

Elastic Beanstalk

CloudFormation

Tools





High level, shrink

wrapped


control

Elastic Beanstalk

CloudFormation

+

+

High level, shrink

wrapped


control

Elastic Beanstalk

CloudFormation

Developer sandboxes with production ready

infrastructure

+

+

Comprehensive control of all aspects of infrastructure

High level, shrink

wrapped


control

Elastic Beanstalk

CloudFormation

Middle ground

AWS OpsWorks





Powerful management framework with Chef support

Elastic Beanstalk

CloudFormation

Stack Layers Management

Managed

environment

Definition of environment such as production or test

Management

services

Scaling, cloning, user access, self healing

Collection of

resources

Blueprint for a collection of resources (instances, EBS, EIPs

etc)

OpsWorks

Apps

Your application

assets

Resources to deploy and run in layers

Simple stack creation through layers

Elastic Beanstalk

CloudFormation

OpsWorks

Layers

Load balancing

HA Proxy installation in an availability zone

Application container

Static, Node.js, Rails, PHP

Database layer

MySQL

Simple stack creation through layers

Elastic Beanstalk

CloudFormation

OpsWorks

Layers Ganglia

MemCached

Custom layers

+

Scaling and auto-healing

Elastic Beanstalk

CloudFormation

OpsWorks

Auto healing

Replacement of an ec2 instance regardless of

scaling thresholds

i.e. always maintain 3 nodes across 3 AZs


Elastic Beanstalk

CloudFormation

OpsWorks

Auto healing

Replacement of an ec2 instance regardless of

scaling thresholds

i.e. always maintain 3 nodes across 3 AZs

Time based scaling

Run an ec2 instance on a given schedule

i.e. run these instances only during weekend peaks

Load based scaling

Add and remove instances from a layer based upon

load rules

i.e. run these instances only when CPU goes over 80%


Elastic Beanstalk

CloudFormation

OpsWorks


Elastic Beanstalk

CloudFormation

OpsWorks Lo

ad b

ased

Ti

me

bas

ed

App deployments

Elastic Beanstalk

CloudFormation

OpsWorks

App deployments

Elastic Beanstalk

CloudFormation

OpsWorks

App deployments

Elastic Beanstalk

CloudFormation

OpsWorks

Customization with chef recipes

Elastic Beanstalk

CloudFormation

OpsWorks

Get started fast with Beanstalk (control the chaos)

Take fine grained control with CloudFormation

(auditable environment specs)

Manage with layers and recipes with OpsWorks

(control with chef customization)

http://aws.amazon.com/elasticbeanstalk/

http://aws.amazon.com/cloudformation/

Find out more…

http://aws.amazon.com/opsworks/

aws.amazon.com

get started with the free tier

Technology

AWS - Managing Your Cloud Assets 2013