Wayin devops-2013

DevOps at WayinOrchestration with AWS Cloud Formation

David M. JohnsonSoftware Developer - Wayin, Inc.

Friday, June 21, 13

Agenda

•What is Cloud Orchestration

• Quick intro to Wayin Hub

• Wayin Hub deployment & application architecture

• How we scale, deploy and do backups

• About Cloud Formation Templates

• About vendor lock-in

• Designing for orchestration

Friday, June 21, 13

“Orchestration describes the automated arrangement, coordination, and management of complex computer systems, middleware, and services.

It is often discussed as having an inherent intelligence or even implicitly autonomic control, but those are largely aspirations or analogies rather than technical descriptions.”

http://en.wikipedia.org/wiki/Orchestration_(computing)

Friday, June 21, 13

“cloud orchestration = automation

+ integration + best practices”

https://www.ibm.com/developerworks/mydeveloperworks/blogs/9e696bfa-94af-4f5a-ab50-c955cca76fd0/entry/orchestrating_the_cloud_to_simplify_and_accelerate_service_delivery1

Friday, June 21, 13

Cloud Orchestration is...

Automating the infrastructure needed to provide a computing service so that resources are automatically allocated and de-allocated to meet changing demand

Friday, June 21, 13

The Orchestra Analogy

Friday, June 21, 13

Benefits

•Saves money

•Saves time

•Repeatable, version-able process

•Less human intervention

•Fewer stupid mistakes

Friday, June 21, 13

Agenda

• What is Cloud Orchestration

•Quick intro to Wayin Hub

• Wayin Hub deployment & application architecture

• How we scale, deploy and do backups

• About Cloud Formation Templates

• About vendor lock-in

• Designing for orchestration

Friday, June 21, 13

Friday, June 21, 13

Friday, June 21, 13

Friday, June 21, 13

Friday, June 21, 13

Friday, June 21, 13

Friday, June 21, 13

Friday, June 21, 13

Agenda

• What is Cloud Orchestration

• Quick intro to Wayin Hub

•Wayin Hub deployment & app architecture

• How we scale, deploy and do backups

• About Cloud Formation Templates

• About vendor lock-in

• Designing for orchestration

Friday, June 21, 13

Glossary• EC2 - server virtualization

• EBS - mountable block storage

• ELB - load balancer

• R53 - domain name services (DNS)

• S3 - file / bucket storage service

• CloudFront - content distribution network (CDN)

• Autoscale - automatically stop & start servers

• CloudWatch / Cloud Alarms - monitoring & triggering

• Security Group - restrict port access on group of instances

• SQS - Simple Queuing Service

Friday, June 21, 13

Application Architecture

Browser

Front end

HTML5jQuery

Backbone

Ubuntu

Back end

Java, JettyJerseyGuice

RESTHTTP

Ubuntu

Cassandra

Redis

Memcache

Queue Service

Twitter Feeds Other Soc. Nets

Friday, June 21, 13

Ubuntu

Jenkins

API Security Group

S3

R53CloudFront

ELB

EC2Browser

Front end

DB Security Group

EBS

SQS

TwitterFeeds

Tracker Security Group

Deployment Architecture

Cloud Formation

// CFN template { "Parameters": {

}, "Mappings": {

}, "Resources": {

}}

Friday, June 21, 13

Cloud Formation

“An easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.

You can use AWS CloudFormation’s sample templates or create your own. Once deployed, you can modify and update the AWS resources in a controlled and predictable way allowing you to version control your AWS infrastructure in the same way as you version control your software.

You can deploy and update a template and its associated collection of resources (called a stack) via the AWS Management Console, CloudFormation command line tools or APIs. CloudFormation is available at no additional charge, and you pay only for the AWS resources needed to run your applications.”

Friday, June 21, 13

Agenda

• What is Cloud Orchestration

• Quick intro to Wayin Hub

• Wayin Hub deployment & application architecture

•How we scale, deploy and do backups

• About Cloud Formation Templates

• About vendor lock-in

• Designing for orchestration

Friday, June 21, 13

Separate auto-scale groups

Ubuntu

API server

WAR

Jetty

Ubuntu

API server

WAR

Jetty

Ubuntu

API server

WAR

Jetty

Min: 3Max: 9

Scale up (quickly):CPU > X% for Y minutes

Scale down (slowly):CPU < A% for B minutes

Ubuntu

Cassandra

Redis

Memcache

DB serverUbuntu

Cassandra

Redis

Memcache

DB server

Min: 2Max: 2

We decide when to scaleup and scale down our DB server instances

Ubuntu

Tracker

WAR

Jetty

Ubuntu

Tracker

WAR

Jetty

Ubuntu

Tracker

WAR

Jetty

Min: 2Max: 20

Scale up (quickly):Queue has > X messages for Y minutes

Scale down (slowly):Queue has < X message for Y minutes

Friday, June 21, 13

CFN Launches new API server instances

1

Self-configuring API servers

DB Security Group

S3 ELBCloudInit script downloads the

deployment bundle2

CloudInit scripts find DB servers by Security Group, starts Jetty

3

CloudInit scripts put server into ELB rotation

4

API Security Group

CFNCloud Watch

Friday, June 21, 13

Self-configuring DB servers

DB Security Group

S3

CloudInit scripts finds other DB servers by Server Group3

CloudInit scripts download latest

deployment bundle

2Human runs the command to increase

DB servers by one

1

Ubuntu

New DB server

New Cassandra node joins cluster when ready7

CFN

CloudInit scripts write Memcache config, start it4

MemcacheCloudInit scripts write Redis config, assume first server is master, start it5

RedisCloudInit scripts write Cassandra config, start it6

Cassandra

Friday, June 21, 13

Automated deployment

API Security Group

GitDeveloper merges code into deployment branch 1

Developer kicks off Jenkins build 2

S3

Jenkins copies bundle to S3 bucket 3

Developer kicks off deployment build 4

Jenkins runs deploy command

via SSH

5

On each server the script runs locally, downloads and installs the bundle

6

Ubuntu

Jenkins

Tracker Security Group

Friday, June 21, 13

Incremental backups copied to S3 bucket5

Automated Backups

S3

Jenkins runs daily backup job

1

Backup script creates EBS snapshot 3

DB Security Group

Cronjob kicks off incremental backups4

Cassandra snapshot backups happen simultaneously

2

Ubuntu

Jenkins

EBS

Friday, June 21, 13

Agenda

• What is Cloud Orchestration

• Quick intro to Wayin Hub

• Wayin Hub deployment & application architecture

• How we scale, deploy and do backups

•About Cloud Formation Templates

• About vendor lock-in

• Designing for orchestration

Friday, June 21, 13

CFN Template skeleton{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "Example stack",

"Parameters": {

},

"Mappings": {

},

"Resources": {

}}

Friday, June 21, 13

CFN Parameters{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "WayinHub Three Server-type Prod Stack",

"Parameters": {

"DnsName" : { "Description" : "DNS name to be created", "Type" : "String", "Default": "stack-test" },

"DnsDomain" : { "Description" : "DNS domain to be used", "Type" : "String", "Default": "twt.pl" },

"KeyPair" : { "Description" : "EC2 Key Pair to allow SSH...", "Type" : "String", "Default": "wayin" },

Friday, June 21, 13

CFN Parameters "ReleaseBucket" : { "Description" : "S3 Bucket of release bundle", "Type" : "String", "Default": "hub-deployment-release" }, "ApiMinServers" : { "Description" : "Minimum number of API servers", "Type" : "Number", "Default": "2", "MinValue": "2" },

"ApiMaxServers" : { "Description" : "Maximum number API servers", "Type" : "Number", "Default": "5", "MinValue": "2" },

"ApiInstanceType": { "Description": "EC2 instance type for API servers", "Type": "String", "Default": "m1.large", "AllowedValues": ["m1.xlarge", "m3.xlarge"], "ConstraintDescription": "must be a valid EC2 type." },

Friday, June 21, 13

CFN Mappings

"Mappings": { "AWSRegionArch2AMI": { "us-east-1": { "64": "ami-e864da81" }, "us-west-2": { "64": "ami-50f97060" } }, "TwoAZs": { "us-east-1": { "AZ1" : "us-east-1a", "AZ2" : "us-east-1b" }, "us-west-2": { "AZ1" : "us-west-2a", "AZ2" : "us-west-2b" } } },

Friday, June 21, 13

CFN Resources "Resources": { "DbAutoScalingLaunchConfiguration": { "Type": "AWS::AutoScaling::LaunchConfiguration", }, "DbAutoScalingGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", }, "ApiAutoScalingLaunchConfiguration": { "Type": "AWS::AutoScaling:: LaunchConfiguration", }, "ApiAutoScalingGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", }, "ApiServerScaleUpPolicy" : { "Type" : "AWS::AutoScaling::ScalingPolicy", }, "ApiServerScaleDownPolicy" : { "Type" : "AWS::AutoScaling::ScalingPolicy", }, "CPUAlarmHigh": { }, "CPUAlarmLow": { },

Friday, June 21, 13

CFN Resources"ApiAutoScalingLaunchConfiguration": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Properties": { "UserData": { "Fn::Base64" : { "Fn::Join": [ "", [ "#!/bin/bash -ex\n",

"# Log all output from this script\n", "exec >/var/log/wayin-bootstrap.log 2>&1\n",

"# Expose various environmental variables, etc.\n", "cat >/etc/profile.d/wayin-env.sh <<EOF\n", "export ELB_NAME=", { "Ref" : "ApiElasticLoadBalancer" }, "\n", "export EC2_REGION=", { "Ref" : "AWS::Region" }, "\n", "export DNS_NAME=", { "Ref" : "DnsName" }, "\n", "export DNS_DOMAIN=", { "Ref" : "DnsDomain" }, "\n", "EOF\n", "# Install various packages\n", "apt-get update\n", "apt-get -y install s3cmd\n", "\n", "# Download our application files\n", "cd /usr/share/wayinhub\n", "s3cmd --config=/etc/s3cfg get s3://",{"Ref":"ReleaseBucket"},"/twithub.tgz\n", "tar xf twithub.tgz\n", "rm -fr twithub.tgz\n", "cd ./init_instance\n", "sh ./init_api_server.sh\n" ]]} },

Friday, June 21, 13

CFN Console

Friday, June 21, 13

Agenda

• What is Cloud Orchestration

• Quick intro to Wayin Hub

• Wayin Hub deployment & application architecture

• How we scale, deploy and do backups

• About Cloud Formation Templates

•About vendor lock-in

• Designing for orchestration

Friday, June 21, 13

Lock-in in automation

• Cloud Formation itself is lock-in

• Plenty of AWS dependencies in our:

• Instance init scripts

• Deployment scripts

• Backup scripts

Friday, June 21, 13

No lock-in in app code

• Bulk of our code is in our application

• AWS dependencies are pluggable

Friday, June 21, 13

Agenda

• What is Cloud Orchestration

• Quick intro to Wayin Hub

• Wayin Hub deployment & application architecture

• How we scale, deploy and do backups

• About Cloud Formation Templates

• About vendor lock-in

•Designing for orchestration

Friday, June 21, 13

Choose right components

• Good

• Peer-to-peer

• Linear scalability

• Bad

• Master-slave

• “Special” servers

• Single point-of-failure

Friday, June 21, 13

Bootstrap everything

• Every instance must self-configure

• Make services easy to discover

Friday, June 21, 13

Keep it simple

• Minimize number of different...

• Server types

• External services

• Learning curves

Friday, June 21, 13

Questions?

Friday, June 21, 13