Immutable Deployments with AWS CloudFormation and AWS Lambda

(and AWS Lambda)

& aoepeople!

E-Commerce: Magento

CMS: TYPO3

Portals: ZF, Symfony,…

Mobile Searchperience: ElasticSearch

250+ people world-wide

(in 8 locations)

Global Enterprise Projects

Infrastructure: AWS

CloudFormation Lambda “Immutable”

/i(m)ˈmyo͞odəb(ə)l/

adjective

unchanging over time or unable to be changed.

“disposable” “ephemeral”

Pet Cattle

not disposable disposable

disposable

not disposable

disposable

Static Resources Build VPC Build

Build Bucket

IAM Setup

VPC

Static Resources

Build

Manual Setup

separate

CloudFormation

Stacks use resources

from underlying stacks as input

parameters Lambda functions,

Monitoring,…

Build Bucket

IAM Setup

Environment A (e.g. prod) Environment B (e.g. stage)

VPC VPC

Static Resources Static Resources

Build Build Build Build

Build X Build X+1 Build X Build X+1

Manual Setup

different scopes (build, environment,

account, global…)

Build Bucket

IAM Setup

Environment A (e.g. prod) Environment B (e.g. stage) Environment C (e.g. qa) Environment D (e.g. dev)

IAM Setup

VPC VPC VPC VPC

Static Resources Static Resources Static Resources Static Resources

Build Build Build Build Build Build Build Build

Build X Build X+1 Build X Build X+1 Build X Build X+1 Build X Build X+1

Manual Setup Manual Setup

Build Bucket Access

Private subnets

Public subnets

ElastiCache (Redis)

with replication groups

for cache and sessions

RDS (multi-az) with

DB subnet group

Bastion

server

s3: media

storage*

Route 53: DNS

configuration CloudFront

distribution

SSL

Certificates

Security group for Varnish servers

Security group for Magento servers

Security group for Load Balancer

Static Resources

Build

Auto Scaling group

Auto Scaling group

Elastic Load

Balancer

Auto-

Scaling

Group

Launch

Configurati

on

Scaling

Policy

Build

Auto Scaling group

Auto Scaling group

Elastic Load

Balancer

Auto-

Scaling

Group

Launch

Configurati

on

Scaling

Policy

Auto Scaling group Auto Scaling group

+1

Baking AMIs

“Chef vs. Puppet?”

http://fbrnc.net/blog/2015/11/

how-to-provision-an-ec2-instance

“…Ansible!” “BASH!”

Keep it simple…!

✔ ✔ (✔)

the “last mile”

most underestimated CFN feature!

var r = require('cfn-response'); exports.handler = function (event, context) { […] var res = {}; if (event.RequestType == 'Create') { res.Password = randomPassword(20); } r.send(event, context, r.SUCCESS, res); };

"IndexerDb": { "Type": "AWS::RDS::DBInstance", "Properties": { [...] "MasterUserPassword": {"Fn::GetAtt": ["GenerateDbPassword", "Password"]}, [...] } },

"GenerateDbPassword": { "Type": "Custom::PasswordGenerator", "Properties": { "ServiceToken": {"Ref": "PasswordGeneratorArn"} } },

• launch some ASGs (set DesiredCapacity) • create database passwords • tag all resources (incl. ElastiCache!) • restore database and media files • (one-time) install scripts (db migrations,…) • detect Varnish backends • wait for healthy backends in the ELBs • run infrastructure tests • cache warming • update Route53 records sets • delete old stacks • …

DesiredCapacity =

Number of Instances in current ASG x 1.2

better safe than sorry…

DesiredCapacity =

Number of Instances in current ASG x 1.2

DesiredCapacity =

Number of Instances in current ASG x 1.2

ScalingPolicy takes care…

"CountInstances": { "Type": "Custom::InstanceCounter", "Properties": { "ServiceToken": {"Ref": "InstanceCounter"}, "AutoScalingGroupTags": [ {"Key": "Environment", "Value": "prod"}, {"Key": "Type", "Value": "Frontend"} ], "Min": 1, "Max": 10, "Factor": "1.5" } },

"FrontendAsg": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { [...] "DesiredCapacity": {"Fn::GetAtt": ["CountInstances", "Count"]}, "Tags": [ {"Key": "Environment", "Value": "prod", "PropagateAtLaunch": true}, {"Key": "Type", "Value": “Frontend", "PropagateAtLaunch": true} ] } },

Update Route53 record set

Update Route53 record set

Delete old stacks

"UpdateR53": { "DependsOn": [ "MagentoWaitCondition", "MagentoWorkerWaitCondition", "Elb"], "Type": "Custom::Route53Update", "Properties": { "ServiceToken": {"Ref": "R53Updater"}, "Name": {"Fn::Join": ["", [ "www-", {"Ref": "EnvironmentName"} ]]}, "HostedZoneId": {"Ref": "HostedZoneId"}, "AliasTargetDNSName": {"Fn::GetAtt": ["Elb", "DNSName"]}, "AliasTargetHostedZoneId": {"Fn::GetAtt": ["Elb", "CanonicalHostedZoneNameID"]}, "Comment": "Updated via CloudFormation/Lambda" } },

"DeleteStacks": { "Condition": "DeleteOldStacks", "DependsOn": ["UpdateR53"], "Type": "Custom::StackDeleter", "Properties": { "ServiceToken": {"Ref": "StackDeleter"}, "TagFilter": { "Environment": {"Ref": "EnvironmentName"}, "Type": "Deployment"}, "ExceptStackName": {"Ref": "AWS::StackName"} } }

+1

Baking AMIs

Tools+CI

Have fun filling out 47 form fields! :)

aoepeople/stackformation

command-line tool (Symfony console, uses

AWS SDK for PHP)

integrates nicely into your CI (Jenknis,…)

pre-process

Userdata

Lambda JS “Template”

superset of CFN template (JSON)

Parameters lookup from other stacks, env vars,…

Template

Parameter

Values

create/ update

Stack policies, Tags,…

Stack magento-stage-build-5

Stack magento-stage-build-6

Stack magento-prod-build-6

Stack magento-prod-build-5

CloudFormation Template

merge & pre-process

“CloudFormation+X” Template(s)

+ Dynamic Parameters

+ Stack Policies

+ Behavior

+ Tags …

Blueprint magento-{env:ENVIRONMENT}-build-{env:BUILD}

Blueprint magento-{env:ENVIRONMENT}-setup

Stack magento-stage-setup

Stack magento-prod-setup

blueprints: - stackname: 'magento-{env:BUILD}' template: 'magento.template' stackPolicy: 'policy.json' OnFailure: 'DO_NOTHING' parameters: Build: '{env:BUILD}' KeyPair: '{var:KeyPair}' VPC: '{resource:setupstack:VPC}' Subnet: '{resource:setupstack:Subnet}' InstanceSg: '{resource:setupstack:InstanceSg}' InstanceProfile: '{output:setupstack:InstanceProfile}' BootAmi: 'ami-06116566' tags: Environment: 'prod' Build: '{env:BUILD}'

enforce “immutability” by denying updates!

aoepeople/cfn-vpc

aoepeople/cfn-lambdahelper

aoepeople/cfn-amibaker

via composer

so we can integrate this into our CI pipeline…

•

aoepeople/awsinspector

command-line tool (Symfony console, uses

AWS SDK for PHP)

Domain models for PHP

$repository = new \AwsInspector\Model\Elb\Repository(); $dns = $repository->findElbsByTags([ 'Environment' => 'deploy', 'Build' => 554, 'Type' => 'Frontend’ ])->getFirst()->getDNSName();

> bin/awsinspector.php ec2:ssh -t Environment:prod –c Type –c Build

filter by tag

Please select an instance [0] i-1033ed9b (Type: Frontend; Environment: prod; Build: 477) [1] i-4ff36ec8 (Type: Backend; Environment: prod ; Build: 477) [2] i-5ab4322b (Type: Worker; Environment: prod; Build: 477) [3] i-705ad42f (Type: Worker; Environment: prod; Build: 476) >

• will take jump hosts into account (ProxyCommand)

• auto-detects your local

(encrypted) private keys

• multiplexed ssh connections

• run commands directly

logins, orders,…

deployments, scaling activity,…

JMeter response time, error rate,…

CPU, load, network I/O,…

correlate metrics from

various sources

Grafana ElasticSearch (Service)

time-series database

leftovers from “disposed” resources

…

“Ready for testing”

(WaitCondition)

setup terminate

instances

• Install JMeter

• Download

testcase from

S3

• Signal “Ready”

and wait

• Create Security

allowing access

to all IPs and

attach to UAT’s

ELB

Auto-Scaling Group

of Load Generator Instances

Spin up test

environment (incl. restoring db+media

snapshot from prod)

Run Stress Test

Delete test

environment* Delete self*

1.

2.

3. 4.

Time

timeout 3600 jmeter –n –t testcase.jmx

*via Lambda custom resource

Pushing samples in near real-time

via custom JMeter backend

listener

“Testing Completed”

(WaitCondition)

JMeter test

CloudFormation Stack

CloudFormation

Stack

Grafana ElasticSearch (Service)

CloudWatch

Lambda

find relevant resources for the current deployment (via tags),

collect metrics, and push them to ElasticSearch

CloudFormation Stack

• Install JMeter

• Download

testcase from S3

• Signal “Ready”

and wait

• Create SG

allowing access

to all IPs and

attach to UAT’s

ELB

Auto-Scaling Group of Load Generator Instances

Run Stress Test 2.

Time

timeout 3600 jmeter –n –t testcase.jmx

Delete test

environment* Delete self*

3. 4.

*via Lambda custom resource

Lambda

Grafana ElasticSearch (Service)

CloudWatch find relevant resources for the

current deployment (via tags),

collect metrics and push them to

ElasticSearch

…

Spin up test

environment (incl. restoring db+media

snapshot from prod)

1.

CloudFormation

Stack

setup terminate

instances JMeter test

Pushing samples in near real-time

via custom JMeter backend listener

“Ready for testing”

(WaitCondition)

“Testing Completed”

(WaitCondition)

https://blogs.aws.amazon.com/application-

management/post/Tx38Z5CAM5WWRXW/Faster-Auto-Scaling-in-AWS-

CloudFormation-Stacks-with-Lambda-backed-Custom-Resou

Provisioning

“Ready for baking”

(WaitCondition)

EC2

AMI Baker

(Lambda) Node.js SDK

AWS CLI

“Baking Completed”

(WaitCondition)

shutdown

(-> terminate)

ec2.createImage

aws ec2 wait image-available

AMI Baker

(Lambda) Node.js SDK

ec2.deleteImage

Custom

CloudFormation

Resource

Time

•

Follow me on twitter!

My blog