Real World CloudFormationAWS User Group Amsterdam

February 2014 !

Howard Glynn Cloud Leader | Cloudreach

howard.glynn@cloudreach.com | @hglynn

Objectives

• The creation process

• Template approaches

• Bootstrapping and managing resources

• The importance of roles

• Other Concepts

• Protection strategies

• Recommendations

!2

!!

!Won’t dive into too much code detail,but will assume familiarity with AWS!

Please feel free to ask questions!

Cloudreach

• AWS Premier Consulting Partner

• London, Amsterdam, Edinburgh, Vancouver…

!3

cloudreach.com

• Describes resources, relationships in a template

• Rich feature set - most AWS services supported

• Complements other AWS and 3rd party tools

• Free to use, but created resources are not

• “Infrastructure as Code”

• Can be tricky!

About CloudFormation

“DMG201”

More Info:

!4

Why Use It?

• Controllable and predictable

• Saves time, helps save money

• Dev, Staging, Prod

• Forces more discipline, captures knowledge

• Multi-region provisioning

• Can aid disaster recovery

• It’s amazing to watch!

!5

Creation Process

• Open your editor, write JSON

• IAM credentials

• Use existing fragments/templates

• Sometimes hard to debug

• An iterative, additive process works well - edit/run/edit...

• Beyond a certain point, you cannot practically manage the JSON - too complex

• Footnote: CloudFormer

!6

• Troposphere (see github) o python code to describe resources

o Often 50% lines less than JSON

o Re-usable, testable

• And/or choose your tools! o Editors: Take time to learn code folding!

o Browsers: Use JSON formatters

o AWS CLI is essential

Real world approaches...

!7

Templates

Template Contents

• Parameters o up to 60 nowadays

• Mappings o for example, mapping regional AMIs (but other approaches…)

• Resources o the bulk of the template - e.g. subnets, security groups, instances,

databases, service logic...

• Outputs o for example, created endpoints, IDs for reference or chaining

!9

Monolithic vs Modular CF

• Monolithic o most examples take this approach

o quick to start, no parameter passing

o least easy to maintain when it grows

o easy to hit CF console upload limits (-> S3)

• Modular o smaller, more manageable

o far more parameter passing - hard

o think about layers, outputs, chaining

!10

cf json

cf json

cf json

cf json

cf json

or

Summary

!11

Monolithic Modular

Simple and Quick

✓ All logic in one place

✘ Logic spanning multiple

files

DRY (Don’t Repeat

Yourself)

✘ Repeated components

must be written multiple times

✓ Same file can be used

many times with different params

Maintainable

✘ Hard to find correct

elements and relationships

✓ Different logical concepts

in different files

Outline 3-Layer Modular CF Web Stack

!12

Layer Name Typical Parameters Typical Outputs

10-base Sets up VPC, NAT etc

VPC CIDR Public Subnet CIDRs NAT Size, EIP SSH Keynames

VPC ID Subnet IDs, SGs Private Route Table

20-db Sets up RDS

VPC ID Private Route Table Private DB Subnet CIDRs DB Name, Size etc

DB Endpoint DB Client SG ID

30-web Autoscaling instances, ELB etc

VPC ID Private Route Table Private Web Subnet CIDRs DB Endpoint, Client SG Public Subnet SGs for ELB Web Instance Params Orchestration Params ….

ELB Endpoints R53

Bootstrapping Instances

How to achieve the automation goal

Our usual goal is a hands-free automated bring up of our resources. Therefore we need to configure instances automatically. A combination of methods can be used:

!1. user-data

2. cfn-init / cloud-init metadata

3. bootstrap orchestration tools

!14

Instance User-data

• Simplest

• Typically o called one-time by EC2 instance

o bash or other shell script, 16kb limit

o available at the http://169.254.169.254/… metadata endpoint

!15

cfn-init / metadata

• If your needs are straightforward, use it!

• Installs o Fetch and parse metadata from CloudFormation

o Installs packages

o Write files to disk

o Can pull code from your github,

o setup users (not windows)

o Enabled/disable and start/stop services

• Need to call cfn-init once in user-data

• Canonical’s cloud-init is similar, not as well supported

!16

Instance Roles

• associate a role to an instance

• roles feature IAM policies

• policies give permissions

• you cannot change the IAM role ona running instance so best practiceis to assign one, even if empty!

!17

Securely accessing S3

• If your instance has a role/policy…

• Combine with metadata

• Securely access S3

• Blog post, Nov 2013http://goo.gl/6bVRLP

• Why…?

!18

Bootstrapping to Chef

• Combo of metadata, role-based S3 access, user-data

• Securely grab validation keys

• Install base chef packages or omnibus installer from metadata / user-data

• Bootstrap the node with the creds (user-data)

• Machine is now under hosted chef control

• Same applies to puppet?

!19

Other Concepts

Wait Cond. and Signals

• In the real world, these can no longer be ignored!

• Stacks now build in parallel

• Examples: o VPC with NAT, no instance

builds until working

o DB before Web, no instance builds until DB available

!21

Windows Instances

• as supportable as Linux instances

• all the same principles apply

• use cloudformation::init to download S3 files

• run individual commands or powershell

• same password retrieval requirement

!!

!22

Only time for a quick mention…

• Tagging

• Notifications

• Custom Resources

!23

“DMG303”

Custom Resources: More Info:

Wrapping Up

Protect Your Stacks

• IAM Roles

• Stack Policies

• DisableApiTermination Protection

• Layer dependencies canbe your friend

• DeletionPolicy attribute

• Have separate DEV andPROD accounts

• For critical actions, stop, check, do it slow!

!25

Spider-Man (c) Marvel

Be aware...

• No dry runs

• Uncontrolled changes (hacks) - actual vs expected?

• Some AMIs/Licensing don’t like CF

• Dependencies are sometimes non-obvious (EIPs and IGWs)

• Blue/green deployments are hard

• Cannot absorb pre-existing resources

!26

Some real world best practices

• Use IAM deny o make users assume a delete role

• Setup Elastic IPs externally o for fixed endpoints o avoids losing them o supply them as parameters

• Don’t embed creds, passwords etc o Use instance roles, secured S3 buckets o Cloud init to pull them to instances

• Don’t let layers get too big/complex o You’ll be too scared to update live

!27

Some real world best practices

• Meaningful stack and resource names o e.g. prefix the environment onto instance names

• CF allows for tags o ENVIRONMENT, STACK, OWNER etc

o e.g. we can control costs by stopping overnight

o leverage the cloud

• Get your dependencies right! o otherwise hard to diagnose intermittent stack fails

o Two types: Implicit “Refs”, Explicit “DependsOn”

!28

Some real world best practices

• Use the AWS CLI ( http://goo.gl/QASmbS )) o supports passing parameters in files

• Take the time to: o parameterise inputs -> reusability

o add potentially useful outputs

• Do your dev/test away from prod o in a different region (validates reusability!)

o in a different account

!29

!30

cloudreach.com/careers/

Dank je wel ! !Questions, comments, thoughts?!

!31

!32