Lessons Learned Managing Large AWS Environments

Lessons learned managing large

AWS EnvironmentsRonald Bradford

http://ronaldbradford.com @RonaldBradford

2013.06

EffectiveMySQL.com - Performance, Scalability & Business Continuity

Consulting experiences with AWS

Several different clientsLargest - 500+ servers

Some 40-50+ servers

Some 2-5 servers

LAMP/RoR/RDS/Windows

ABOUT MySELF

Enterprise Data Architecture

24 years with RDBMS - 13 years with MySQL

Using AWS 4+ years

Published author - 4 books

Accomplished presenter - 8 years

Work at Independent MySQL Consultant

Ronald BRADFORD

Covering

1. Products

2. Cost

3. Web Scale

4. Security

5. Instrumentation

6. Failure

AWS Products & Ecosystem

ABOUT AWS

Many, many products and features

EC2, S3, EBS, ELB, RDS, EMR, VPC, CDN, SWF, SQS, SES, SNS, IAM, ...

Mechanical Turk

Flexible Payments Service (FPS)

AMAZON WEB SERVICES30+

AWS CONSOLE

May 2013 Aug 2012

Announcements

Product Announcements

Pricing Changes

New instance types

New features (e.g. IOPS)

New Products (e.g. Redshift/ OpsWorks)

http://aws.amazon.com/about-aws/newsletters/

Announcements

Product Announcements

Pricing Changes

New instance types

New features (e.g. IOPS)

New Products (e.g. Redshift/ OpsWorks)

Examples in presentation

http://aws.amazon.com/about-aws/newsletters/

ECOSYSTEM

AWS Marketplacehttps://aws.amazon.com/marketplace/

Over 800

Product growth

When I started

No RDS, In-memory Cache, DynamoDB, Glacier

No Elastic Beanstalk, OpsWorks

No management console

AWS Costs2

operating cost

Are you monitoring your costs?

Hourly

Operating Cost

https://github.com/ronaldbradford/aws

$ ec2_cost.sh

Operating Cost

https://github.com/ronaldbradford/aws

$ ec2_cost.sh

$29,000 p.m.

Your Money

What is AWS costing you?

Instance types/sizes

Cost options

http://aws.amazon.com/ec2/instance-types

http://aws.amazon.com/ec2/pricing

Instance Types

General-purpose

Compute-optimized

Memory-optimized

Storage-optimized

Instance Prices

$Large Instance (m1.large)

Instance Prices

$On Demand $0.24 Per hour investment

Reserved $0.136 * + Annual contract ( +$ 0.043)

Spot $0.03+ * Can be terminated (budget)

Large Instance (m1.large)

Instance Prices

40% saving

up to 80+% saving

Was $0.32 til 11/19/2012Was $0.26 til 1/16/2013

Light/Medium/Heavy utilization

SPOT EXAMPLE

One hour (24 cents)

1 x Large - Reserved

7.5G, 4 CPUs, 850G

8 x Large - Spot

1 x Eight Extra Large - Spot (cc2.8xlarge)

60G, 88 CPUs, 3.4T,10Gb NIC

SPOT EXAMPLE

One hour (24 cents)

1 x Large - Reserved

7.5G, 4 CPUs, 850G

8 x Large - Spot

1 x Eight Extra Large - Spot (cc2.8xlarge)

60G, 88 CPUs, 3.4T,10Gb NIC

price has changed 3 times in 8 months

SPOT HISTORY

$ ec2-describe-spot-price-history -t m1.large -d Linux/UNIX SPOTINSTANCEPRICE 0.030000 2013-05-28T17:20:41-0500 m1.large Linux/UNIX us-east-1aSPOTINSTANCEPRICE 0.100000 2013-05-28T17:07:02-0500 m1.large Linux/UNIX us-east-1aSPOTINSTANCEPRICE 0.030000 2013-05-28T16:37:51-0500 m1.large Linux/UNIX us-east-1aSPOTINSTANCEPRICE 0.100000 2013-05-28T16:31:03-0500 m1.large Linux/UNIX us-east-1aSPOTINSTANCEPRICE 0.030000 2013-05-28T16:24:48-0500 m1.large Linux/UNIX us-east-1dSPOTINSTANCEPRICE 0.030000 2013-05-28T16:24:48-0500 m1.large Linux/UNIX us-east-1aSPOTINSTANCEPRICE 0.100000 2013-05-28T16:15:03-0500 m1.large Linux/UNIX us-east-1aSPOTINSTANCEPRICE 0.060000 2013-05-28T16:08:34-0500 m1.large Linux/UNIX us-east-1dSPOTINSTANCEPRICE 0.030000 2013-05-28T16:01:59-0500 m1.large Linux/UNIX us-east-1bSPOTINSTANCEPRICE 0.240000 2013-05-28T15:55:12-0500 m1.large Linux/UNIX us-east-1bSPOTINSTANCEPRICE 0.030000 2013-05-28T15:48:32-0500 m1.large Linux/UNIX us-east-1bSPOTINSTANCEPRICE 0.030000 2013-05-28T15:42:07-0500 m1.large Linux/UNIX us-east-1aSPOTINSTANCEPRICE 0.045000 2013-05-28T15:35:47-0500 m1.large Linux/UNIX us-east-1aSPOTINSTANCEPRICE 0.050000 2013-05-28T15:35:47-0500 m1.large Linux/UNIX us-east-1bSPOTINSTANCEPRICE 0.400000 2013-05-28T15:29:15-0500 m1.large Linux/UNIX us-east-1bSPOTINSTANCEPRICE 0.260000 2013-05-28T15:22:47-0500 m1.large Linux/UNIX us-east-1bSPOTINSTANCEPRICE 0.030000 2013-05-28T15:16:01-0500 m1.large Linux/UNIX us-east-1dSPOTINSTANCEPRICE 0.030000 2013-05-28T15:16:01-0500 m1.large Linux/UNIX us-east-1aSPOTINSTANCEPRICE 0.026000 2013-05-28T15:09:30-0500 m1.large Linux/UNIX us-east-1a

3c to 10c Zone A3c to 40c Zone B2013

SPOT HISTORY

$ ec2-describe-spot-price-history -t m1.large -d Linux/UNIX 0.0260 2012-09-27T09:45:46-0800 m1.large Linux/UNIX us-east-1b0.0260 2012-09-27T09:45:46-0800 m1.large Linux/UNIX us-east-1d0.0290 2012-09-27T09:38:37-0800 m1.large Linux/UNIX us-east-1b0.0370 2012-09-27T09:38:37-0800 m1.large Linux/UNIX us-east-1d0.0600 2012-09-27T09:31:29-0800 m1.large Linux/UNIX us-east-1b0.1700 2012-09-27T09:31:29-0800 m1.large Linux/UNIX us-east-1d0.1600 2012-09-27T09:24:20-0800 m1.large Linux/UNIX us-east-1d0.0600 2012-09-27T09:17:11-0800 m1.large Linux/UNIX us-east-1b0.0900 2012-09-27T09:17:11-0800 m1.large Linux/UNIX us-east-1d0.0260 2012-09-27T09:09:55-0800 m1.large Linux/UNIX us-east-1c0.0260 2012-09-27T09:09:55-0800 m1.large Linux/UNIX us-east-1b

2.6c to 17c (1/2 of 34c)One AZ only2012

Using SPOTS

Is your volume predicable?

Splitting on-demand/spot instances

Can work be done asynchronously?

i.e. can be queued

Is work restartable?

Using SPOTS

Is your volume predicable?

Splitting on-demand/spot instances

Can work be done asynchronously?

i.e. can be queued

Is work restartable? WARNING: Not for general workloads

Instance sizes

Evaluating the right instance size

What is your bottleneck?

Instance sizes

Evaluating the right instance size

What is your bottleneck?

Developing a tool to recommend savings

TRUSTED ADVISOR

AWS now offers Trusted AdvisorRecommendations to save money

Improve performance

Close security problems

http://aws.amazon.com/premiumsupport/trustedadvisor/

COST SAVINGS

Other players

http://www.newvem.com/http://www.cloudyn.com/

OTHER COST SAvings

CDN - Cloudfront

Bandwidth

Reduce response size (e.g. 10%)

Storage

old EBS snapshots

Remove unused instances

http://aws.amazon.com/cloudfront/

NEW: Announced 1/9/2103 CloudWatch Alarm Actions

Web Scale(hint: no humans)

ABOUT WEB SCALE

GUI = #FAIL

CLI is necessary

Manual CLI use is slow

Automation in crucial

Parallel

AWS CLI’s

Different for EC2, ELB, RDS etc

Updated frequently (i.e. monthly)

$ git clone https://github.com/ronaldbradford/aws.git$ cd aws/scripts$ ./aws_cli_configure.sh

AWS CLI’s

Different for EC2, ELB, RDS etc

Updated frequently (i.e. monthly)

$ git clone https://github.com/ronaldbradford/aws.git$ cd aws/scripts$ ./aws_cli_configure.sh

Simple helper

http://aws.amazon.com/archives/Amazon-EC2

Identifiers

Access Key ID

Private Access Key

X.509 Certificates (2 of)

Private (*) & Public

AWS Account ID

Canonical User IDhttps://portal.aws.amazon.com/gp/aws/securityCredentials

CLI Examples

Launch Script

Demand/Spot or switch between

Verify SSH

Verify MySQL

Verify replication in sync

Add to ELB

CLI Examples

Audit Script

Consolidates information

Parallel operations

Unused EC2/EBS etc

Feeds reporting

ELB/EC2 usage

CLI EXAMPLES

Others

Cost Measurement

Cloning (optimizes scale-up)

Move servers between load balancers

Spot History graphing

Spot History email alerts

AWS Security4

SECURITY

Do not give away the front door keys

Do not open all the windows

SECURITY OPTIONS

Keypairs

Security groups

Virtual Private Cloud (VPC)

Identity and Access Management (IAM)

Multi-factor authentication

Learn the different benefits

http://aws.amazon.com/mfa/

SECURITY TIPS

Restrict open access to port 80/443

Jump box

Restrict IP Access

Additional authentication

Per user SSH authentication

Do not use keypair

products

Many Others (AWS Summit 2013)

Cloudaware

Enstratius

AlertLogic

SafeNet

Instrumentation5

Instrumentation

What is important to you?

Instrumentation

All server stats

Instrumentation

All server stats

Sampling issues

Instrumentation

All server stats

Sampling issues

Deceiving averages (frequency)

REQUESTS PER SEC

5 second averages, not 1 minute samplehttps://github.com/ronaldbradford/reqstat

REQUESTS PER SEC

-1,500 RPS

outliers

outliersI care about these

TESTING

End to end testing critical

Network latency

ELB performance

products

AWS Cloudwatch

Datadog

Boundary

CopperEgg

AppDynamics

products

AWS Cloudwatch

Datadog

Boundary

CopperEgg

AppDynamics

What features matter?

Failure6

FAILURE

Instances fail

FAILURE

Instances fail

Outages occur

AWS scheduled reboots

FAILURE

Instances fail

Outages occur

AWS scheduled reboots

Be prepared

Chaos Monkey

http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html

CONCLUSION

Cost Management (saving money)

CONCLUSION

CLI automation

CONCLUSION

CLI automation

Instrumentation (inc business metrics)

CONCLUSION

CLI automation

Distribute your application & data

CONCLUSION

CLI automation

Distribute your application & data

Disaster is inevitable

AWS for FREE

http://aws.amazon.com/free/

Free EC2 t1.micro for a year

Free RDS t1.micro for a year

S3, DynamoDB, SimpleDB, +++

EffectiveMySQL.com - Performance, Scalability & Business Continuityhttp://effectiveMySQL.comRonald Bradford

Lessons Learned Managing Large AWS Environments

Technology

MQ Monitoring on AWS · MQ on AWS Options Quick Start Guide for AWS (MQV9) Partner World Licensing (non-prod environments) BYOL from your current license stack Purchase of MQ from

Managing your AWS Infrastructure at Scale · greater advantage of the benefits of dynamic cloud environments: ... Amazon Web Services – Managing Your AWS Infrastructure at Scale

Introducing Personal Learning Environments to Informal ... · Introducing Personal Learning Environments to Informal Learners: Lessons Learned from the OpenLearn Case Study Alexander

DOMAIN-SPECIFIC MODELING ENVIRONMENTS By Gregory … · DOMAIN-SPECIFIC MODELING ENVIRONMENTS By Gregory G. Nordstrom ... Analysis and Lessons Learned ... V. CASE STUDY

AWS re:Invent 2016: How I learned to embrace DevOps and Configure Infrastructure at Scale (WIN402)

Oracle VirtualBox Amazon AWS Your own computer …Exercise... · There are several environments available to do exercises for DS201: Oracle VirtualBox Amazon AWS Your own computer

What we learned from the AWS Outage

CORE+: TRIAL heritage , lessons learned and trial environments future plans

Aws cost optimization: lessons learned, strategies, tips and tools

AWS Chicago 2016 Lessons Learned Deploying the ELK Stack

UberCloud Containers on Kubernetesqnib.org/data/hpcw19/0_Intro_1_UberCloud.pdf · Ready to deploy in different Cloud Environments (Azure, AWS, GCP, Oracle Cloud) and on-premise environments

Lessons Learned while Exploring Cloud-Native Architectures ... · PDF fileLessons Learned while Exploring Cloud-Native Architectures for ... data on AWS/NGAP platform. OPeNDAP/HDF

AWS Well-Architected Framework - IT World Canada...Amazon Web Services – AWS Well-Architected Framework Page 2 Based on what we have learned we have created the AWS Well-Architected

XebiaLabs DevOps Platform on the AWS Cloud · in the AWS Cloud, or in private or hybrid cloud environments. XebiaLabs supports deployments to a variety of AWS products and services,

Setting Up Multiuser Environments in the AWS loud (for ... · Amazon Web Services – Setting Up Multiuser Environments October 2013 Page 1 of 21 Setting Up Multiuser Environments

Lessons Learned on Capgemini’s COMPLETE Managed Services Platform (DMG208) | AWS re:Invent 2013

AWS Lambda and Serverless framework: lessons learned while building a serverless company

Setting Up Multiuser Environments in the AWS Cloud (for Classroom

Development environments in the AWS cloud - Vancouver PHP Meetup

AWS Summit Berlin 2013 - EyeEm - A Scalable Cloud Architecture - Lessons Learned