Real life aws_user_data_auto_scale

Real Life AWS: user-data & AutoScale

Adam NeumannSolutions Architect at IndexMedia

an@indexmedia.com

&

Part 1: user-data

1.What is user-data?2.A problem3.user-data in the solution4.Further work

What is user-data?

• user specified instance attribute• accessible via HTTP and curl/wget• usage wrapped by cloud-init• 'runs' the user data once on boot• great for configuring instances

The concept

• any script, or init-cloud format• #! /bin/bash etc etc• cmd line arg or via UI:

A Problem

• too hard to spin up new instances• complex, undocumented, manual• fab, bash, and misc scripts• slow: 30 min per instance• first: consolidate, then: user-data

IndexMedia example 1

• small user-data script, bootstrap• load big script from S3• setup apache, hadoop, virtualenv,...• 16 mins, vanilla ubuntu to serving• automated: yes, optimal: no• could be improved...

IndexMedia example 2

• split script into 2 scripts :)• 1: heavy, general, static = 15 min• runs on vanilla ubuntu, creates AMI• 2: light, instance specific = 90 secs• runs on AMI• from 30 min manual to 90 sec auto

Further work

• investigate cloud-init format• include Puppet/Chef (first, learn

Puppet/Chef...)• integrate into CI process

(environment becomes versioned, deployable, etc)

user-data summary

• great way to configure instances• UI and cmd line accessible• enables automated+repeatable

deployment• reduced deployment time, 30 mins to

90 secs

Part 2: AutoScale

1.What is AutoScale2.Our (common) problem3.Show me the examples!4.Learnings

Part 2: AutoScale

• cmd line only = no UI = hidden = unknown = scary

• basically:1.launching & terminating instances–within some bounds–based upon metrics• a second order system :)

AutoScale concepts

1. Launch Configuration

individual: instance type, AMI, user-data, etc

2. AutoScale Group

collective: ideal/max/min, AZs

3. Policy

how: a reaction, increase, decrease

4. Alerts

when: an event, high/low load, CW metrics

Our (common) problem

• maintain consistent experience for users, ie. Pip load time

• turn on servers when load is high• bootstrapped, must turn off servers

when load is low!• enter AutoScale...

Solution = AutoScale

• servers start/stop based upon current system state

• spending on IT matches user demand exactly!

• 4 commands to set it up...

1/4: Launch Configuration

as-create-launch-config gorilla_config_7d2 \--image-id ami-XXXXXXX \        --instance-type t1.micro \--group "gorilla-live"         --key indexmedia-live \--user-data-file /path/to/bootstrap.sh

2/4: Group

as-create-auto-scaling-group as_group \--launch-configuration gorilla_config_7d2 \--availability-zones us-west-1a us-west-1b \--min-size 10 --max-size 20 \--load-balancers my_balancer \--health-check-type EC2 \        --grace-period 120

3/4: Policy

as-put-scaling-policy gorilla_scaleup_policy \--auto-scaling-group as_group \        --type ChangeInCapacity \--adjustment=1

4/4: Alarm

mon-put-metric-alarm "gorilla high load" \    --dimensions "AutoScalingGroupName=as_group" \    --namespace "AWS/EC2" \    --statistic Average --metric-name CPUUtilization \    --comparison-operator GreaterThanThreshold \    --threshold 80 \    --evaluation-periods 5 --period 60 \    --alarm-actions ${POLICY-ARN}

... and done! Learnings

• 4 steps (well 6...) = elastic system• feedback = second order system• avoid bad harmonics!• shorter instance boot time == quicker

system response time• validate alarms & behaviour w/ load

tests on AWS

AutoScale summary

• cmd line only; just jump in!• 4 distinct concepts, one per cmd• quick boot = shorter grace period =

quicker scaling up (enter user-data...)• key to our multi-AZ, HA elastic

cluster, problem solved!

Thanks + Questions?

Adam NeumannSolutions Architect at IndexMedia

an@indexmedia.com