╔══════════════════════════════════════════╗ ║ OPEN SOURCE TOOLING FOR ECS ║ ║ ║ ║ Noah Zoschke ║ ║ noah@convox.com ║ ║ @nzoschke ║ ║ ║ ║ 12/8/2015 ║ ╚══════════════════════════════════════════╝

CONVOX

EC2 CONTAINER SERVICE(BATTERIES NOT INCLUDED)

API

• Clusters

• TaskDefinitions

• Tasks and Services

Bring Your Own

• Instances

• ecs-agent

• Load Balancers

• Logging

• Builds / Images

• Tools...

CONVOX OPEN SOURCE PAAS ⟷ IAAS

Racks ⟷ ASG, CF, Dynamo, EC2, ECS, IAM, VPC

Apps ⟷ CF, ECS, ELB, Kinesis

Scale ⟷ ASG, CF, ECS

Environments ⟷ KMS, S3

Builds ⟷ S3

Logs ⟷ Kinesis, Lambda

Metrics ⟷ CloudWatch, Kinesis

Notifications ⟷ SNS

CONVOX REST API$ convox api get /apps [ { "balancer": "myapp-1749418666.us-east-1.elb.amazonaws.com", "name": "myapp", "release": "REXIQURVKXE", "status": "running" }, { "balancer": "myapp2-689551992.us-east-1.elb.amazonaws.com", "name": "myapp2", "release": "RNEFHNIUSKF", "status": "running" }, { "balancer": "myapp3-435098803.us-east-1.elb.amazonaws.com", "name": "myapp3", "release": "RORJKBNVKDD", "status": "running" } ]

CONVOX REST API$ convox api get /apps/myapp/processes [ { "app": "myapp", "command": "bin/web", "cpu": 0.0329, "host": "10.0.3.135", "id": "13254981d20", "image": "registry.internal:5000/myapp-web:BHLRYHSMXNM", "memory": 0.2068, "name": "web", "ports": [ "80:3000", "443:3001" ], "release": "REXIQURVKXE", "started": "2015-01-01T00:00:00Z" } ]

BEHIND THE SCENESaws && docker || convox

RACK INSTALL CLOUDFORMATION PARAMETERIZED INFRASTRUCTURE→ Ami ami-c5fa5aae → InstanceCount 3 → InstanceType t2.small → Password PuDpyqGTmxBN8ziGJ9UiMfrfGZfHDG → Tenancy default → Version 20151204013151 → VolumeSize 30 → VPCCIDR 10.0.0.0/16

↑ Balancer convox AWS::ElasticLoadBalancing::LoadBalancer ↑ Cluster convox-Cluster-1JI343QBLSMYJ AWS::ECS::Cluster ↑ DynamoBuilds convox-builds AWS::DynamoDB::Table ↑ DynamoReleases convox-releases AWS::DynamoDB::Table ↑ EncryptionKey arn:aws:kms:...:key/d40c0153... Custom::KMSKey ↑ IamRole convox-IamRole-M1YZSNXNS1F7 AWS::IAM::Role ↑ Instances convox-Instances-PCWRQ6OWDWTT AWS::AutoScaling::AutoScalingGroup ↑ Kinesis convox-Kinesis-C09RDWFR8NOE AWS::Kinesis::Stream ↑ NotificationTopic arn:aws:sns:...:convox-notifications AWS::SNS::Topic ↑ Settings convox-settings-13c91daqrj90z AWS::S3::Bucket ↑ Vpc vpc-b27ff8d6 AWS::EC2::VPC

← Dashboard convox-820546104.us-east-1.elb.amazonaws.com ← Kinesis convox-Kinesis-C09RDWFR8NOE

CLOUDFORMATION CUSTOM RESOURCES

┌─────────────────────────────────────┐ │POST arn:aws:lambda:... │ │{ │ │ ResourceProperties: { │ │ Description: "Master Encryption",│ ┌─────────────────────────────────────┐ ┌───────────────────────────┐ │ KeyUsage: "ENCRYPT_DECRYPT" │ │aws kms create-key \│ │200: OK │ │ } │ │ --description "Master Encryption" \│ │400: LimitExceededException│ │} │ │ --key-usage ENCRYPT_DECRYPT │ │500: KMSInternalException │ └─────────────────────────────────────┘ └─────────────────────────────────────┘ └───────────────────────────┘ ┌────────────────┐ ┌──────────────┐──────────────────────▶┌───────────┐ │ CloudFormation │──────────────────────▶│ Lambda │ │AWS KMS API│ └────────────────┘ CREATE_IN_PROGRESS └──────────────┘◀──────────────────────└───────────┘ ▲ │ │ │ │ CREATE_COMPLETE │ │ OR ▼ │ CREATE_FAILED ┌─────────────┐ └────────────────────────────────│ S3 │ └─────────────┘

APP CREATE CLOUDFORMATION PARAMETERIZED CONTAINERS

→ Cluster convox-Cluster-1JI343QBLSMYJ → Cpu 200 → Environment https://httpd-settings-1e3ej4u01z4bv.s3.amazonaws.com/releases/RSAQCOYHGPV/env → Key arn:aws:kms:us-east-1:901416387788:key/d40c0153-4a57-4d50-9ca0-99a974daca11 → Release RSAQCOYHGPV → VPC vpc-b27ff8d6 → WebCommand → WebDesiredCount 1 → WebImage convox-820546104.us-east-1.elb.amazonaws.com:5000/httpd-web:BQIWNCMIYZG → WebMemory 256 → WebPort80Balancer 80 → WebPort80Certificate → WebPort80Host 42563 → WebPort80Secure No

↑ Balancer httpd AWS::ElasticLoadBalancing::LoadBalancer ↑ Kinesis httpd-Kinesis-FO32SUUFLX24 AWS::Kinesis::Stream ↑ LogsAccess AKIAIFI65IDSEURPK62Q AWS::IAM::AccessKey ↑ LogsUser httpd-LogsUser-96BAE2EL9TNL AWS::IAM::User ↑ ServiceRole httpd-ServiceRole-19LN8R18BIVRW AWS::IAM::Role ↑ Settings httpd-settings-1e3ej4u01z4bv AWS::S3::Bucket ↑ WebECSService arn:aws:ecs:...:service/httpd-web-SATOEEBOQNF Custom::ECSService ↑ WebECSTaskDefinition arn:aws:ecs:...:task-definition/httpd-web:6 Custom::ECSTaskDefinition

← BalancerWebHost httpd-908645489.us-east-1.elb.amazonaws.com ← Kinesis httpd-Kinesis-FO32SUUFLX24 ← Settings httpd-settings-1e3ej4u01z4bv ← WebPort80Balancer 80

APP DEPLOY DOCKER APPLICATION MANIFEST ⟷ AWS

┌──────────────────────────────────────────────────────────────────────────────────────────────────┐ │web: Task Definition httpd-web:6 │ │ command: bin/web Service httpd-web-SATOEEBOQNF │ │ build: . Docker Image httpd-web:BQIWNCMIYZG │ │ ports: │ │ - 80:80 ELB 80 : 52452 : 80 │ │ - 443:80 ELB (SSL) 443 : 52452 : 80 │ │ │ │worker: Task Definition httpd-worker:6 │ │ command: bin/worker Service httpd-worker-SHAOPEQONEF │ │ build: . Docker Image httpd-worker:BQIWNCMIYZG │ │ links: │ │ - redis REDIS_URL=rer45wxl0uj8jn6.1qae5u.ng.0001.usw2.cache.amazonaws.com:6379│ │ - rabbit RABBIT_URL=httpd-1222973998.us-west-2.elb.amazonaws.com:5672 │ │ │ │rabbit: Task Definition httpd-rabbit:6 │ │ command: rabbitmq-server Service httpd-rabbit-SPNFHGMWNUU │ │ image: rabbitmq Docker Image httpd-rabbit:BQUWNCMIYZG │ │ ports: │ │ - 5672 ELB (Internal) 5672 : 24324 : 5672 │ │ │ │redis: │ │ image: convox/redis AWS::ElastiCache::CacheCluster │ └──────────────────────────────────────────────────────────────────────────────────────────────────┘

APP BUILD PRIVATE BUILD AND REGISTRY SERVICE

┌─────────────────────────────────────────┐ │ convox API │ │ POST /apps/httpd/build │ ┌──────────────────┐ │ │ │ │ │ ┌─────────────────────────────┐ │ ┌──────────────────┐ │ │ │ │ pull httpd-web:BLASTBUILD │◀───┼───────┤ Convox Registry │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ ├─────┼─────▶│ pull rabbitmq │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │tag httpd-rabbit:BQUWNCMIYZG │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ ┌────────────────┐ │ DockerHub │ │ ┌─────────────────────────────┐ │ │ ├───────────▶│ S3 │ │ and │ │ │ push $REGISTRY_HOST ├────┼──────▶│ │◀───────────┤ RegistryBucket │ │Private Registries│ │ └─────────────────────────────┘ │ │ │ └────────────────┘ │ │ │ ┌─────────────────────────────┐ │ │ │ │ ├─────┼─────▶│ pull debian │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │ build Dockerfile │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │ tag httpd-web:BQUWNCMIYZG │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ └──────────────────┘ │ │ push $REGISTRY_HOST ├────┼──────▶└──────────────────┘ │ └─────────────────────────────┘ │ │ ┌─────────────────────────────┐ │ │ │tag httpd-worker:BQUWNCMIYZG │ │ │ └─────────────────────────────┘ │ │ ┌─────────────────────────────┐ │ │ │ ... │ │ │ └─────────────────────────────┘ │ │ │ │ │ └─────────────────────────────────────────┘

…UNTIL EC2 CONTAINER REGISTRY IS GA

APP LOGS AGENT, DOCKER APIS, KINESIS, LAMBDA

┌──────────────────────────────────────────────────────────┐ ┌──────────────────┐ │ EC2 Instance in ECS Cluster │ │ app1 Kinesis │ │ │ │ ┌────────┐ │ ┌───────────────────────────────────────────┐ │ ┌──────────────┐ ┌──────────────────────────────────┐ │ ┌─┼───▶│shard 1 │ │──┐ │ Lambda w/ EventSourceMapping │ │ │ │ │ │ │ │ │ └────────┘ │ │ │ ┌──────────────────────────────────────┐ │ │ │ │ │ │ │ │ └──────────────────┘ │ │ │function(event, context) { │ │ │ │ app1 │ │ app2 │ │ │ │ │ │ event.records.forEach(function(r) { │ │ │ │ web.1 │ │ worker.1 │ │ │ │ │ │ winston.info(r.kinesis.data) │ │ │ │ │ │ │ │ │ └─┼▶│ }) │──┼────────▶┌───────────────┐ │ │ │ │ │ │ │ │ │ context.done() │ │ │ │ │ └──────────────┘ └──────────────────────────────────┘ │ │ ┌──────────────────┐ │ │} │ │ │ │ │ │ │ │ │ │ app2 Kinesis │ │ │ │ │ │ │ │ │ ┌─────────────────────┘ │ │ │ ┌────────┐ │ │ └──────────────────────────────────────┘ │ │ Syslog Server │ │ ▼ ▼ │ │ │ ┌─▶│shard 1 │ │ │ ┌────────────────────────────────┐ │ │ │ │ ┌────────────┐ ┌────────────┐─────────────┼───┘ │ │ └────────┘ │ │ │function(event, context) { ... }│──┼────────▶│ │ │ │ dockerd │◀─────────────│convox/agent│─────────────┼─────┼─┘ ┌────────┐ │ │ └────────────────────────────────┘ │ │ │ │ └────────────┘ └────────────┘─────────────┼─────┼───▶│shard 2 │ │ │ ┌────────────────────────────────┐ │ │ │ │ ▲ ┌────────────────────────────────────┐ │ │ └────────┘ │────┼─▶│function(event, context) { ... }│───────┼────────▶└───────────────┘ │ │ │GET docker /events (create) │ │ │ . │ │ └────────────────────────────────┘ │ │ ▼ │ GET ENV "Kinesis", "Process"│ │ │ . │ │ │ │ ┌────────────┐ │ GET Docker /logs?follow=1 │ │ │ . │ └───────────────────────────────────────────┘ │ │ ecs-agent │ │ PUT Kinesis /records │ │ │ ┌────────┐ │ │ └────────────┘ └────────────────────────────────────┘ │ │ │shard N │ │ │ │ │ └────────┘ │ └──────────────────────────────────────────────────────────┘ └──────────────────┘

DEBUGGING RUN, EXEC, SSH OVER WEB SOCKETS

$ convox run web bash root@3e4160f0c4d0:/app#

$ convox ps ID NAME RELEASE CPU MEM STARTED COMMAND 551967b75abd web RHQZEJZFCSD 0.39% 21.04% 2 hours ago rails server -b 0.0.0.0 f5ec95c38f58 worker RHQZEJZFCSD 0.00% 30.35% 2 hours ago sidekiq

$ convox exec 551967b75abd bash root@281d0a9c33a:/app#

$ convox exec 551967b75abd ps ax PID USER TIME COMMAND 1 root 0:00 sh -c bin/web 6 root 0:00 {web} /bin/sh bin/web 9 root 0:00 unicorn master -c unicorn.rb 11 root 0:00 unicorn worker[0] -c unicorn.rb

SCALING ONE APP ⟶ MANY SERVICES

Service Name Task Definition Desired Running ═══════════════════════════════════════════════════════════════════════════ myapp-clock-SVQQEUPGZPS myapp-clock:106 1 1 myapp-scheduler-SSMOCJRAGOM myapp-scheduler:183 1 1 myapp-web-SLHARAVBAWZ myapp-web:119 2 2 myapp-runner-SEGBMHLWREH myapp-runner:163 4 4

CAPACITY PLANNING FITTING MANY APPS IN ONE CLUSTER

| | | | | | | | | | | | | | | | | +---+ | | | . | | | | . | | | | . | | | | . | | | | . | | | | . | | | | . | | | +---+ | | | | | | | | | | | | | | | | | | | | ,---, +---+ | | | . | | . | | +---' . +---, ,---' . +---+---+--+ | . . . | . | | . . . | . . . | | + . +---+ . '---, | . ,---+ . . . | | | . | . | . . . | | . | | . . . | | +---+ . +---+ . | +---+---+---+---+ | | . . . | . | . | | . . . | . . . | | + . +---+ . +---+ | . . . + . . . | | | . | . . . . . | | . . . | . . . | | +---+---+---+---+---+---+---+---+---+--+

• rolling app deploys need slack capacity

• process type with load balancer needs entire instance

• scheduler can fail to place tasks behind the scenes...

noah@convox.com @nzoschke

Discuss these techniques and get involvedGitHub https://github.com/convox Slack http://invite.convox.com/

_ _ _ _ | |_| |__ __ _ _ __ | | _____| | | __| '_ \ / _` | '_ \| |/ / __| | | |_| | | | (_| | | | | <\__ \_| \__|_| |_|\__,_|_| |_|_|\_\___(_)