Monitoring Containers at Scale - September Webinar Series

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Brandon Chavis, AWS Solutions ArchitectIlan Rabinovitch, Director of Technical Community, Datadog

20th September 2016

Monitoring Containers at Scale

Agenda

Introduction to ECS

AWS Logging options: Cloudwatch Logs, Cloudwatch, Cloudtrail

Logging containers with Datadog

Amazon EC2 Container Service (ECS)

Container Managementat Any Scale

Flexible ContainerPlacement

Integrationwith the AWS Platform

Components of Amazon ECS

TaskOne or more containers running together on an Instance

Task DefinitionDefinition of containers and environment configuration

ClusterFleet of EC2 instances on which tasks run

Cluster ManagerManage cluster resource and state of tasks

SchedulerPlaces tasks onto cluster

AgentCoordinate EC2 instances and Manager

Cluster, Scheduler, Task Scheduler

ManagerCluster

Task Definition

Task

Agent

Monitoring & Logging

AWS logging tools:

CloudwatchCloudwatch LogsCloudtrail

CloudWatch Logs with awslogs driver

Amazon CloudWatch Logs

Amazon CloudWatch Logs

Amazon CloudWatch Logs

Amazon CloudWatch Logs

Amazon S3

Amazon Kinesis

AWS Lambda

Amazon Elasticsearch Service

Amazon ECS Store

Stream

Process

Search

CloudWatch Logs driver

Configuring Logging in Task Definition

logConfiguration task definition parameter

Requires version 1.18 or greater of the Docker Remote API

Maps to docker run --log-driver option

Log drivers: json-file, syslog, journald, gelf, fluentd, awslogs

Configuring Logging in Task Definition

"containerDefinitions": [ { "memory": 300, "portMappings": [ {

"hostPort": 80, "containerPort": 80 } ],

"entryPoint": [ "sh", "-c" ], "logConfiguration": {

"logDriver": "awslogs", "options": {

"awslogs-group": "awslogs-test", "awslogs-region": "us-west-2", "awslogs-stream-prefix": "nginx" }

}, "name": "simple-app", "image": "httpd:2.4", "command": [ "/bin/sh -c \"echo 'Congratulations! Your application is now running on a container in Amazon ECS.'

> /usr/local/apache2/htdocs/index.html && httpd-foreground\"" ], "cpu": 10 } ], "family": "cw-logs-example"

}

Monitoring with Amazon CloudWatch

Metric data sent to CloudWatch in 1-minute periods and recorded for a period of two weeks

Available metrics: CPUReservation, MemoryReservation, CPUUtilization, MemoryUtilization

Available dimensions: ClusterName, ServiceName

Monitoring with Amazon CloudWatch

Monitoring with Amazon CloudWatch

Monitoring with Amazon CloudWatch

Use the Amazon CloudWatch Monitoring Scripts to monitor additional metrics, e.g. disk space:

# Edit crontab> crontab -e

# Add command to report disk space utilization to CloudWatch every five minutes*/5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used --disk-space-avail --disk-path=/ --from-cron

Logging Amazon ECS API with AWS CloudTrail

{ "eventVersion": "1.03", "userIdentity": {…}, "eventTime": "2015-10-12T13:57:33Z", "eventSource": "ecs.amazonaws.com", "eventName": "CreateCluster", "awsRegion": "eu-west-1", "sourceIPAddress": "54.240.197.227", "userAgent": "console.amazonaws.com", "requestParameters": { "clusterName": "ecs-cli" },

Logging Amazon ECS API with AWS CloudTrail

"responseElements": { "cluster": { "clusterArn": "arn:aws:ecs:eu-west-1:560846014933:cluster/ecs-cli", "pendingTasksCount": 0, "registeredContainerInstancesCount": 0, "status": "ACTIVE", "runningTasksCount": 0, "clusterName": "ecs-cli", "activeServicesCount": 0 } }, […]

Monitoring Amazon ECS with Datadog

• SaaS based infrastructure and application monitoring• Focus on modern environments

• Cloud, Containers, Micro Services• Processing nearly a trillion data points per day• Intelligent Alerting and Insightful Dashboards

Datadog Overview

Operating Systems, Cloud Providers (AWS), Containers, Web Servers, Datastores, Caches, Queues and more...

Monitor Everything

CloudWatch and ECS

ResourcesCPUReservationMemoryReservationCPUUtilizationMemoryUtilization

How do we get at the upper layers?

Pseudo-files

• Provide visibility into container metrics via the file system. • Generally under: /cgroup/<resource>/docker/$CONTAINER_ID/ or/sys/fs/cgroup/<resource>/docker/$CONTAINER_ID/

Pseudo-files: CPU Metrics$ cat /sys/fs/cgroup/cpuacct/docker/$CONTAINER_ID/cpuacct.stat> user 2451 # time spent running processes since boot> system 966 # time spent executing system calls since boot

$ cat /sys/fs/cgroup/cpu/docker/$CONTAINER_ID/cpu.stat> nr_periods 565 # Number of enforcement intervals that have elapsed

> nr_throttled 559 # Number of times the group has been throttled

> throttled_time 12119585961 # Total time that members of the group were throttled (12.12 seconds)

Pseudo-files: CPU Throttling

Docker API• Detailed streaming metrics as JSON HTTP socket

$ curl -v --unix-socket /var/run/docker.sock http://localhost/containers/28d7a95f468e/stats

STATS Command

# Usage: docker stats CONTAINER [CONTAINER...]$ docker stats $CONTAINER_ID CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O BLOCK I/Oecb37227ac84 0.12% 71.53 MiB/490 MiB 14.60% 900.2 MB/275.5 MB 266.8 MB/872.7 MB

Side Car Containers

Agents and Daemons

• Ideally we’d want to schedule an agent or daemon on each node via ECS Tasks.

• Current Solutions:1. Bake it into your image.2. Install on each host at provision time.3. Automate with User Scripts and Launch Configs

Grant Privileges via IAM$ aws iam create-role \ --role-name ecs-monitoring \ --assume-role-policy-document file://trust.policy

$ aws iam put-role-policy --role-name ecs-monitoring --policy-name ecs-monitoring-policy --policy-document file://ecs.policy

$ aws iam create-instance-profile --instance-profile-name ECSNode

$ aws iam add-role-to-instance-profile \ --instance-profile-name ECSNode \ --role-name ecs-monitoring

Create A User Script

Auto-Scale!

$ aws autoscaling create-launch-configuration --launch-configuration MyECSCluster --key-name my-key --image-id AMI_ID --instance-type INSTANCE_TYPE --user-data file://launch-script.txt --iam-instance-profile IAM_ROLE

Full Stack Monitoring

Docker API ECS & CloudWatch

Monitoring AgentContainer

Containers List &Metadata

Additional Metadata(Tags, events, etc)

Host Level Metrics

Monitoring Amazon ECS with Datadog

Aren’t we still missing a layer?

Operating Systems, Cloud Providers (AWS), Containers, Web Servers, Datastores, Caches, Queues and more...

Monitor Everything

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Service Discovery

Service Discovery

Docker API ECS & CloudWatch

Monitoring AgentContainer

A O A O

Containers List &Metadata

Additional Metadata(Tags, etc)

Config Backend

Integration ConfigurationsHost Level Metrics

Custom Metrics

• Instrument custom applications

• You know your key transactions best.

• Use async protocols like STATSD

Demo

Monday, October 24, 2016 JW Marriot Austin

https://aws.amazon.com/events/devday-austin

Free, one-day developer event featuring tracks, labs, and workshops around Serverless,

Containers, IoT, and Mobile

Q&A If you want to learn more, register for our upcoming DevDay Austin: