KELK Stack on AWS

SteamhausKELK ON AWS

Who am I?Sean Clerkin Senior Site Reliability Engineer

Logging is difficult

No centralised loggingUser needs OS

knowledgeDistribution

Of keys

Enemy of autoscaling

Log rotation

Users download logs unnecessarily

Doesn’t scale To many servers

Slow to find issues

Alertingis hard

Sshing to servers :(

SteamhausKELK ON AWS

ELK is awesomE

SteamhausKELK ON AWS

ELK on ec2

SteamhausKELK ON AWS

KELK on AWS

• Low maintenance - No ec2, Uses entirely AWS serverless technologies and services

• ALB, Cloudfront and Cloudtrail logs are ingested as well as EC2 logs

• Logs are archived in S3 for long term storage, and indexed in Elasticsearch for short term analytics

• Automated with Terraform

• Open source

Kinesis: buffering and delivering instance logs Elasticsearch: Indexing and log storage Lambda: processing and delivering S3 logs Kibana: Search and analytics

SteamhausKELK ON AWS

How does it work?

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

SteamhausKELK ON AWS

Automationcode

Sample Web Stack

VPC

ALB

EC2

LoggingStack

Kinesis

ElasticsearchService

Lambda

S3

CloudfrontPython

Terraform

Do try this at home!github.com/steamhaus/kelk-example

SteamhausKELK ON AWS

Callouts from the build• It’s not production ready, built for readability • Nailing iam and bucket policies can take a while! • Testing lambda - create a test event in the UI • Use Terraform, rinse and repeat

SteamhausKELK ON AWS

Any Questions..?

Thank you :)Contact ushello@steamhaus.co.uk0161 820 2020@steamhausmcr

Locate us

Fourways House57 Hilton Street Manchester M1 2EJ