1. 1. 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Jon Handler, Principal Solutions Architect, AWS August 14, 2017 Deep Dive: Log Analytics with Amazon Elasticsearch Service
2. 2. What produces data? Metering Records Mobile Apps IoT Sensors Web Clickstream Enterprise Documents Application Logs [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/ht docs/test
3. 3. The problem with all of that data
4. 4. The solution
5. 5. Amazon Elasticsearch Service is a cost-effective managed service that makes it easy to deploy, manage, and scale open-source Elasticsearch for log analytics, full-text search, and more. Amazon Elasticsearch Service
6. 6. Data source / Kinesis Firehose Agent Amazon Kinesis Firehose Amazon Elasticsearch Service Kibana Log analytics architecture
7. 7. Easy to Use Deploy a production-ready Elasticsearch cluster in minutes Simplifies time-consuming management tasks such as software patching, failure recovery, backups, and monitoring Open Get direct access to the Elasticsearch open-source API Fully compatible with the open-source Elasticsearch API, for all code and applications Secure Secure Elasticsearch clusters with AWS Identity and Access Management (IAM) policies with fine-grained access control access for users and endpoints Automatically applies security patches without disruption, keeping Elasticsearch environments secure Available Provides high availability using Zone Awareness, which replicates data between two Availability Zones Monitors the health of clusters and automatically replaces failed nodes, without service disruption AWS Integrated Integrates with Amazon Kinesis Firehose, AWS IoT, and Amazon CloudWatch Logs for seamless data ingestion AWS CloudTrail for auditing, AWS Identity and Access Management (IAM) for security, and AWS CloudFormation for cloud orchestration Scalable Scale clusters from a single node up to 20 nodes Configure clusters to meet performance requirements by selecting from a range of instance types and storage options, including SSD-powered EBS volumes Amazon Elasticsearch Service Benefits
8. 8. Amazon Elasticsearch Service leading use cases Log Analytics & Operational Monitoring Monitor the performance of applications, web servers, and hardware Easy to use, powerful data visualization tools to detect issues quickly Dig into logs in an intuitive, fine-grained way Kibana provides fast, easy visualization Search Application or website provides search capabilities over diverse documents Tasked with making this knowledge base searchable and accessible Text matching, faceting, filtering, fuzzy search, autocomplete, highlighting, and other search features Query API to support application search
9. 9. Leading enterprises trust Amazon Elasticsearch Service for their search and analytics applications Media & Entertainment Online Services Technology Other
10. 10. Adobe Developer Platform (Adobe I/O) P R O B L E M Cost-effective monitor for XL amount of log data Over 200,000 API calls per second at peak - destinations, response times, bandwidth Integrate seamlessly with other components of AWS ecosystem S O L U T I O N Log data is routed with Amazon Kinesis to Amazon Elasticsearch Service, then displayed using AES Kibana Adobe team can easily see traffic patterns and error rates, quickly identifying anomalies and potential challenges B E N E F I T S Management and operational simplicity Flexibility to try out different cluster config during dev and test Amazon Kinesis Streams Spark Streaming Amazon Elasticsearch Service Data Sources 1
11. 11. McGraw-Hill Education P R O B L E M Supporting a wide catalog across multiple services in multiple jurisdictions Over 100 million learning events each month Tests, quizzes, learning modules begun / completed / abandoned S O L U T I O N Search and analyze test results, student/teacher interaction, teacher effectiveness, student progress Analytics of applications and infrastructure are now integrated to understand operations in real time B E N E F I T S Confidence to scale throughout the school year. From 0 to 32 TB in 9 months Focus on their business, not their infrastructure
12. 12. Get set up right
13. 13. Easy to use and scalable AWS SDK AWS CLI AWS CloudFormation Elasticsearch data nodes Elasticsearch master nodes Elastic Load Balancing AWS IAM CloudWatchCloudTrail Amazon Elasticsearch Service domain
14. 14. Data pattern Amazon ES cluster logs_01.21.2017 logs_01.22.2017 logs_01.23.2017 logs_01.24.2017 logs_01.25.2017 logs_01.26.2017 logs_01.27.2017 Shard 1 Shard 2 Shard 3 host ident auth timestamp etc. Each index has multiple shards Each shard contains a set of documents Each document contains a set of fields and values One index per day
15. 15. Deployment of indices to a cluster Index 1 Shard 1 Shard 2 Shard 3 Index 2 Shard 1 Shard 2 Shard 3 Amazon ES cluster 1 2 3 1 2 3 1 2 3 1 2 3 Primary Replica 1 3 3 1 Instance 1, Master 2 1 1 2 Instance 2 3 2 2 3 Instance 3
16. 16. How many instances? The index size will be about the same as the corpus of source documents Double this if you are deploying an index replica Size based on storage requirements Either local storage or up to 1.5 TB of Amazon Elastic Block Store (EBS) per instance Example: 2 TB corpus will need 4 instances Assuming a replica and using EBS Given 1.5 TB of storage per instance, this gives 6TB of storage
17. 17. How many shards? Best practice: shards should be < 50GB Divide index size by ~40GB to get initial shard count Adjust for workload: more shards for more write capacity, fewer shards for better read latency Target ~3 active shards per instance Always use at least 1 replica for production! Example: 2 TB corpus will need 50 shards 2,000 GB / 40GB per shard = 50 shards
18. 18. Instance type recommendations Instance Workload T2 Entry point. Dev and test. OK for dedicated masters. M3, M4 Equal read and write volumes. R3, R4 Read-heavy or workloads with high memory demands (e.g., aggregations). C4 High concurrency/indexing workloads I2 Up to 1.6 TB of SSD instance storage.
19. 19. Cluster with no dedicated masters Amazon ES cluster 1 3 3 1 Instance 1, Master 2 1 1 2 Instance 2 3 2 2 3 Instance 3
20. 20. Cluster with dedicated masters Amazon ES cluster 1 3 3 1 Instance 1 2 1 1 2 Instance 2 3 2 2 3 Instance 3Dedicated master nodes Data nodes: queries and updates
21. 21. Master node recommendations Number of data nodes Master node instance type < 10 m3.medium+ < 20 m4.large+ = 3
22. 22. Cluster with zone awareness Amazon ES cluster 1 3 Instance 1 2 1 2 Instance 2 3 2 1 Instance 3 Availability Zone 1 Availability Zone 2 2 1 Instance 4 3 3
23. 23. Small use cases Logstash co-located on the Application instance SigV4 signing via provided output plugin Up to 200 GB of data m3.medium + 100G EBS data nodes 3x m3.medium master nodes Application Instance
24. 24. Large use cases Amazon DynamoDB AWS Lambda Amazon S3 bucket Amazon CloudWatch Data flows from instances and applications via Lambda; CWL is implicit SigV4 signing via Lambda/roles Up to 5 TB of data r3.2xlarge + 512 GB EBS data nodes 3x m3.medium master nodes
25. 25. XL use cases Amazon Kinesis Ingest supported through high-volume technologies like Spark or Kinesis Up to 60 TB of data today R3.8xlarge + 640GB data nodes 3x m3.xlarge master nodes Amazon EMR
26. 26. Best practices Data nodes = Storage needed/Storage per node Use GP2 EBS volumes Use 3 dedicated master nodes for production deployments Enable Zone Awareness Set indices.fielddata.cache.size = 40
27. 27. Kinesis Firehose
28. 28. Kinesis Firehose overview Delivery Stream: Underlying AWS resource Destination: Amazon ES, Amazon Redshift, or Amazon S3 Record: Put records in streams to deliver to destinations
29. 29. Kinesis Firehose delivery architecture with transformations S3 bucket source records data source source records Amazon Elasticsearch Service Firehose delivery stream transformed records delivery failure Data transformation function transformation failure
30. 30. Lambda blueprints for common use cases
31. 31. Transform this 199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245 To this { "verb": "GET", "ident": "-", "bytes": 6245, "@timestamp": "1995-07-01T00:00:01", "request": "GET /history/apollo/ HTTP/1.0", "host": "199.72.81.55", "authuser": "-", "@timestamp_utc": "1995-07-01T04:00:01+00:00", "timezone": "-0400", "response": 200 }
32. 32. Kinesis Firehose features for ingest Serverless scale Error handling S3 Backup
33. 33. Best practices Use smaller buffer sizes to increase throughput, but be careful of concurrency Use index rotation based on sizing Default: stream limits: 2,000 transactions/second, 5,000 records/second, and 5 MB/second
34. 34. Log analysis with aggregations
35. 35. host:199.72.81.55 with 1, 4, 8, 12, 30, 42, 58, 100 ... Look up 199.72.81.55 Field data GET GET POST GET PUT GET GET POST Buckets GET POST PUT 5 2 1 Counts
36. 36. Amazon ES aggregations Buckets a collection of documents meeting some criterion Metrics calculations on the content of buckets Bucket: time Metric:count
37. 37. A more complicated aggregation Bucket: ARN Bucket: Region Bucket: eventName Metric: Count
38. 38. Demo
39. 39. Best practices Elasticsearch provides statistical evaluations based on field data gathered from matching documents Visualizations are based on buckets/metrics Use a histogram on the x-axis first, then sub-aggregate
40. 40. Run Elasticsearch in the AWS cloud with Amazon Elasticsearch Service Use Kinesis Firehose to ingest data simply Kibana for monitoring, Elasticsearch queries for deeper analysisAmazon Elasticsearch Service
41. 41. What to do next Centralized logging solution: https://aws.amazon.com/answers/logging/centralized-logging/ Our overview page on AWS: https://aws.amazon.com/elasticsearch-service/ Questions? Contact me at handler@amazon.com
42. 42. Thank you!