Amazon Elasticsearch Service - Developer Guide

Amazon Elasticsearch ServiceDeveloper Guide

API Version 2015-01-01

Amazon Elasticsearch Service Developer Guide

Amazon Elasticsearch Service: Developer GuideCopyright © 2018 Amazon Web Services, Inc. and/or its affiliates. All rights reserved.

Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any mannerthat is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks notowned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored byAmazon.


Table of ContentsWhat Is Amazon Elasticsearch Service? .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Features of Amazon Elasticsearch Service .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2Supported Elasticsearch Versions .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2Getting Started with Amazon Elasticsearch Service .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Signing Up for AWS .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Accessing Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Amazon ES Regions and Endpoints ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Choosing Instance Types .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Scaling in Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Using Amazon EBS Volumes for Storage .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Signing Requests ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Related Services .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Pricing for Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Getting Started with Amazon ES Domains .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Step 1: Creating an Amazon ES Domain .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Step 2: Uploading Data for Indexing .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10Step 3: Searching Documents in an Amazon ES Domain .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Step 4: Deleting an Amazon ES Domain .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Creating and Configuring Amazon ES Domains .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Creating Amazon ES Domains .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Creating Amazon ES Domains (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Creating Amazon ES Domains (AWS CLI) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Creating Amazon ES Domains (AWS SDKs) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Configuring Amazon ES Domains .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Configuring Amazon ES Domains (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Configuring Amazon ES Domains (AWS CLI) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20Configuring Amazon ES Domains (AWS SDKs) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Configuring EBS-based Storage .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Configuring EBS-based Storage (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Configuring EBS-based Storage (AWS CLI) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Configuring EBS-based Storage (AWS SDKs) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Modifying VPC Access Configuration .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Configuring VPC Access (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Configuring Access Policies ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Configuring Access Policies (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Configuring Access Policies (AWS CLI) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Configuring Access Policies (AWS SDKs) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Configuring Automatic Snapshots .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Configuring Snapshots (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Configuring Snapshots (AWS CLI) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Configuring Snapshots (AWS SDKs) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Configuring Advanced Options .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Configuring Advanced Options (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Configuring Advanced Options (AWS CLI) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Configuring Advanced Options (AWS SDKs) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Configuring Slow Logs .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Enabling Slow Logs Publishing (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Enabling Slow Logs Publishing (AWS CLI) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Enabling Slow Logs Publishing (AWS SDKs) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Setting Elasticsearch Logging Thresholds .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Viewing Slow Logs .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Access Control ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Types of Policies ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Resource-based Policies ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

API Version 2015-01-01iii


Identity-based policies ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34IP-based Policies ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Signing Amazon ES Requests ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35When Policies Collide .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Policy Element Reference .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Advanced Options and API Considerations .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Configuring Access Policies ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Additional Sample Policies ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Managing Amazon ES Domains .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44About Configuration Changes .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44Charges for Configuration Changes .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Enabling Zone Awareness (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Monitoring Cluster Metrics and Statistics with Amazon CloudWatch (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Cluster Metrics ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Dedicated Master Node Metrics ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50EBS Volume Metrics ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Logging Configuration Service Calls Using AWS CloudTrail .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53Amazon ES Information in CloudTrail .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55Understanding Amazon ES Log File Entries ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Tagging Amazon Elasticsearch Service Domains .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55Working with Tags (Console) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56Working with Tags (AWS CLI) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57Working with Tags (AWS SDKs) ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Indexing Data .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Introduction to Indexing .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Programmatic Indexing .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Python .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60Java .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Loading Streaming Data into Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66Loading Streaming Data into Amazon ES from Amazon S3 .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Setting Up to Load Streaming Data into Amazon ES from Amazon S3 .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67Loading Streaming Data into Amazon ES from Amazon Kinesis ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Setting Up to Load Streaming Data into Amazon ES from Amazon Kinesis ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70Loading Streaming Data into Amazon ES from Amazon Kinesis Firehose .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73Loading Streaming Data into Amazon ES from Amazon DynamoDB .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73Loading Streaming Data into Amazon ES from Amazon CloudWatch .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73Loading Data into Amazon ES from AWS IoT .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Working with Index Snapshots .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74Manual Snapshot Prerequisites ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74Registering a Manual Snapshot Directory .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76Taking Manual Snapshots .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77Restoring Snapshots .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Migrating to a Different Elasticsearch Version .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80Kibana and Logstash .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

Kibana .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82Using a Proxy to Access Amazon ES from Kibana .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82Configuring Kibana to Use a WMS Map Server ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84Connecting a Local Kibana Server to Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Loading Bulk Data with the Logstash Plugin .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85Best Practices .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

Sizing Amazon ES Domains .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87Calculating Storage Requirements .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87Choosing the Number of Shards .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88Choosing Instance Types and Testing .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Dedicated Master Nodes .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Recommended CloudWatch Alarms .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Petabyte Scale .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

API Version 2015-01-01iv


VPC Support ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96Limitations .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98About Access Policies on VPC Domains .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98Prerequisites ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99Creating a VPC .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99Reserving IP Addresses in a VPC Subnet .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100Service-Linked Role for VPC Access .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101Migrating from Public Access to VPC Access .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101Amazon VPC Documentation .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Encryption at Rest ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103Enabling Encryption of Data at Rest ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103Disabling Encryption of Data at Rest ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103Monitoring Domains That Encrypt Data at Rest ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103Other Considerations .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Handling AWS Service Errors ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105Failed Cluster Nodes .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105Red Cluster Status .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Recovering from a Continuous Heavy Processing Load .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106Yellow Cluster Status .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107ClusterBlockException .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Lack of Available Storage Space .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107Block Disks Due to Low Memory .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

JVM OutOfMemoryError ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108Troubleshooting .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Kibana: I Can't Sign AWS Service Requests to the Kibana Service Endpoint ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109Kibana: I Don't See the Indices for My Elasticsearch Domain in Kibana 4 .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110Kibana: I Get a Browser Error When I Use Kibana to View My Data .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111Integrations: I Don't See a Service Role for Amazon ES in the IAM Console .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111Domain Creation: Unauthorized Operation When Selecting VPC Access .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112Domain Creation: Stuck at Loading After Choosing VPC Access .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112SDKs: I Get Certificate Errors When I Try to Use an SDK .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

General Reference .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114Supported Instance Types .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114Supported Elasticsearch Operations .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

Notable API Differences .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116Version 6.0 .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117Version 5.5 .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118Version 5.3 .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119Version 5.1 .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119Version 2.3 .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120Version 1.5 .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Supported Plugins .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121Output Plugins .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Other Supported Resources .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123Amazon ES Configuration API Reference .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Actions .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124AddTags .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125CreateElasticsearchDomain .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126DeleteElasticsearchDomain .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130DeleteElasticsearchServiceRole .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132DescribeElasticsearchDomain .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133DescribeElasticsearchDomainConfig .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134DescribeElasticsearchDomains .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137DescribeElasticsearchInstanceTypeLimits ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140ListDomainNames .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143ListElasticsearchInstanceTypes .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144ListElasticsearchVersions .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

API Version 2015-01-01v


ListTags .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147RemoveTags .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148UpdateElasticsearchDomainConfig .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

Data Types .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154AdvancedOptions .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154AdvancedOptionsStatus .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155ARN ..... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155CreateElasticsearchDomainRequest ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155DomainID .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156DomainName .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156DomainNameList ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156EBSOptions .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157ElasticsearchClusterConfig .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157ElasticsearchDomainConfig .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158ElasticsearchDomainStatus .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158ElasticsearchDomainStatusList ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160EncryptionAtRestOptions .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160EncryptionAtRestOptionsStatus .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161EndpointsMap .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161LogPublishingOptions .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161LogPublishingOptionsStatus .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162OptionState .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162OptionStatus .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162ServiceURL .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163SnapshotOptions .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163SnapshotOptionsStatus .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163Tag .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164TagKey .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164TagList ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164TagValue .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164VPCDerivedInfo .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164VPCDerivedInfoStatus .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165VPCOptions .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165VPCOptionsStatus .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

Errors ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166Limits ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

Cluster and Instance Limits ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167EBS Volume Size Limits ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167Network Limits ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169Java Process Limit ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

Using Service-Linked Roles .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171Service-Linked Role Permissions for Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171Creating a Service-Linked Role for Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171Editing a Service-Linked Role for Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172Deleting a Service-Linked Role for Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

Cleaning Up a Service-Linked Role .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172Manually Delete a Service-Linked Role .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

Document History for Amazon ES .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173AWS Glossary .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

API Version 2015-01-01vi


What Is Amazon ElasticsearchService?

Amazon Elasticsearch Service (Amazon ES) is a managed service that makes it easy to create a domainand deploy, operate, and scale Elasticsearch clusters in the AWS Cloud. Elasticsearch is a popular open-source search and analytics engine for use cases such as log analytics, real-time application monitoring,and clickstream analytics. With Amazon ES, you get direct access to the Elasticsearch APIs so thatexisting code and applications work seamlessly with the service.

Amazon ES provisions all the resources for your Elasticsearch cluster and launches the cluster. It alsoautomatically detects and replaces failed Elasticsearch nodes, reducing the overhead associated withself-managed infrastructures. You can scale your cluster with a single API call or a few clicks in theconsole.

To get started using the service, you create an Amazon ES domain. An Amazon ES domain is anElasticsearch cluster in the AWS Cloud that has the compute and storage resources that you specify. Forexample, you can specify the number of instances, instance types, and storage options.

Additionally, Amazon ES offers the following benefits of a managed service:

• Cluster scaling options

• Self-healing clusters

• Replication for high availability

• Data durability

• Enhanced security

• Node monitoring

You can use the Amazon ES console to set up and configure your domain in minutes. If you preferprogrammatic access, you can use the AWS SDKs or the AWS CLI.

There are no upfront costs to set up clusters, and you pay only for the service resources that you use.

Topics• Features of Amazon Elasticsearch Service (p. 2)

• Supported Elasticsearch Versions (p. 2)

• Getting Started with Amazon Elasticsearch Service (p. 3)

• Signing Up for AWS (p. 3)

• Accessing Amazon Elasticsearch Service (p. 4)

• Regions and Endpoints for Amazon Elasticsearch Service (p. 4)

• Choosing Instance Types (p. 5)

• Scaling in Amazon Elasticsearch Service (p. 5)

• Using Amazon EBS Volumes for Storage (p. 5)

• Signing Service Requests (p. 5)

• Related Services (p. 6)

• Pricing for Amazon Elasticsearch Service (p. 7)


http://aws.amazon.com/code

http://docs.aws.amazon.com/cli/latest/userguide/

Amazon Elasticsearch Service Developer GuideFeatures of Amazon Elasticsearch Service

Features of Amazon Elasticsearch ServiceAmazon ES includes the following features:

Scale

• Numerous configurations of CPU, memory, and storage capacity, known as instance types• Up to 1.5 PB of instance storage• Amazon EBS storage volumes

Security

• AWS Identity and Access Management (IAM) access control• Easy integration with Amazon VPC and VPC security groups• Encryption of data at rest

Stability

• Multiple geographical locations for your resources, known as regions and Availability Zones• Dedicated master nodes to offload cluster management tasks• Automated snapshots to back up and restore Amazon ES domains and replicate domains across

Availability Zones• Cluster node allocation across two Availability Zones in the same region, known as zone awareness

Integration with Popular Services

• Data visualization using Kibana• Integration with Amazon CloudWatch for monitoring Amazon ES domain metrics and setting alarms• Integration with AWS CloudTrail for auditing configuration API calls to Amazon ES domains• Integration with Amazon S3, Amazon Kinesis, and Amazon DynamoDB for loading streaming data into

Amazon ES

Supported Elasticsearch VersionsAmazon ES currently supports the following Elasticsearch versions:

• 6.0• 5.5• 5.3• 5.1• 2.3• 1.5

Compared to earlier versions of Elasticsearch, the 5.x and 6.x versions offer powerful features that makethem faster, more secure, and easier to use. Here are some highlights:

• Support for Painless scripting – Painless is an Elasticsearch built-in scripting language. Painless letsyou run advanced queries against your data and automate operations like partial index updates in afast, highly secure way.


https://www.elastic.co/guide/en/elasticsearch/reference/6.0/index.html






Amazon Elasticsearch Service Developer GuideGetting Started with Amazon Elasticsearch Service

• Higher indexing performance – The 5.x and 6.x versions of Elasticsearch provide better indexingcapabilities that significantly increase the throughput of data updates.

• Improved aggregations – The 5.x and 6.x versions of Elasticsearch offer several aggregationimprovements, such as recalculating aggregations only when the data changes. These versions alsodeliver faster query performance.

For more information about the differences between Elasticsearch versions and the APIs that Amazon ESsupports, see the the section called “Supported Elasticsearch Operations” (p. 116).

If you start a new Elasticsearch project, we strongly recommend that you choose the latest supportedElasticsearch version. If you have an existing domain that uses an older Elasticsearch version, you canchoose to keep the domain or migrate your data. For more information, see Migrating to a DifferentElasticsearch Version (p. 80).

Getting Started with Amazon Elasticsearch ServiceTo get started, sign up for an AWS account if you don't already have one. For more information, seeSigning Up for AWS (p. 3).

After you are set up with an account, complete the Getting Started (p. 8) tutorial for AmazonElasticsearch Service. Consult the following introductory topics if you need more information whilelearning about the service.

Get Up and Running

• Signing Up for AWS (p. 3)• Accessing Amazon ES (p. 4)• Getting Started with Amazon ES Domains (p. 8)

Learn the Basics

• Regions and Endpoints for Amazon ES (p. 4)• Amazon Resource Names and AWS Namespaces• Scaling in Amazon ES (p. 5)• Choosing an Elasticsearch Version (p. 2)

Choose Instance Types and Storage

• Choosing an Instance Type (p. 5)• Configuring EBS-based Storage (p. 21)

Stay Secure

• Signing Service Requests (p. 5)• Configuring Access Policies (p. 23)

Signing Up for AWSIf you're not already an AWS customer, your first step is to create an AWS account. If you already have anAWS account, you are automatically signed up for Amazon ES. Your AWS account enables you to access


http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html

Amazon Elasticsearch Service Developer GuideAccessing Amazon ES

Amazon ES and other AWS services such as Amazon S3 and Amazon EC2. There are no sign-up fees,and you don't incur charges until you create a domain. As with other AWS services, you pay only for theresources that you use.

To create an AWS account

1. Open https://aws.amazon.com/, and then choose Create an AWS Account.

NoteThis might be unavailable in your browser if you previously signed into the AWSManagement Console. In that case, choose Sign in to a different account, and then chooseCreate a new AWS account.

2. Follow the online instructions.

Part of the sign-up procedure involves receiving a phone call and entering a PIN using the phonekeypad.

You must enter payment information before you can begin using Amazon ES. Note your AWS accountnumber, because you'll need it later.

Accessing Amazon Elasticsearch ServiceYou can access Amazon ES through the Amazon ES console, the AWS SDKs, or the AWS CLI.

• The Amazon ES console lets you create, configure, and monitor your domains. Using the console is theeasiest way to get started with Amazon ES.

• The AWS SDKs support all the Amazon ES configuration API operations, making it easy to manage yourdomains using your preferred technology. The SDKs automatically sign requests as needed using yourAWS credentials.

• The AWS CLI wraps all the Amazon ES configuration API operations, providing a simple way tocreate and configure domains. The AWS CLI automatically signs requests as needed using your AWScredentials.

Regions and Endpoints for Amazon ElasticsearchService

Amazon ES provides regional endpoints for accessing the configuration API and domain-specificendpoints for accessing the Elasticsearch APIs. You use the configuration service to create and manageyour domains. The regional configuration service endpoints have this format:

es.region.amazonaws.com

For example, es.us-east-1.amazonaws.com. For a list of supported regions, see Regions andEndpoints in the AWS General Reference.

Amazon ES provides a single service endpoint for a domain:

• http://search-domainname-domainid.us-east-1.es.amazonaws.com

You use a domain's endpoint to upload data, submit search requests, and perform any other supportedElasticsearch operations (p. 116).


https://aws.amazon.com/

https://console.aws.amazon.com/es/



http://docs.aws.amazon.com/general/latest/gr/rande.html#elasticsearch-service-regions


Amazon Elasticsearch Service Developer GuideChoosing Instance Types

Choosing Instance TypesAn instance type defines the CPU, RAM, storage capacity, and hourly cost for an instance, the AmazonMachine Image (AMI) that runs as a virtual server in the AWS Cloud. Choose the instance type and thenumber of instances for your domain based on the amount of data you have and number of requests youanticipate. For guidance, see the section called “Sizing Amazon ES Domains” (p. 87).

For general information about instance types, see Amazon Elasticsearch Service Pricing.

Scaling in Amazon Elasticsearch ServiceWhen you create a domain, you choose an initial number of Elasticsearch instances and an instance type.However, these initial choices might not be adequate over time. You can easily accommodate growthby scaling your Amazon ES domain horizontally (more instances) or vertically (larger instance types).Scaling is simple and requires no downtime. To learn more, see the section called “Configuring AmazonES Domains” (p. 18) and the section called “About Configuration Changes” (p. 44).

Using Amazon EBS Volumes for StorageYou have the option of configuring your Amazon ES domain to use an Amazon EBS volume for storingindices rather than the default storage provided by the instance. An Amazon EBS volume is a durable,block-level storage device that you can attach to a single instance. Amazon ES supports the followingEBS volume types:

• Magnetic• General Purpose (SSD)• Provisioned IOPS (SSD)

For an overview, see Amazon EBS Volumes in the Amazon EC2 documentation. For procedures thatshow you how to use Amazon EBS volumes for your Amazon ES domain, see Configuring EBS-basedStorage (p. 21). For information about the minimum and maximum size of supported EBS volumes inan Amazon ES domain, see EBS Volume Size Limits (p. 167).

Signing Service RequestsIf you use a programming language that AWS provides an SDK for, we recommend that you use the SDKto submit HTTP requests to AWS. The AWS SDKs greatly simplify the process of signing requests, andsave you a significant amount of time compared to natively accessing the Elasticsearch APIs. The SDKsintegrate easily with your development environment and provide easy access to related commands. Youalso can use the Amazon ES console and AWS CLI to submit signed requests with no additional effort.

If you choose to call the Elasticsearch APIs directly, you must sign your own requests. Configurationservice requests must always be signed. All requests must be signed unless you configure anonymousaccess for those services. Use the following procedure to sign a request:

1. Calculate a digital signature using a cryptographic hash function. The input must include the text ofyour request and your secret access key.

The function returns a hash value based on your input.2. Include the digital signature in the Authorization header of your request.


https://aws.amazon.com/elasticsearch-service/pricing/

http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/EBSVolumes.html

Amazon Elasticsearch Service Developer GuideRelated Services

The service recalculates the signature using the same hash function and input that you used. If theresulting signature matches the signature in the request, the service processes the request. Otherwise,the service rejects the request.

Amazon ES supports authentication using AWS Signature Version 4. For more information, see SignatureVersion 4 Signing Process. For some examples of signed requests to the Elasticsearch APIs, see thesection called “Programmatic Indexing” (p. 60).

NoteThe service ignores parameters passed in URLs for HTTP POST requests that are signed withSignature Version 4.

Related ServicesAmazon ES commonly is used with the following services:

AWS CloudTrail

Use AWS CloudTrail to get a history of the Amazon ES API calls and related events for your account.CloudTrail is a web service that records API calls from your accounts and delivers the resulting logfiles to your Amazon S3 bucket. You also can use CloudTrail to track changes that were made to yourAWS resources. For more information, see Auditing Amazon Elasticsearch Service Domains with AWSCloudTrail (p. 53).

Amazon CloudWatch

An Amazon ES domain automatically sends metrics to Amazon CloudWatch so that you can gatherand analyze performance statistics. You can monitor these metrics by using the AWS CLI or the AWSSDKs. For more information, see Monitoring Cluster Metrics and Statistics with Amazon CloudWatch(Console) (p. 46).

Kinesis

Kinesis is a managed service that scales elastically for real-time processing of streaming data ata massive scale. Amazon ES provides Lambda sample code for integration with Kinesis. For moreinformation, see Loading Streaming Data into Amazon ES From Kinesis (p. 70).

Amazon S3

Amazon Simple Storage Service (Amazon S3) provides storage for the Internet. You can use AmazonS3 to store and retrieve any amount of data at any time, from anywhere on the web. Amazon ESprovides Lambda sample code for integration with Amazon S3. For more information, see LoadingStreaming Data into Amazon ES from Amazon S3 (p. 66).

AWS IAM

AWS Identity and Access Management (IAM) is a web service that you can use to manage users anduser permissions in AWS. You can use IAM to create user-based access policies for your Amazon ESdomains. For more information, see the IAM documentation.

Amazon ES integrates with the following services to provide data ingestion:

AWS Lambda

AWS Lambda is a zero-administration compute platform for backend web developers that runsyour code in the AWS Cloud. Amazon ES provides sample code to run on Lambda that integrateswith Kinesis and Amazon S3. For more information, see Loading Streaming Data into AmazonES (p. 66).


http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html


http://aws.amazon.com/documentation/cloudtrail/

http://aws.amazon.com/documentation/cloudwatch/

http://aws.amazon.com/documentation/kinesis/

http://aws.amazon.com/documentation/s3/

http://aws.amazon.com/iam/

http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html

http://aws.amazon.com/documentation/lambda/

Amazon Elasticsearch Service Developer GuidePricing for Amazon ES

Amazon DynamoDB

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictableperformance with seamless scalability. Amazon ES provides a Logstash plugin to support DynamoDBStreams and to sign AWS service requests.

Pricing for Amazon Elasticsearch ServiceWith AWS, you pay only for what you use. For Amazon ES, you pay for each hour of use of an EC2instance and for the cumulative size of any EBS storage volumes attached to your instances. StandardAWS data transfer charges also apply.

However, a notable data transfer exception exists. If you use zone awareness (p. 45), Amazon ESdoes not bill for traffic between the two Availability Zones in which the domain resides. Significant datatransfer occurs within a domain during shard allocation and rebalancing. Amazon ES neither meters norbills for this traffic.

For full pricing details, see Amazon Elasticsearch Service Pricing. For information on charges incurredduring configuration changes, see the section called “Charges for Configuration Changes” (p. 45).

If you qualify for the AWS Free Tier, you receive up to 750 hours per month of use with thet2.micro.elasticsearch or t2.small.elasticsearch instance types. You also receive up to 10GB of Amazon EBS storage (Magnetic or General Purpose). For more information, see AWS Free Tier.


http://aws.amazon.com/documentation/dynamodb/

https://aws.amazon.com/ec2/pricing/

https://aws.amazon.com/ec2/pricing/


http://aws.amazon.com/free/

Amazon Elasticsearch Service Developer GuideStep 1: Creating an Amazon ES Domain

Getting Started with AmazonElasticsearch Service

This tutorial shows you how to use Amazon Elasticsearch Service (Amazon ES) to create and configurea domain. It walks you through the basic steps to get a domain up and running quickly. For a detailedwalkthrough, see Creating and Configuring Amazon ES Domains (p. 13) and the other topics within thisguide.

You can complete the following steps by using the Amazon ES console, the AWS CLI, or the AWS SDK:

1. Create an Amazon ES domain (p. 8)

2. Upload data to an Amazon ES domain for indexing (p. 10)

3. Search documents in an Amazon ES domain (p. 11)

4. Delete an Amazon ES domain (p. 12)

For information about installing and setting up the AWS CLI, see the AWS Command Line Interface UserGuide.

NoteIf you want to use the AWS CLI to complete the tutorial, you must use version 1.8.9 or later. Tosee which version of the AWS CLI that you have installed, run the aws --version.

Step 1: Create an Amazon ES DomainImportantThis process is a concise tutorial for configuring a test domain. It should not be used to createproduction domains. For a comprehensive version of the same process, see Creating andConfiguring Amazon ES Domains (p. 13).

An Amazon Elasticsearch Service (Amazon ES) domain encapsulates the Elasticsearch engine instancesthat process HTTP requests to AWS, the indexed data that you want to search, snapshots of the domain,access policies, and metadata. You can create an Amazon ES domain by using the Amazon ES console,the AWS CLI, or the AWS SDK. If you don't already have an account, see the section called “Signing Up forAWS” (p. 3).

To create an Amazon ES domain (console)

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.

2. Under Analytics, choose Elasticsearch Service.

3. On the Define domain page, for Elasticsearch domain name, type a name for the domain. In thisGetting Started tutorial, we use the domain name movies for the examples that we provide later inthe tutorial.

4. For Version, choose an Elasticsearch version for your domain. We recommend that you choose thelatest supported version. For more information, see the section called “Supported ElasticsearchVersions” (p. 2).




https://aws.amazon.com

Amazon Elasticsearch Service Developer GuideStep 1: Creating an Amazon ES Domain

5. Choose Next.6. For Instance count, choose the number of instances that you want. For this tutorial, you can use the

default value of 1.7. For Instance type, choose an instance type for the Amazon ES domain. For this tutoral, we

recommend t2.small.elasticsearch, a small and inexpensive instance type suitable for testingpurposes.

8. For now, you can ignore the Enable dedicated master and Enable zone awareness check boxes.For more information about both, see About Dedicated Master Nodes (p. 90) and Enabling ZoneAwareness (p. 45).

9. For Storage type, choose EBS.

The EBS volume type and EBS volume size boxes appear.

a. For EBS volume type, choose General Purpose (SSD). For more information, see Amazon EBSVolume Types.

b. For EBS volume size, type the size in GB of the external storage for each data node. For thistutorial, you can use the default value of 10.

10. For now, you can ignore Enable encryption at rest. For more information about the feature, seeEncryption at Rest (p. 103).

11. For Automated snapshot start hour, use the default value. For more information, see the sectioncalled “Configuring Automatic Snapshots” (p. 25).

12. Choose Next.13. For simplicity in this tutorial, we'll use an IP-based access policy. On the Set up access page, in the

Network configuration section, choose Public access.14. For Set the domain access policy to, choose Allow access to the domain from specific IP(s) and

enter your public IP address, which you can find by searching for "What is my IP?" on Google. Thenchoose OK.

To learn more about public access, VPC access, and access policies in general, see AccessControl (p. 32) and VPC Support (p. 96).

15. Choose Next.16. On the Review page, review your domain configuration, and then choose Confirm.

NoteNew domains take up to ten minutes to initialize. After your domain is initialized, you canupload data and make changes to the domain.

To create an Amazon ES domain (AWS CLI)

• Run the following command to create an Amazon ES domain.

The command creates a domain named movies with Elasticsearch version 6.0. It specifies oneinstance of the t2.small.elasticsearch instance type. The instance type requires EBS storage,so it specifies a 10 GB volume. Finally, the command applies an IP-based access policy that restrictsaccess to the domain to a single IP address.

You need to replace your_ip_address in the command with your public IP address, which you canfind by searching for "What is my IP?" on Google.

aws es create-elasticsearch-domain --domain-name movies --elasticsearch-version 6.0 --elasticsearch-cluster-config InstanceType=t2.small.elasticsearch,InstanceCount=1 --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=10 --access-policies '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"*"},"Action":["es:*"],"Condition":{"IpAddress":{"aws:SourceIp":["your_ip_address"]}}}]}'


http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html


https://www.google.com

https://www.google.com

Amazon Elasticsearch Service Developer GuideStep 2: Uploading Data for Indexing

NoteInitializing a domain and its resources takes approximately ten minutes. When initialization iscomplete, the endpoint of the domain is available for index and Amazon ES requests.

Use the following command to query the status of the new domain:

aws es describe-elasticsearch-domain --domain movies

To create an Amazon ES domain (AWS SDKs)

The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the Amazon ESConfiguration API Reference (p. 124), including the CreateElasticsearchDomain action. For moreinformation about installing and using the AWS SDKs, see AWS Software Development Kits.

Step 2: Upload Data to an Amazon ES Domain forIndexing

You can upload data to an Amazon ES domain for indexing using the Elasticsearch index and bulk APIsfrom the command line.

• Use the index API to add or update a single Elasticsearch document.• Use the bulk API to add or update multiple Elasticsearch documents that are described in the same

JSON file.

The following example requests use curl, a common HTTP client. Clients like curl can't perform therequest signing that is required if your access policies specify IAM users or roles. To successfully performthe instructions in this step, you must use an IP address-based access policy that allows unauthenticatedaccess, like you configured in step 1 (p. 8).

You can install curl on Windows and use it from the command prompt, but Windows users might find itmore convenient to use a tool like Cygwin or the Windows Subsystem for Linux. macOS and most Linuxdistributions come with curl pre-installed.

To upload a single document to an Amazon ES domain

• Run the following command to add a single document to the movies domain:

curl -XPUT elasticsearch_domain_endpoint/movies/movie/1 -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}' -H 'Content-Type: application/json'

For a detailed explanation of this command and how to make signed requests to Amazon ES, seeIndexing Data (p. 59).

To upload a JSON file that contains multiple documents to an Amazon ES domain

1. Create a file called bulk_movies.json. Copy and paste the following content into it, including thetrailing newline:

{ "index" : { "_index": "movies", "_type" : "movie", "_id" : "2" } }{"director": "Frankenheimer, John", "genre": ["Drama", "Mystery", "Thriller"], "year": 1962, "actor": ["Lansbury, Angela", "Sinatra, Frank", "Leigh, Janet", "Harvey,



https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

https://curl.haxx.se/

https://www.cygwin.com/

https://msdn.microsoft.com/en-us/commandline/wsl/about

Amazon Elasticsearch Service Developer GuideStep 3: Searching Documents in an Amazon ES Domain

Laurence", "Silva, Henry", "Frees, Paul", "Gregory, James", "Bissell, Whit", "McGiver, John", "Parrish, Leslie", "Edwards, James", "Flowers, Bess", "Dhiegh, Khigh", "Payne, Julie", "Kleeb, Helen", "Gray, Joe", "Nalder, Reggie", "Stevens, Bert", "Masters, Michael", "Lowell, Tom"], "title": "The Manchurian Candidate"}{ "index" : { "_index": "movies", "_type" : "movie", "_id" : "3" } }{"director": "Baird, Stuart", "genre": ["Action", "Crime", "Thriller"], "year": 1998, "actor": ["Downey Jr., Robert", "Jones, Tommy Lee", "Snipes, Wesley", "Pantoliano, Joe", "Jacob, Ir\u00e8ne", "Nelligan, Kate", "Roebuck, Daniel", "Malahide, Patrick", "Richardson, LaTanya", "Wood, Tom", "Kosik, Thomas", "Stellate, Nick", "Minkoff, Robert", "Brown, Spitfire", "Foster, Reese", "Spielbauer, Bruce", "Mukherji, Kevin", "Cray, Ed", "Fordham, David", "Jett, Charlie"], "title": "U.S. Marshals"}{ "index" : { "_index": "movies", "_type" : "movie", "_id" : "4" } }{"director": "Ray, Nicholas", "genre": ["Drama", "Romance"], "year": 1955, "actor": ["Hopper, Dennis", "Wood, Natalie", "Dean, James", "Mineo, Sal", "Backus, Jim", "Platt, Edward", "Ray, Nicholas", "Hopper, William", "Allen, Corey", "Birch, Paul", "Hudson, Rochelle", "Doran, Ann", "Hicks, Chuck", "Leigh, Nelson", "Williams, Robert", "Wessel, Dick", "Bryar, Paul", "Sessions, Almira", "McMahon, David", "Peters Jr., House"], "title": "Rebel Without a Cause"}

2. Run the following command to upload the file to the movies domain:

curl -XPOST elasticsearch_domain_endpoint/_bulk --data-binary @bulk_movies.json -H 'Content-Type: application/json'

For more information about the bulk file format, see Indexing Data (p. 59).

NoteThe service supports migrating data from manual snapshots taken on both Amazon ES and self-managed Elasticsearch clusters. Restoring indices from a self-managed Elasticsearch cluster isa common way to migrate data into Amazon ES. For more information, see the section called“Restoring Snapshots” (p. 78).

Step 3: Search Documents in an Amazon ESDomain

To search documents in an Amazon ES domain, use the Elasticsearch search API. Alternatively, you canuse Kibana (p. 82) to search documents in the domain.

NoteRequests using curl are unauthenticated and rely on an IP-based access policy.

To search documents from the command line

• Run the following command to search the movies domain for the word mars:

curl -XGET 'elasticsearch_domain_endpoint/movies/_search?q=mars'

To search documents from an Amazon ES domain by using Kibana (console)

1. Point your browser to the Kibana plugin for your Amazon ES domain. You can find the Kibanaendpoint on your domain dashboard on the Amazon ES console. The URL follows the format of:

https://domain.region.es.amazonaws.com/_plugin/kibana/

2. To use Kibana, you must configure at least one index pattern. Kibana uses these patterns to identitywhich indices you want to analyze. For this tutorial, enter movies and choose Create.


https://www.elastic.co/guide/en/elasticsearch/reference/current/_the_search_api.html

Amazon Elasticsearch Service Developer GuideStep 4: Deleting an Amazon ES Domain

3. The Index Patterns screen shows your various document fields, fields like actor and director. Fornow, choose Discover to search your data.

4. In the search bar, type mars, and then press Enter. Note how the similarity score (_score) increaseswhen you search for the phrase mars attacks.

Step 4: Delete an Amazon ES DomainBecause the movies domain from this tutorial is for test purposes, you should delete it when you arefinished experimenting to avoid incurring charges.

To delete an Amazon ES domain (console)

1. Log in to the Amazon Elasticsearch Service console.2. In the navigation pane, under My domains, choose the movies domain.3. Choose Delete Elasticsearch domain.4. Choose Delete domain.5. Select the Delete the domain check box, and then choose Delete.

To delete an Amazon ES domain (AWS CLI)

• Run the following command to delete the movies domain:

aws es delete-elasticsearch-domain --domain-name movies

NoteDeleting a domain deletes all billable Amazon ES resources. However, any manual snapshots ofthe domain that you created using the native Elasticsearch API are not deleted. Consider savinga snapshot if you might need to recreate the Amazon ES domain in the future. If you don't planto recreate the domain, you can safely delete any snapshots that you created manually.

To delete an Amazon ES domain (AWS SDKs)

The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the Amazon ESConfiguration API Reference (p. 124), including the DeleteElasticsearchDomain action. For moreinformation about installing and using the AWS SDKs, see AWS Software Development Kits.



Amazon Elasticsearch Service Developer GuideCreating Amazon ES Domains

Creating and Configuring AmazonElasticsearch Service Domains

This chapter describes how to create and configure Amazon Elasticsearch Service (Amazon ES) domains.An Amazon ES domain is the hardware, software, and data exposed by Amazon Elasticsearch Serviceendpoints.

Unlike the brief instructions in the Getting Started (p. 8) tutorial, this chapter describes all options andprovides relevant reference information. You can complete each procedure by using instructions for theAmazon ES console, the AWS Command Line Interface (AWS CLI), or the AWS SDKs.

Topics• Creating Amazon ES Domains (p. 13)• Configuring Amazon ES Domains (p. 18)• Configuring EBS-based Storage (p. 21)• Modifying VPC Access Configuration (p. 23)• Configuring Access Policies (p. 23)• Configuring Automatic Snapshots (p. 25)• Configuring Advanced Options (p. 26)• Configuring Slow Logs (p. 28)

Creating Amazon ES DomainsThis section describes how to create Amazon ES domains by using the Amazon ES console or by usingthe AWS CLI with the create-elasticsearch-domain command. The procedures for the AWS CLIinclude syntax and examples.

Creating Amazon ES Domains (Console)Use the following procedure to create an Amazon ES domain by using the console.

To create an Amazon ES domain (console)

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. Choose Create a new domain.

Alternatively, choose Get Started if this is the first Amazon ES domain that you will create for yourAWS account.

4. On the Define domain page, for Domain name, type a name for your domain. The domain namemust meet the following criteria:

• Uniquely identifies a domain• Starts with a lowercase letter• Contains between 3 and 28 characters



Amazon Elasticsearch Service Developer GuideCreating Amazon ES Domains (Console)

• Contains only lowercase letters a-z, the numbers 0-9, and the hyphen (-)5. For Version, choose an Elasticsearch version for your domain. We recommend that you choose the

latest version. For more information, see the section called “Supported Elasticsearch Versions” (p. 2).6. Choose Next.7. For Instance count, choose the number of instances that you want.

The default is one. For maximum values, see the section called “Cluster and InstanceLimits” (p. 167). We recommend a minimum of three instances to avoid potential Elasticsearchissues, such as the split brain issue. If you have three dedicated master nodes (p. 90), westill recommend a minimum of two data nodes for replication. Single node clusters are fine fordevelopment and testing, but should not be used for production workloads. For more guidance, seethe section called “Sizing Amazon ES Domains” (p. 87).

8. For Instance type, choose an instance type for the data nodes.

To see a list of the instance types that Amazon ES supports, see Supported Instance Types (p. 114).9. (Optional) If you must ensure cluster stability or if you have a domain that has more than 10

instances, enable a dedicated master node. Dedicated master nodes increase cluster stability and arerequired for a domain that has an instance count greater than 10. For more information, see AboutDedicated Master Nodes (p. 90).

a. Select the Enable dedicated master check box.b. For Dedicated master instance type, choose an instance type for the dedicated master node.

For a list of the instance types that Amazon ES supports, see Supported InstanceTypes (p. 114).

Note

• You can choose an instance type for the dedicated master node that differs fromthe instance type that you choose for the data nodes. For example, you might selectgeneral purpose or storage-optimized instances for your data nodes, but compute-optimized instances for your dedicated master nodes.

c. For Dedicated master instance count, choose the number of instances for the dedicated masternode.

We recommend choosing an odd number of instances to avoid potential Elasticsearch issues,such as the split brain issue. The default and recommended number is three.

10. (Optional) To provide high availability for data nodes, select the Enable zone awareness check box.

Zone awareness distributes Amazon ES data nodes across two Availability Zones in the same region.If you enable zone awareness, you must have an even number of instances in the instance count,and you must use the native Elasticsearch API to create replica shards for your cluster. This processallows for the even distribution of shards across two Availability Zones. For more information, seeEnabling Zone Awareness (p. 45).

11. For Storage type, choose either Instance (the default) or EBS.

If your Amazon ES domain requires more storage, use an EBS volume for storage rather than thestorage that is attached to the selected instance type. Domains with large indices or large numbersof indices often benefit from the increased storage capacity of EBS volumes. For guidance oncreating especially large domains, see Petabyte Scale (p. 94). If you choose EBS, the followingboxes appear:

a. For EBS volume type, choose an EBS volume type.

If you choose Provisioned IOPS (SSD) for the EBS volume type, for Provisioned IOPS, type thebaseline IOPS performance that you want. For more information, see Amazon EBS Volumes inthe Amazon EC2 documentation.


https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#split-brain

https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html#getting-started-shards-and-replicas



Amazon Elasticsearch Service Developer GuideCreating Amazon ES Domains (Console)

b. For EBS volume size, type the size of the EBS volume that you want to attach to each datanode.

EBS volume size is per node. You can calculate the total cluster size for the Amazon ES domainusing the following formula: (number of data nodes) * (EBS volume size). The minimum andmaximum size of an EBS volume depends on both the specified EBS volume type and theinstance type that it's attached to. To learn more, see EBS Volume Size Limits (p. 167).

12. (Optional) To enable encryption of data at rest, select the Enable encryption at rest check box.

Select (Default) aws/es to have Amazon ES create a KMS encryption key on your behalf (or use theone that it already created). Otherwise, choose your own KMS encryption key from the KMS masterkey menu. To learn more, see Encryption at Rest (p. 103).

13. For Automated snapshot start hour, choose the hour for automated daily snapshots of domainindices.

For more information and recommendations, see the section called “Configuring AutomaticSnapshots” (p. 25).

14. (Optional) Choose Advanced options. For a summary of options, see the section called “ConfiguringAdvanced Options” (p. 26)

15. Choose Next.16. On the Set up access page, in the Network configuration section, choose either Public Access or

VPC access. If you choose Public access, skip to step 17. If you choose VPC access, ensure that youhave met the prerequisites (p. 99), and then do the following:

a. For VPC, choose the ID of the VPC that you want to use.

NoteThe VPC and domain must be in the same AWS Region, and you must select a VPCwith tenancy set to Default. Amazon ES does not yet support VPCs that use dedicatedtenancy.

b. For Subnet, choose a subnet. If you enabled zone awareness in step 10, you must choose twosubnets. Amazon ES will place a VPC endpoint and elastic network interfaces (ENIs) in the subnetor subnets.

NoteYou must reserve sufficient IP addresses for the network interfaces in the subnet (orsubnets). For more information, see Reserving IP Addresses in a VPC Subnet (p. 100).

c. For Security groups, choose the VPC security groups that need access to the Amazon ESdomain. For more information, see VPC Support (p. 96).

d. For IAM role, keep the default role. Amazon ES uses this predefined role (also known as aservice-linked role) to access your VPC and to place a VPC endpoint and network interfaces inthe subnet of the VPC. For more information, see Service-Linked Role for VPC Access (p. 101).

17. For Set the domain access policy to, choose a preconfigured policy from the Select a templatedropdown list and edit it to meet the needs of your domain. Alternatively, you can add one or moreIdentity and Access Management (IAM) policy statements in the Add or edit the access policy box.For more information, see Access Control (p. 32), the section called “Configuring Access Policies” (p. 23), and the section called “About Access Policies on VPC Domains” (p. 98).

NoteIf you chose VPC access in step 16, the IP-based policy template is not available in thedropdown list, and you can't configure an IP-based policy manually. Instead, you can usesecurity groups to control which IP addresses can access the domain. To learn more, see thesection called “About Access Policies on VPC Domains” (p. 98).

18. Choose Next.19. On the Review page, review your domain configuration, and then choose Confirm and create.20. Choose OK.


http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_SecurityGroups.html

Amazon Elasticsearch Service Developer GuideCreating Amazon ES Domains (AWS CLI)

NoteNew domains take up to ten minutes to initialize. After your domain is initialized, you canupload data and make changes to the domain.

Creating Amazon ES Domains (AWS CLI)Instead of creating an Amazon ES domain by using the console, you can create a domain by using theAWS CLI. Use the following syntax to create an Amazon ES domain.

aws es create-elasticsearch-domain --domain-name <value>

[--elasticsearch-version <value>] [--elasticsearch-cluster-config <value>] [--ebs-options <value>] [--access-policies <value>] [--snapshot-options <value>] [--vpc-options <value>] [--advanced-options <value>] [--log-publishing-options <value>] [--cli-input-json <value>] [--generate-cli-skeleton <value>] [--encryption-at-rest-options <value>]

The following table provides more information about each of the optional parameters.

Optional Parameter Description

--elasticsearch-version Specifies the Elasticsearch version of the domain. If not specified,the default value is 1.5. For more information, see Choosing anElasticsearch Version (p. 2).

--elasticsearch-cluster-config

Specifies the instance type and count of the domain, whetherzone awareness is enabled, and whether the domain uses adedicated master node. Dedicated master nodes increase clusterstability and are required for a domain that has an instance countgreater than 10. For more information, see Configuring AmazonES Domains (p. 20).

--ebs-options Specifies whether the domain uses an EBS volume for storage.If true, this parameter must also specify the EBS volume type,size, and, if applicable, IOPS value. For more information, seeConfiguring EBS-based Storage (p. 22).

--access-policies Specifies the access policy for the domain. For more information,see Configuring Access Policies (p. 24).

--snapshot-options Specifies the hour in UTC during which the service performsa daily automated snapshot of the indices in the domain. Thedefault value is 0, or midnight, which means that the snapshotis taken anytime between midnight and 1:00 AM. For moreinformation, see Configuring Snapshots (p. 26).

--advanced-options Specifies whether to allow references to indices in the bodies ofHTTP request objects. For more information, see ConfiguringAdvanced Options (p. 27).

--generate-cli-skeleton Displays JSON for all specified parameters. Save the outputto a file so that you can later read the file with the --cli-


Amazon Elasticsearch Service Developer GuideCreating Amazon ES Domains (AWS CLI)

Optional Parameter Description

input-json parameter rather than typing the parametersat the command line. For more information, see Generate CLISkeleton and CLI Input JSON Parameters in the AWS CommandLine Interface User Guide.

--cli-input-json Specifies the name of a JSON file that contains a set of CLIparameters. For more information, see Generate CLI Skeleton andCLI Input JSON Parameters in the AWS Command Line InterfaceUser Guide.

--log-publishing-options Specifies whether Amazon ES should publish Elasticsearch slowlogs to CloudWatch. For more information, see the section called“Configuring Slow Logs” (p. 28).

--vpc-options Specifies whether to launch the Amazon ES domain within anAmazon VPC (VPC). To learn more, see VPC Support (p. 96).

--encryption-at-rest-options

Specifies whether to enable encryption of data at rest (p. 103).

Examples

The first example demonstrates the following Amazon ES domain configuration:

• Creates an Amazon ES domain named weblogs with Elasticsearch version 5.5• Populates the domain with two instances of the m4.large.elasticsearch instance type• Uses a 100 GB Magnetic disk EBS volume for storage for each data node• Allows anonymous access, but only from a single IP address: 192.0.2.0/32

aws es create-elasticsearch-domain --domain-name weblogs --elasticsearch-version 5.5 --elasticsearch-cluster-config InstanceType=m4.large.elasticsearch,InstanceCount=2 --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=100 --access-policies '{"Version": "2012-10-17", "Statement": [{"Action": "es:*", "Principal":"*","Effect": "Allow", "Condition": {"IpAddress":{"aws:SourceIp":["192.0.2.0/32"]}}}]}'

The next example demonstrates the following Amazon ES domain configuration:

• Creates an Amazon ES domain named weblogs with Elasticsearch version 5.5• Populates the domain with six instances of the m4.large.elasticsearch instance type• Uses a 100 GB General Purpose (SSD) EBS volume for storage for each data node• Restricts access to the service to a single user, identified by the user's AWS account ID: 555555555555• Enables zone awareness

aws es create-elasticsearch-domain --domain-name weblogs --elasticsearch-version 5.5 --elasticsearch-cluster-config InstanceType=m4.large.elasticsearch,InstanceCount=6,ZoneAwarenessEnabled=true --ebs-options EBSEnabled=true,VolumeType=gp2,VolumeSize=100 --access-policies '{"Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::555555555555:root" }, "Action":"es:*", "Resource": "arn:aws:es:us-east-1:555555555555:domain/logs/*" } ] }'

The next example demonstrates the following Amazon ES domain configuration:


http://docs.aws.amazon.com/cli/latest/userguide/generate-cli-skeleton.html




Amazon Elasticsearch Service Developer GuideCreating Amazon ES Domains (AWS SDKs)

• Creates an Amazon ES domain named weblogs with Elasticsearch version 5.5• Populates the domain with ten instances of the m4.xlarge.elasticsearch instance type• Populates the domain with three instances of the m4.large.elasticsearch instance type to serve as

dedicated master nodes• Uses a 100 GB Provisioned IOPS EBS volume for storage, configured with a baseline performance of

1000 IOPS for each data node• Restricts access to a single user and to a single subresource, the _search API• Configures automated daily snapshots of the indices for 03:00 UTC

aws es create-elasticsearch-domain --domain-name weblogs --elasticsearch-version 5.5 --elasticsearch-cluster-config InstanceType=m4.xlarge.elasticsearch,InstanceCount=10,DedicatedMasterEnabled=true,DedicatedMasterType=m4.large.elasticsearch,DedicatedMasterCount=3 --ebs-options EBSEnabled=true,VolumeType=io1,VolumeSize=100,Iops=1000 --access-policies '{"Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::555555555555:root" }, "Action": "es:*", "Resource": "arn:aws:es:us-east-1:555555555555:domain/mylogs/_search" } ] }' --snapshot-options AutomatedSnapshotStartHour=3

NoteIf you attempt to create an Amazon ES domain and a domain with the same name alreadyexists, the CLI does not report an error. Instead, it returns details for the existing domain.

Creating Amazon ES Domains (AWS SDKs)The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the Amazon ESConfiguration API Reference (p. 124), including CreateElasticsearchDomain. For more informationabout installing and using the AWS SDKs, see AWS Software Development Kits.

Configuring Amazon ES DomainsTo meet the demands of increased traffic and data, you can update your Amazon ES domainconfiguration with any of the following changes:

• Change the instance count• Change the instance type• Enable or disable dedicated master nodes• Enable or disable Zone Awareness• Configure storage configuration• Change the start time for automated snapshots of domain indices• Change the VPC subnets and security groups• Configure advanced options

NoteFor information about configuring a domain to use an EBS volume for storage, see ConfiguringEBS-based Storage (p. 21).

Configuring Amazon ES Domains (Console)Use the following procedure to update your Amazon ES configuration by using the console.



Amazon Elasticsearch Service Developer GuideConfiguring Amazon ES Domains (Console)

To configure an Amazon ES domain (console)



3. In the navigation pane, under My domains, choose the domain that you want to update.

4. Choose Configure cluster.

5. On the Configure cluster page, update the configuration of the domain.

The cluster is a collection of one or more data nodes, optional dedicated master nodes, and storagerequired to run Amazon ES and operate your domain.

a. If you want to change the instance type for data nodes, for Instance type, choose a newinstance type.

To see a list of the instance types that Amazon ES supports, see Supported InstanceTypes (p. 114).

b. If you want to change the instance count, for Instance count, choose an integer from one totwenty. To request an increase up to 100 instances per domain, create a case with the AWSSupport Center.

c. If you want to improve cluster stability or if your domain has an instance count greater than 10,enable a dedicated master node for your cluster. For more information, see About DedicatedMaster Nodes (p. 90).

i. Select the Enable dedicated master check box.

ii. For Dedicated master instance type, choose an instance type for the dedicated masternode.

You can choose an instance type for the dedicated master node that differs from theinstance type that you choose for the data nodes.

To see a list of the instance types that Amazon ES supports, see Supported InstanceTypes (p. 114).

iii. For Dedicated master instance count, choose the number of instances for the dedicatedmaster node.

We recommend choosing an odd number of instances to avoid potential Amazon ES issues,such as the split brain issue. The default and recommended number is three.

d. If you want to enable zone awareness, select the Enable zone awareness check box. If youenable zone awareness, you must have an even number of instances in your instance count. Thisallows for the even distribution of shards across two Availability Zones in the same region.

e. If you want to change the hour during which the service takes automated daily snapshots of theprimary index shards of your Amazon ES domain, for Automated snapshot start hour, choosean integer.

f. If you didn't enable VPC access when you created the domain, skip to step 7. If you enabled VPCaccess, you can change the subnet that the VPC endpoint is placed in, and you can change thesecurity groups:

i. For Subnets, choose a subnet. The subnet must have a sufficient number of IP addressesreserved for the network interfaces. If you enabled zone awareness, you must choose twosubnets. The subnets must be in different Availability Zones in the same region. For moreinformation, see VPC Support (p. 96).

ii. For Security groups, add the security groups that need access to the domain.

g. (Optional) Choose Advanced options. For a summary of options, see the section called“Configuring Advanced Options” (p. 26)

h. Choose Submit.API Version 2015-01-01

19


https://console.aws.amazon.com/support/home#/


https://www.elastic.co/guide/en/elasticsearch/reference/5.3/modules-node.html#split-brain

Amazon Elasticsearch Service Developer GuideConfiguring Amazon ES Domains (AWS CLI)

Configuring Amazon ES Domains (AWS CLI)Use the elasticsearch-cluster-config option to configure your Amazon ES cluster by using theAWS CLI. The following syntax is used by both the create-elasticsearch-domain and update-elasticsearch-domain-config commands.

Syntax

--elasticsearch-cluster-config InstanceType=<value>,InstanceCount=<value>,DedicatedMasterEnabled=<value>,DedicatedMasterType=<value>,DedicatedMasterCount=<value>,ZoneAwarenessEnabled=<value>

NoteDo not include spaces between parameters for the same option.

The following table describes the parameters in more detail.

Parameter Valid Values Description

InstanceType Any supportedinstance type. SeeSupported InstanceTypes.

The hardware configuration of the computerthat hosts the instance. The default ism4.large.elasticsearch.

InstanceCount Integer The number of instances in the AmazonES domain. The default is one, and themaximum default limit is twenty. To requestan increase up to 100 instances per domain,create a case with the AWS Support Center.

DedicatedMasterEnabled true or false Specifies whether to use a dedicated masternode for the Amazon ES domain. The defaultvalue is false.

DedicatedMasterType Any supportedinstance type

The hardware configuration of the computerthat hosts the master node. The default ism4.large.elasticsearch.

DedicatedMasterCount Integer The number of instances used for thededicated master node. The default is three.

ZoneAwarenessEnabled true or false Specifies whether to enable zone awarenessfor the Amazon ES domain. The defaultvalue is false.

Examples

The following example creates an Amazon ES domain named mylogs with Elasticsearch version 5.5 withtwo instances of the m4.large.elasticsearch instance type and zone awareness enabled:

aws es create-elasticsearch-domain --domain-name mylogs --elasticsearch-version 5.5 --elasticsearch-cluster-config InstanceType=m4.large.elasticsearch,InstanceCount=2,DedicatedMasterEnabled=false,ZoneAwarenessEnabled=true

However, you likely will want to reconfigure your new Amazon ES domain as network traffic grows andas the quantity and size of documents increase. For example, you might decide to use a larger instance


http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-instance-types.html

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-instance-types.html


Amazon Elasticsearch Service Developer GuideConfiguring Amazon ES Domains (AWS SDKs)

type, use more instances, and enable a dedicated master node. The following example updates thedomain configuration with these changes:

aws es update-elasticsearch-domain-config --domain-name mylogs --elasticsearch-cluster-config InstanceType=m4.xlarge.elasticsearch,InstanceCount=3,DedicatedMasterEnabled=true,DedicatedMasterType=m4.large.elasticsearch,DedicatedMasterCount=3

Configuring Amazon ES Domains (AWS SDKs)The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the Amazon ESConfiguration API Reference (p. 124), including UpdateElasticsearchDomainConfig. For moreinformation about installing and using the AWS SDKs, see AWS Software Development Kits.

Configuring EBS-based StorageAn Amazon EBS volume is a block-level storage device that you can attach to a single instance. EBSvolumes enable you to independently scale the storage resources of your Amazon ES domain from itscompute resources. EBS volumes are most useful for domains with large datasets, but without the needfor large compute resources. EBS volumes are much larger than the default storage provided by theinstance. Amazon Elasticsearch Service supports the following EBS volume types:

• General Purpose (SSD)

• Provisioned IOPS (SSD)

• Magnetic

NoteWhen changing an EBS volume type from provisioned IOPS to non-provisioned EBS volumetypes, set the IOPS value to 0.

WarningCurrently, if the data node that is attached to an EBS volume fails, the EBS volume also fails.

Configuring EBS-based Storage (Console)Use the following procedure to enable EBS-based storage by using the console.

To enable EBS-based storage (console)



3. In the navigation pane, under My domains, choose the domain that you want to configure.

4. Choose Configure cluster.

5. For Storage type, choose EBS.

6. For EBS volume type, choose an EBS volume type.

If you choose Provisioned IOPS (SSD) for the EBS volume type, for Provisioned IOPS, type thebaseline IOPS performance that you want.

7. For EBS volume size, type the size that you want for the EBS volume.

EBS volume size is per node. You can calculate the total cluster size for the Amazon ES domainusing the following formula: (number of data nodes) * (EBS volume size). The minimum and




Amazon Elasticsearch Service Developer GuideConfiguring EBS-based Storage (AWS CLI)

maximum size of an EBS volume depends on both the specified EBS volume type and the instancetype to which it is attached. To learn more, see EBS Volume Size Limits (p. 167).

8. Choose Submit.

NoteSet the IOPS value for a Provisioned IOPS EBS volume to no more than 30 times the maximumstorage of the volume. For example, if your volume has a maximum size of 100 GB, you can'tassign an IOPS value for it that is greater than 3000.

For more information, see Amazon EBS Volumes in the Amazon EC2 documentation.

Configuring EBS-based Storage (AWS CLI)Use the --ebs-options option to configure EBS-based storage by using the AWS CLI. The followingsyntax is used by both the create-elasticsearch-domain and update-elasticsearch-domain-config commands.

Syntax

--ebs-options EBSEnabled=<value>,VolumeType=<value>,VolumeSize=<value>,IOPS=<value>


EBSEnabled true or false Specifies whether to use an EBS volume for storagerather than the storage provided by the instance. Thedefault value is false.

VolumeType Any of the following:

• gp2 (General PurposeSSD)

• io1 (Provisioned IOPSSSD)

• standard (Magnetic)

The EBS volume type to use with the Amazon ESdomain.

VolumeSize Integer Specifies the size of the EBS volume for each data node.The minimum and maximum size of an EBS volumedepends on both the specified EBS volume type and theinstance type to which it is attached. To see a table thatshows the minimum and maximum EBS size for eachinstance type, see Service Limits.

IOPS Integer Specifies the baseline I/O performance for the EBSvolume. This parameter is used only by ProvisionedIOPS (SSD) volumes. The minimum value is 1000. Themaximum value is 16000.

NoteWe recommend that you do not set the IOPS value for a Provisioned IOPS EBS volume to morethan 30 times the maximum storage of the volume. For example, if your volume has a maximumsize of 100 GB, you should not assign an IOPS value for it that is greater than 3000. For moreinformation, including use cases for each volume type, see Amazon EBS Volume Types in theAmazon EC2 documentation.

Examples



http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html


Amazon Elasticsearch Service Developer GuideConfiguring EBS-based Storage (AWS SDKs)

The following example creates a domain named mylogs with Elasticsearch version 5.5 with a 10 GBGeneral Purpose EBS volume:

aws es create-elasticsearch-domain --domain-name=mylogs --elasticsearch-version 5.5 --ebs-options EBSEnabled=true,VolumeType=gp2,VolumeSize=10

However, you might need a larger EBS volume as the size of your search indices increases. For example,you might opt for a 100 GB Provisioned IOPS volume with a baseline I/O performance of 3000 IOPS. Thefollowing example updates the domain configuration with those changes:

aws es update-elasticsearch-domain-config --domain-name=mylogs --ebs-options EBSEnabled=true,VolumeType=io1,VolumeSize=100,IOPS=3000

Configuring EBS-based Storage (AWS SDKs)The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the AmazonES Configuration API Reference (p. 124), including the --ebs-options parameter forUpdateElasticsearchDomainConfig. For more information about installing and using the AWSSDKs, see AWS Software Development Kits.

Modifying VPC Access ConfigurationIf you configured a domain to reside within a VPC, you can modify the configuration using the AmazonES console. To migrate a public domain to a VPC domain, see the section called “Migrating from PublicAccess to VPC Access” (p. 101).

Configuring VPC Access (Console)Use the following procedure to configure VPC access by using the console.

To configure VPC access (console)

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. In the navigation pane, under My domains, choose the domain that you want to configure.4. Choose Configure cluster.5. In the Network configuration section, for Subnets, choose a subnet. If you enabled zone awareness,

you must choose two subnets. The subnets must be in different Availability Zones in the sameregion. For more information, see VPC Support (p. 96).

NoteYou must reserve sufficient IP addresses for the network interfaces in the subnet (orsubnets). For more information, see the section called “Reserving IP Addresses in a VPCSubnet” (p. 100).

6. For Security groups, add the security groups that need access to the domain.7. Choose Submit.

Configuring Access PoliciesAmazon Elasticsearch Service offers several ways to configure access to your Amazon ES domains. Formore information, see Access Control (p. 32).




Amazon Elasticsearch Service Developer GuideConfiguring Access Policies (Console)

The console provides preconfigured access policies that you can customize for the specific needs ofyour domain. You also can import access policies from other Amazon ES domains. For information onhow these access policies interact with VPC access, see the section called “About Access Policies on VPCDomains” (p. 98).

Configuring Access Policies (Console)Use the following procedure to configure access policies by using the console.

To configure access policies (console)

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. In the navigation pane, under My domains, choose the domain that you want to update.4. Choose Modify access policy.5. Edit the access policy.

Alternatively, choose one of the policy templates from the Select a template dropdown list, andthen edit it as needed for your domain.

Preconfigured Access Policy Description

Allow or deny access to one or moreAWS accounts or IAM users

Allows or denies access to one or more AWS accounts orIAM users or roles.

Allow access to the domain fromspecific IP(s)

This policy is used to restrict anonymous access to aspecific IP address or range of IP addresses.

NoteIf you enabled VPC access for your domain, thispreconfigured policy is not available. Instead,you can use security groups to control which IPaddresses can access the domain. To learn more,see the section called “About Access Policies onVPC Domains” (p. 98).

Deny access to the domain This policy allows access only through the Amazon ESconsole or by the owner of the AWS account who createdthe domain.

Copy access policy from anotherdomain

This template provides a convenient way to import anexisting access policy from another domain.

Allow open access to the domain This policy is not recommended for domains with publicendpoints. It allows anyone to delete, modify, or accessindices in the domain. It is intended only as a conveniencefor testing. Don't load sensitive data into a domain thatuses this setting.

6. Choose Submit.

Configuring Access Policies (AWS CLI)Use the --access-policies option to configure access policies by using the AWS CLI. The followingsyntax is used by both the create-elasticsearch-domain and update-elasticsearch-domain-config commands.




Amazon Elasticsearch Service Developer GuideConfiguring Access Policies (AWS SDKs)

Syntax

--access-policies=<value>

Parameter ValidValues

Description

--access-policies

JSON Specifies the access policy for the Amazon ES domain.

Example

The following resource-based policy example restricts access to the service to a single user, identified bythe user's AWS account ID, 555555555555, in the Principal policy element. This user receives access toindex1, but can't access other indices in the domain:

aws es update-elasticsearch-domain-config --domain-name mylogs --access-policies '{"Version": "2012-10-17", "Statement": [ { "Effect": "Allow","Principal": {"AWS": "arn:aws:iam::123456789012:root" },"Action":"es:*","Resource":"arn:aws:es:us-east-1:555555555555:domain/index1/*" } ] }'

TipIf you configure access policies using the AWS CLI, you can use one of many online tools tominify the JSON policy statement.

Configuring Access Policies (AWS SDKs)The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the AmazonES Configuration API Reference (p. 124), including the --access-policies parameter forUpdateElasticsearchDomainConfig. For more information about installing and using the AWSSDKs, see AWS Software Development Kits.

Configuring Automatic SnapshotsAmazon Elasticsearch Service provides automatic daily snapshots of a domain's primary index shardsand the number of replica shards. By default, the service takes automatic snapshots at midnight, but youshould choose a time when the service is under minimal load.

For information on working with these snapshots, see the section called “Restoring Snapshots” (p. 78).

WarningThe service stops taking snapshots of Amazon ES indices while the health of a cluster is red. Anydocuments that you add to a red cluster, even to indices with a health status of green, can belost in the event of a cluster failure due to this lack of backups. To prevent loss of data, returnthe health of your cluster to green before uploading additional data to any index in the cluster.

Configuring Snapshots (Console)Use the following procedure to configure daily automatic index snapshots by using the console.

To configure automatic snapshots



Amazon Elasticsearch Service Developer GuideConfiguring Snapshots (AWS CLI)

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. In the navigation pane, under My domains, choose the domain that you want to update.4. Choose Configure cluster.5. For Automated snapshot start hour, choose the new hour for the service to take automated

snapshots.6. Choose Submit.

Configuring Snapshots (AWS CLI)Use the following syntax for the --snapshot-options option. The syntax for the option is the samefor both the create-elasticsearch-domain and update-elasticsearch-domain-configcommands.

Syntax

--snapshot-options AutomatedSnapshotStartHour=<value>


AutomatedSnapshotStartHour Integerbetween 0 and23

Specifies the hour in UTC during which theservice performs a daily automated snapshotof the indices in the new domain. The defaultvalue is 0, or midnight, which means that thesnapshot is taken anytime between midnightand 1:00 AM.

Example

The following example configures automatic snapshots at 01:00 UTC:

aws es update-elasticsearch-domain-config --domain-name mylogs --region us-east-2 --snapshot-options AutomatedSnapshotStartHour=1

Configuring Snapshots (AWS SDKs)The AWS SDKs (except the Android and iOS SDKs) support all the actions that are defined in the AmazonES Configuration API Reference (p. 124). This includes the --snapshots-options parameter forUpdateElasticsearchDomainConfig. For more information about installing and using the AWSSDKs, see AWS Software Development Kits.

Configuring Advanced OptionsUse advanced options to configure the following:

rest.action.multi.allow_explicit

Specifies whether explicit references to indices are allowed inside the body of HTTP requests. Settingthis property to false prevents users from bypassing access control for subresources. By default, the




Amazon Elasticsearch Service Developer GuideConfiguring Advanced Options (Console)

value is true. For more information, see URL-based access control and the section called “AdvancedOptions and API Considerations” (p. 41).

indices.fielddata.cache.size

Specifies the percentage of Java heap space that is allocated to field data. By default, this setting isunbounded.

NoteMany customers query rotating daily indices. We recommend that you begin benchmarktesting with indices.fielddata.cache.size configured to 40% of the JVM heap formost such use cases. However, if you have very large indices you might need a large fielddata cache.

indices.query.bool.max_clause_count

Specifies the maximum number of clauses allowed in a Lucene Boolean query. 1024 is the default.Queries with more than the permitted number of clauses result in a TooManyClauses error. Formore information, see the Lucene documentation.

Configuring Advanced Options (Console)To configure advanced options (console)

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. In the navigation pane, under My domains, choose the domain that you want to update.4. Choose Configure cluster.5. Choose Advanced options.6. Specify the options that you want and choose Submit.

Configuring Advanced Options (AWS CLI)Use the following syntax for the --advanced-options option. The syntax for the option is the samefor both the create-elasticsearch-domain and update-elasticsearch-domain-configcommands.

Syntax

--advanced-options rest.action.multi.allow_explicit=<true|false>, indices.fielddata.cache.size=<percentage_heap>, indices.query.bool.max_clause_count=<int>

Parameter Valid Values

rest.action.multi.allow_explicit=<true|false>

indices.fielddata.cache.size=<percentage_heap>

--advanced-options

indices.query.bool.max_clause_count=<int>

Example

The following example disables explicit references to indices in the HTTP request bodies. It also limits thefield data cache to 40 percent of the total Java heap:


https://www.elastic.co/guide/en/elasticsearch/reference/current/url-access-control.html

https://lucene.apache.org/core/6_6_0/core/org/apache/lucene/search/BooleanQuery.html


Amazon Elasticsearch Service Developer GuideConfiguring Advanced Options (AWS SDKs)

aws es update-elasticsearch-domain-config --domain-name mylogs --region us-east-1 --advanced-options rest.action.multi.allow_explicit_index=false, indices.fielddata.cache.size=40

Configuring Advanced Options (AWS SDKs)The AWS SDKs (except the Android and iOS SDKs) support all of the actions defined in the AmazonES Configuration API Reference (p. 124), including the --advanced-options parameter forUpdateElasticsearchDomainConfig. For more information about installing and using the AWSSDKs, see AWS Software Development Kits.

Configuring Slow LogsSlow logs are an Elasticsearch feature that Amazon ES exposes through Amazon CloudWatch Logs. Theselogs are useful for troubleshooting performance issues, but are disabled by default. If enabled, standardCloudWatch pricing applies.

Amazon ES exposes two slow logs: search and index. To learn more about these logs, see Slow Log in theElasticsearch documentation.

Enabling Slow Logs Publishing (Console)Use the following procedure to enable the publishing of slow logs to CloudWatch.

To enable slow logs publishing to CloudWatch

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. In the navigation pane, under My domains, choose the domain that you want to update.4. On the Logs tab, choose Enable for the log that you want.5. Create a CloudWatch log group, or choose an existing one.

NoteIf you plan to enable search and index slow logs, we recommend publishing each to its ownlog group. This separation makes the logs easier to scan.

6. Choose an access policy that contains the appropriate permissions, or create a policy using the JSONthat the console provides:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "es.amazonaws.com" }, "Action": [ "logs:PutLogEvents", "logs:CreateLogStream" ], "Resource": "cw_log_group_arn" } ]}



https://aws.amazon.com/cloudwatch/pricing/

https://aws.amazon.com/cloudwatch/pricing/

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html


Amazon Elasticsearch Service Developer GuideEnabling Slow Logs Publishing (AWS CLI)

ImportantCloudWatch Logs supports 10 resource policies per region. If you plan to enable slow logsfor several Amazon ES domains, you should create and reuse a broader policy that includesmultiple log groups to avoid reaching this limit.

7. Choose Enable.

The status of your domain changes from Active to Processing. The status must return to Activebefore log publishing is enabled. This process can take up to 30 minutes.

After you enable log publishing, see the section called “Setting Elasticsearch LoggingThresholds” (p. 30).

Enabling Slow Logs Publishing (AWS CLI)Before you can enable log publishing, you need a CloudWatch log group. If you don't already have one,you can create one using the following command:

aws logs create-log-group --log-group-name my-log-group

Type the next command to find the log group's ARN, and then make a note of it:

aws logs describe-log-groups --log-group-name my-log-group

Now you can give Amazon ES permissions to write to the log group. You must provide the log group'sARN near the end of the command:

aws logs put-resource-policy --policy-name my-policy --policy-document '{ "Version": "2012-10-17", "Statement": [{ "Sid": "", "Effect": "Allow", "Principal": { "Service": "es.amazonaws.com"}, "Action":[ "logs:PutLogEvents"," logs:PutLogEventsBatch","logs:CreateLogStream"],"Resource": "cw_log_group_arn"}]}'

ImportantCloudWatch Logs supports 10 resource policies per region. If you plan to enable slow logs forseveral Amazon ES domains, you should create and reuse a broader policy that includes multiplelog groups to avoid reaching this limit.

Finally, you can use the --log-publishing-options option to enable publishing. The syntax for theoption is the same for both the create-elasticsearch-domain and update-elasticsearch-domain-config commands.

Parameter Valid Values

SEARCH_SLOW_LOGS={CloudWatchLogsLogGroupArn=cw_log_group_arn,Enabled=true|false}

--log-publishing-options

INDEX_SLOW_LOGS={CloudWatchLogsLogGroupArn=cw_log_group_arn,Enabled=true|false}

NoteIf you plan to enable search and index slow logs, we recommend publishing each to its own loggroup. This separation makes the logs easier to scan.

Example


http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutResourcePolicy.html

http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutResourcePolicy.html

Amazon Elasticsearch Service Developer GuideEnabling Slow Logs Publishing (AWS SDKs)

The following example enables the publishing of search and index slow logs for the specified domain:

aws es update-elasticsearch-domain-config --domain-name my-domain --log-publishing-options "SEARCH_SLOW_LOGS={CloudWatchLogsLogGroupArn=arn:aws:logs:us-east-1:123456789012:log-group:my-log-group,Enabled=true},INDEX_SLOW_LOGS={CloudWatchLogsLogGroupArn=arn:aws:logs:us-east-1:123456789012:log-group:my-other-log-group,Enabled=true}"

To disable publishing to CloudWatch, run the same command with Enabled=false.


Enabling Slow Logs Publishing (AWS SDKs)Before you can enable slow logs publishing, you must first create a CloudWatch log group, get its ARN,and give Amazon ES permissions to write to it. The relevant operations are documented in the AmazonCloudWatch Logs API Reference:

• CreateLogGroup

• DescribeLogGroup

• PutResourcePolicy

You can access these operations using the AWS SDKs.

The AWS SDKs (except the Android and iOS SDKs) support all the operations that are defined in theAmazon ES Configuration API Reference (p. 124), including the --log-publishing-options optionfor CreateElasticsearchDomain and UpdateElasticsearchDomainConfig.


Setting Elasticsearch Logging ThresholdsElasticsearch disables slow logs by default. After you enable the publishing of slow logs to CloudWatch,you still must specify logging thresholds for each Elasticsearch index. These thresholds define preciselywhat should be logged and at which log level. Settings vary slightly by Elasticsearch version. Forsummaries of each version, see Slow Log in the Elasticsearch documentation.

You specify these settings through the Elasticsearch REST API. An example follows:

curl -XPUT elasticsearch_domain_endpoint/index/_settings --data '{"index.search.slowlog.threshold.query.warn": "5s","index.search.slowlog.threshold.query.info": "2s"}' -H 'Content-Type: application/json'

To test that slow logs are publishing successfully, consider starting with very low values to verify thatlogs appear in CloudWatch, and then increase the thresholds to more useful levels.

If the logs don't appear, check the following:

• Does the CloudWatch log group exist? Check the CloudWatch console.• Does Amazon ES have permissions to write to the log group? Check the Amazon ES console.• Is the Amazon ES domain configured to publish to the log group? Check the Amazon ES

console, use the AWS CLI describe-elasticsearch-domain-config option, or callDescribeElasticsearchDomainConfig using one of the SDKs.


http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/

http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/

https://aws.amazon.com/tools/#sdk


Amazon Elasticsearch Service Developer GuideViewing Slow Logs

• Are the Elasticsearch logging thresholds low enough that your requests are exceeding them? To reviewyour thresholds for an index, use the following command:

curl -XGET elasticsearch_domain_endpoint/index/_settings?pretty

If you want to disable slow logs for an index, return any thresholds that you changed to their defaultvalues of -1.

Disabling publishing to CloudWatch using the Amazon ES console or AWS CLI does not stop Elasticsearchfrom generating logs; it only stops the publishing of those logs. Be sure to check your index settings ifyou no longer need the slow logs.

Viewing Slow LogsViewing the slow logs in CloudWatch is just like viewing any other CloudWatch log. For moreinformation, see View Log Data in the Amazon CloudWatch Logs User Guide.

Here are some considerations for viewing the logs:

• Amazon ES publishes only the first 255,000 characters of each line of the slow logs to CloudWatch.Any remaining content is truncated.

• In CloudWatch, the log stream names have suffixes of -index-slow-logs or -search-slow-logsto help identify their contents.


http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/ViewingLogData.html

Amazon Elasticsearch Service Developer GuideTypes of Policies

Amazon Elasticsearch Service AccessControl

Amazon Elasticsearch Service offers several ways of controlling access to your domains. This sectioncovers the various policy types, how they interact with each other, and how to create your own, custompolicies.

ImportantVPC support introduces some additional considerations to Amazon ES access control. For moreinformation, see the section called “About Access Policies on VPC Domains” (p. 98).

Types of PoliciesAmazon ES supports three types of access policies:

• the section called “Resource-based Policies” (p. 32)• the section called “Identity-based policies” (p. 34)• the section called “IP-based Policies” (p. 35)

Resource-based PoliciesYou attach resource-based policies to domains. These policies specify which actions a principal canperform on the domain's subresources. Subresources include Elasticsearch indices and APIs.

The Principal element specifies the accounts, users, or roles that are allowed access. The Resourceelement specifies which subresources these principals can access. The following resource-based policygrants test-user full access (es:*) to test-domain:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::123456789012:user/test-user" ] }, "Action": [ "es:*" ], "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/*" } ]}

Two important considerations apply to this policy:

• These privileges apply only to this domain. Unless you create additional policies, test-user can'taccess other domains or even view a list of them in the Amazon ES dashboard.

• The trailing /* in the Resource element is significant. Despite having full access, test-user canperform these actions only on the domain's subresources, not on the domain's configuration.


http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html

http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_resource.html

Amazon Elasticsearch Service Developer GuideResource-based Policies

For example, test-user can make requests against an index (GET https://search-test-domain.us-west-1.es.amazonaws.com/test-index), but can't update the domain'sconfiguration (POST https://es.us-west-1.amazonaws.com/2015-01-01/es/domain/test-domain/config). Note the difference between the two endpoints. Accessing the configurationAPI (p. 124) requires an identity-based policy (p. 34).

To further restrict test-user, you can apply the following policy:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::123456789012:user/test-user" ] }, "Action": [ "es:ESHttpGet" ], "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/test-index/_search" } ]}

Now test-user can perform only one operation: searches against test-index. All other indices withinthe domain are inaccessible, and without permissions to use the es:ESHttpPut or es:ESHttpPostactions, test-user can't add or modify documents.

Next, you might decide to configure a role for power users. This policy allows power-user-role accessto all HTTP methods, except for the ability to delete a critical index and its documents:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::123456789012:role/power-user-role" ] }, "Action": [ "es:ESHttpDelete", "es:ESHttpGet", "es:ESHttpHead", "es:ESHttpPost", "es:ESHttpPut" ], "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/*" }, { "Effect": "Deny", "Principal": { "AWS": [ "arn:aws:iam::123456789012:role/power-user-role" ] }, "Action": [ "es:ESHttpDelete"


Amazon Elasticsearch Service Developer GuideIdentity-based policies

], "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/critical-index*" } ]}

For information about all available actions, see the section called “Policy Element Reference” (p. 36).

Identity-based policiesUnlike resource-based policies, which you attach to domains in Amazon ES, you attach identity-basedpolicies to users or roles using the AWS Identity and Access Management (IAM) service. Just like resource-based policies (p. 32), identity-based policies specify who can access a service, which actions they canperform, and if applicable, the resources on which they can perform those actions.

While they certainly don't have to be, identify-based policies tend to be more generic. They often governthe basic, service-level actions a user can perform. After you have these policies in place, you can useresource-based policies in Amazon ES to offer users additional permissions.

Because identity-based policies attach to users or roles (principals), the JSON doesn't specify a principal.The following policy grants access to actions that begin with Describe and List and allows GETrequests against all domains. This combination of actions provides read-only access:

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "es:Describe*", "es:List*", "es:ESHttpGet" ], "Effect": "Allow", "Resource": "*" } ]}

An administrator might have full access to Amazon ES:

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "es:*" ], "Effect": "Allow", "Resource": "*" } ]}

For more information about the differences between resource-based and identity-based policies, see IAMPolicies in the IAM User Guide.

NoteUsers with the AWS managed AmazonESReadOnlyAccess policy can't see cluster health statusin the console. To allow them to see cluster health status, add the "es:ESHttpGet" action toan access policy and attach it to their accounts or roles.


http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html

http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html

Amazon Elasticsearch Service Developer GuideIP-based Policies

IP-based PoliciesIP-based policies restrict access to a domain to one or more IP addresses or CIDR blocks. Technically, IP-based policies are not a distinct type of policy. Instead, they are just resource-based policies that specifyan anonymous principal and include a special Condition element.

The primary appeal of IP-based policies is that they allow unsigned requests to an Amazon ES domain,which lets you use clients like curl and the section called “Kibana” (p. 82) or access the domainthrough a proxy server. To learn more, see the section called “Using a Proxy to Access Amazon ES fromKibana” (p. 82).

NoteIf you enabled VPC access for your domain, you can't configure an IP-based policy. Instead,you can use security groups to control which IP addresses can access the domain. For moreinformation, see the section called “About Access Policies on VPC Domains” (p. 98).

The following IP-based access policy grants all requests that originate from 12.345.678.901 access totest-domain:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [ "es:*" ], "Condition": { "IpAddress": { "aws:SourceIp": [ "12.345.678.901" ] } }, "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/*" } ]}

Signing Amazon ES RequestsEven if you configure a completely open resource-based access policy, all requests to the Amazon ESconfiguration API must be signed. If your policies specify IAM users or roles, requests to the ElasticsearchAPIs also must be signed. The signing method differs by API:

• To make calls to the Amazon ES configuration API, we recommend that you use one of the AWS SDKs.The SDKs greatly simplify the process and can save you a significant amount of time compared tocreating and signing your own requests.

• To make calls to the Elasticsearch APIs, you must sign your own requests. For sample code, see thesection called “Programmatic Indexing” (p. 60).

To sign a request, you calculate a digital signature using a cryptographic hash function, which returnsa hash value based on the input. The input includes the text of your request and your secret access key.


http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition.html



https://aws.amazon.com/tools/#sdk

Amazon Elasticsearch Service Developer GuideWhen Policies Collide

The hash function returns a hash value that you include in the request as your signature. The signature ispart of the Authorization header of your request.

After receiving your request, Amazon ES recalculates the signature using the same hash function andinput that you used to sign the request. If the resulting signature matches the signature in the request,Amazon ES processes the request. Otherwise, Amazon ES rejects the request.

Amazon ES supports authentication using AWS Signature Version 4. For more information, see SignatureVersion 4 Signing Process.

NoteThe service ignores parameters passed in URLs for HTTP POST requests that are signed withSignature Version 4.

When Policies CollideComplexities arise when policies disagree or make no explicit mention of a user. Understanding How IAMWorks in the IAM User Guide provides a concise summary of policy evaluation logic:

• By default, all requests are denied.• An explicit allow overrides this default.• An explicit deny overrides any allows.

For example, if a resource-based policy grants you access to a domain, but an identify-based policydenies you access, you are denied access. If an identity-based policy grants access and a resource-basedpolicy does not specify whether or not you should have access, you are allowed access. See the followingtable of intersecting policies for a full summary of outcomes.

Allowed in Resource-based Policy

Denied in Resource-based Policy

Neither Allowed norDenied in Resource-based Policy

Allowed in Identity-based Policy

Allow Deny Allow

Denied in Identity-based Policy

Deny Deny Deny

Neither Allowed norDenied in Identity-based Policy

Allow Deny Deny

Policy Element ReferenceAmazon ES supports all the policy elements that are documented in the IAM Policy Elements Reference.The following table shows the most common elements.

JSON Policy Element Summary

Version The current version of the policy language is 2012-10-17. All access policiesshould specify this value.

Effect This element specifies whether the statement allows or denies access to thespecified actions. Valid values are Allow or Deny.




http://docs.aws.amazon.com/IAM/latest/UserGuide/intro-structure.html#intro-structure-authorization

http://docs.aws.amazon.com/IAM/latest/UserGuide/intro-structure.html#intro-structure-authorization

http://docs.aws.amazon.com/IAM/latest/UserGuide/AccessPolicyLanguage_ElementDescriptions.html

Amazon Elasticsearch Service Developer GuidePolicy Element Reference


Principal This element specifies the AWS account or IAM user or role that is allowed ordenied access to a resource and can take several forms:

• AWS accounts: "Principal":{"AWS": ["123456789012"]} or"Principal":{"AWS": ["arn:aws:iam::123456789012:root"]}

• IAM users: "Principal":{"AWS":["arn:aws:iam::123456789012:user/test-user"]}

• IAM roles: "Principal":{"AWS":["arn:aws:iam::123456789012:role/test-role"]}

Specifying the * wildcard enables anonymous access to the domain, whichwe don't recommend unless you add an IP-based condition.




Action Amazon ES uses the following actions for HTTP methods:

• es:ESHttpDelete

• es:ESHttpGet

• es:ESHttpHead

• es:ESHttpPost

• es:ESHttpPut

Amazon ES uses the following actions for the configuration API (p. 124):

• es:AddTags

• es:CreateElasticsearchDomain

• es:DeleteElasticsearchDomain

• es:DeleteElasticsearchServiceRole

• es:DescribeElasticsearchDomain

• es:DescribeElasticsearchDomainConfig

• es:DescribeElasticsearchDomains

• es:DescribeElasticsearchInstanceTypeLimits

• es:ListDomainNames

• es:ListElasticsearchInstanceTypes

• es:ListElasticsearchVersions

• es:ListTags

• es:RemoveTags

• es:UpdateElasticsearchDomainConfig

TipYou can use wildcards to specify a subset of actions, such as"Action":"es:*" or "Action":"es:Describe*".

Certain es: actions support resource-level permissions. For example, youcan give a user permissions to delete one particular domain without givingthat user permissions to delete any domain. Other actions apply only to theservice itself. For example, es:ListDomainNames makes no sense in thecontext of a single domain and thus requires a wildcard.

ImportantResource-based policies differ from resource-level permissions.Resource-based policies (p. 32) are full JSON policies that attachto domains. Resource-level permissions let you restrict actions toparticular domains or subresources. In practice, you can think ofresource-level permissions as an optional part of a resource- oridentity-based policy.

The following identity-based policy (p. 34) lists all es: actions andgroups them according to whether they apply to the domain subresources(test-domain/*), to the domain configuration (test-domain), or only tothe service (*):

{ "Version": "2012-10-17",



JSON Policy Element Summary "Statement": [ { "Effect": "Allow", "Action": [ "es:ESHttpDelete", "es:ESHttpGet", "es:ESHttpHead", "es:ESHttpPost", "es:ESHttpPut" ], "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/*" }, { "Effect": "Allow", "Action": [ "es:CreateElasticsearchDomain", "es:DeleteElasticsearchDomain", "es:DescribeElasticsearchDomain", "es:DescribeElasticsearchDomainConfig", "es:DescribeElasticsearchDomains", "es:UpdateElasticsearchDomainConfig" ], "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain" }, { "Effect": "Allow", "Action": [ "es:AddTags", "es:DeleteElasticsearchServiceRole", "es:DescribeElasticsearchInstanceTypeLimits", "es:ListDomainNames", "es:ListElasticsearchInstanceTypes", "es:ListElasticsearchVersions", "es:ListTags", "es:RemoveTags" ], "Resource": "*" } ]}

NoteWhile resource-level permissions fores:CreateElasticsearchDomain might seem unintuitive—after all, why give a user permissions to create a domainthat already exists?—the use of a wildcard lets you enforce asimple naming scheme for your domains, such as "Resource":"arn:aws:es:us-west-1:987654321098:domain/my-team-name-*".

Of course, nothing prevents you from including actions alongside lessrestrictive resource elements, such as the following:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [



JSON Policy Element Summary "es:ESHttpGet", "es:DescribeElasticsearchDomain" ], "Resource": "*" } ]}

To learn more about pairing actions and resources, see the Resourceelement in this table.

Condition Amazon ES supports all the conditions that are described in Available GlobalCondition Keys in the IAM User Guide.

When configuring an IP-based policy (p. 35), you specify the IP addressesor CIDR block as a condition, such as the following:

"Condition": { "IpAddress": { "aws:SourceIp": [ "192.0.2.0/32" ] }}


http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#AvailableKeys

http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#AvailableKeys

Amazon Elasticsearch Service Developer GuideAdvanced Options and API Considerations


Resource Amazon ES uses Resource elements in three basic ways:

• For actions that apply to Amazon ES itself, like es:ListDomainNames, orto allow full access, use the following syntax:

"Resource": "*"

• For actions that involve a domain's configuration, likees:DescribeElasticsearchDomain, you can use the following syntax:

"Resource": "arn:aws:es:region:aws-account-id:domain/domain-name"

• For actions that apply to a domain's subresources, like es:ESHttpGet,you can use the following syntax:

"Resource": "arn:aws:es:region:aws-account-id:domain/domain-name/*"

You don't have to use a wildcard. Amazon ES lets you define a differentaccess policy for each Elasticsearch index or API. For example, you mightlimit a user's permissions to the test-index index:

"Resource": "arn:aws:es:region:aws-account-id:domain/domain-name/test-index"

Instead of full access to test-index, you might prefer to limit the policyto just the search API.

"Resource": "arn:aws:es:region:aws-account-id:domain/domain-name/test-index/_search"

You can even control access to individual documents:

"Resource": "arn:aws:es:region:aws-account-id:domain/domain-name/test-index/test-type/1"

Essentially, if Elasticsearch expresses the subresource as an endpoint, youcan control access to it.

For details about which actions support resource-level permissions, see theAction element in this table.

Advanced Options and API ConsiderationsAmazon ES has several advanced options, one of which has access control implications:rest.action.multi.allow_explicit_index. At its default setting of true, it allows users to bypasssubresource permissions under certain circumstances.

For example, consider the following resource-based policy:


https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html

Amazon Elasticsearch Service Developer GuideAdvanced Options and API Considerations

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::123456789012:user/test-user" ] }, "Action": [ "es:ESHttp*" ], "Resource": [ "arn:aws:es:us-west-1:987654321098:domain/test-domain/test-index/*", "arn:aws:es:us-west-1:987654321098:domain/test-domain/_bulk" ] }, { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::123456789012:user/test-user" ] }, "Action": [ "es:ESHttpGet" ], "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/restricted-index/*" } ]}

This policy grants test-user full access to test-index and the Elasticsearch bulk API. It also allowsGET requests to restricted-index.

The following indexing request, as you might expect, fails due to a permissions error:

PUT https://search-test-domain.us-west-1.es.amazonaws.com/restricted-index/movie/1{ "title": "Your Name", "director": "Makoto Shinkai", "year": "2016"}

Unlike the index API, the bulk API lets you create, update, and delete many documents in a single call.You often specify these operations in the request body, however, rather than in the request URL. BecauseAmazon ES uses URLs to control access to domain subresources, test-user can, in fact, use the bulk APIto make changes to restricted-index. Even though the user lacks POST permissions on the index, thefollowing request succeeds:

POST https://search-test-domain.us-west-1.es.amazonaws.com/_bulk{ "index" : { "_index": "restricted-index", "_type" : "movie", "_id" : "1" } }{ "title": "Your Name", "director": "Makoto Shinkai", "year": "2016" }

In this situation, the access policy fails to fulfill its intent. To prevent users from bypassing these kindsof restrictions, you can change rest.action.multi.allow_explicit_index to false. If this valueis false, all calls to the bulk, mget, and msearch APIs that specify index names in the request body stopworking. In other words, calls to _bulk no longer work, but calls to test-index/_bulk do. This secondendpoint contains an index name, so you don't need to specify one in the request body.



https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-multi-get.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html

Amazon Elasticsearch Service Developer GuideConfiguring Access Policies

Kibana (p. 82) relies heavily on mget and msearch, so it is unlikely to work properly after this change.For partial remediation, you can leave rest.action.multi.allow_explicit as true and denycertain users access to one or more of these APIs.

For information about changing this setting, see the section called “Configuring AdvancedOptions” (p. 26).

Similarly, the following resource-based policy contains two subtle issues:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123456789012:user/test-user" }, "Action": "es:ESHttp*", "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/*" }, { "Effect": "Deny", "Principal": { "AWS": "arn:aws:iam::123456789012:user/test-user" }, "Action": "es:*", "Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/restricted-index/*" } ]}

• Despite the explicit deny, test-user can still make calls such as GET https://search-test-domain.us-west-1.es.amazonaws.com/_all/_search and GET https://search-test-domain.us-west-1.es.amazonaws.com/*/_search to access the documents in restricted-index.

• Because the Resource element references restricted-index/*, test-user doesn't havepermissions to directly access the index's documents. The user does, however, have permissions todelete the entire index. To prevent access and deletion, the policy instead must specify restricted-index*.

Rather than mixing broad allows and focused denies, the safest approach is to follow the principle ofleast privilege and grant only the permissions that are required to perform a task.

Configuring Access Policies• For instructions on creating or modifying resource- and IP-based policies in Amazon ES, see the section

called “Configuring Access Policies ” (p. 23).• For instructions on creating or modifying identity-based policies in IAM, see Creating IAM Policies in

the IAM User Guide.

Additional Sample PoliciesAlthough this chapter includes many sample policies, AWS access control is a complex subject that is bestunderstood through examples. For more, see Example Policies in the IAM User Guide.


https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege

http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html

http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_examples.html

Amazon Elasticsearch Service Developer GuideAbout Configuration Changes

Managing Amazon ElasticsearchService Domains

As the size and number of documents in your Amazon Elasticsearch Service (Amazon ES) domain growand as network traffic increases, you likely will need to update the configuration of your Elasticsearchcluster. To know when it's time to reconfigure your domain, you need to monitor domain metrics. Youmight also need to audit data-related API calls to your domain or assign tags to your domain. Thissection describes how to perform these and other tasks related to managing your domains.

Topics

• About Configuration Changes (p. 44)

• Charges for Configuration Changes (p. 45)

• Enabling Zone Awareness (Console) (p. 45)

• Monitoring Cluster Metrics and Statistics with Amazon CloudWatch (Console) (p. 46)

• Auditing Amazon Elasticsearch Service Domains with AWS CloudTrail (p. 53)

• Tagging Amazon Elasticsearch Service Domains (p. 55)

About Configuration ChangesAmazon ES uses a blue/green deployment process when updating domains. Blue/green typically refersto the practice of running two production environments, one live and one idle, and switching the twoas you make software changes. In the case of Amazon ES, it refers to the practice of creating a newenvironment for domain updates and routing users to the new environment after those updates arecomplete. The practice minimizes downtime and maintains the original environment in the event thatdeployment to the new environment is unsuccessful.

Domain updates occur when you make most configuration changes, but they also occur when theAmazon ES team makes certain software changes to the service. If you make configuration changes, thedomain state changes to Processing. If the Amazon ES team makes software changes, the state remainsActive. In both cases, you can review the cluster health and Amazon CloudWatch metrics and see thatthe number of nodes in the cluster temporarily doubles while the domain update occurs. In the followingillustration, you can see the number of nodes doubling from 11 to 22 during a configuration change andreturning to 11 when the update is complete.

This temporary doubling can strain the cluster's dedicated master nodes (p. 90), which suddenly havetwice as many nodes to manage. It is important to maintain sufficient capacity on dedicated masternodes to handle the overhead that is associated with these blue/green deployments.


Amazon Elasticsearch Service Developer GuideCharges for Configuration Changes

ImportantYou do not incur any additional charges during configuration changes and service maintenance.You are billed only for the number of nodes that you request for your cluster. For specifics, seethe section called “Charges for Configuration Changes” (p. 45).

To prevent overloading dedicated master nodes, you can monitor usage with the Amazon CloudWatchmetrics (p. 46). For recommended maximum values, see the section called “RecommendedCloudWatch Alarms” (p. 92).

Charges for Configuration ChangesIf you change the configuration for a domain, Amazon ES creates a new cluster as described in thesection called “About Configuration Changes” (p. 44). During the migration of old to new, you incurthe following charges:

• If you change the instance type, you are charged for both clusters for the first hour. After the first hour,you are charged only for the new cluster.

Example: You change the configuration from three m3.xlarge instances to four m4.large instances.For the first hour, you are charged for both clusters (3 * m3.xlarge + 4 * m4.large). After the firsthour, you are charged only for the new cluster (4 * m4.large).

• If you don’t change the instance type, you are charged only for the largest cluster for the first hour.After the first hour, you are charged only for the new cluster.

Example: You change the configuration from six m3.xlarge instances to three m3.xlarge instances.For the first hour, you are charged for the largest cluster (6 * m3.xlarge). After the first hour, you arecharged only for the new cluster (3 * m3.xlarge).

Enabling Zone Awareness (Console)Each AWS Region is a separate geographic area with multiple, isolated locations known as AvailabilityZones. To prevent data loss and minimize downtime in the event of node and data center failure, you canuse the Amazon ES console to allocate nodes and replica index shards that belong to an Elasticsearchcluster across two Availability Zones in the same region. This allocation is known as zone awareness.If you enable zone awareness, you also must use the native Elasticsearch API to create replica shardsfor your cluster. Amazon ES distributes the replicas across the nodes in the Availability Zones, whichincreases the availability of your cluster. Enabling zone awareness for a cluster slightly increases networklatencies.

ImportantZone awareness requires an even number of instances in the instance count. The defaultconfiguration for any index is a replica count of 1. If you specify a replica count of 0 for an index,zone awareness doesn't replicate the shards to the second Availability Zone. Without replicashards, there are no replicas to distribute to a second Availability Zone, and enabling the featuredoesn't provide protection from data loss.

If you enable zone awareness and you use a VPC endpoint to connect to your domain, you must specifyAvailability Zones for the domain. Some instance types might not be available in some AvailabilityZones. For more information about using VPC endpoints, see VPCs and VPC Endpoints for Amazon ESDomains (p. 96).

The following illustration shows a four-node cluster with zone awareness enabled. The service placesall the primary index shards in one Availability Zone and all the replica shards in the second AvailabilityZone.


http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-zoneawareness

Amazon Elasticsearch Service Developer GuideMonitoring Cluster Metrics and Statistics

with Amazon CloudWatch (Console)

To enable zone awareness (console)

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. In the navigation pane, under My domains, choose your Amazon ES domain.4. Choose Configure cluster.5. In the Node configuration pane, choose Enable zone awareness.6. Choose Submit.

For more information, see Regions and Availability Zones in the EC2 documentation.

Monitoring Cluster Metrics and Statistics withAmazon CloudWatch (Console)

Amazon ES domains send performance metrics to Amazon CloudWatch every minute. If you use generalpurpose or magnetic EBS volumes, the EBS volume metrics only update every five minutes. Use theMonitoring tab in the Amazon Elasticsearch Service console to view these metrics, provided at no extracharge.

Statistics provide you with broader insight into each metric. For example, view the Average statistic forthe CPUUtilization metric to compute the average CPU utilization for all nodes in the cluster. Each of themetrics falls into one of three categories:



http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html

Amazon Elasticsearch Service Developer GuideCluster Metrics

• Cluster metrics (p. 47)• Dedicated master node metrics (p. 50)• EBS volume metrics (p. 52)

NoteThe service archives the metrics for two weeks before discarding them.

To view configurable statistics for a metric (console)

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. In the navigation pane, under My domains, choose your Amazon ES domain.4. Choose the Monitoring tab.5. Choose the metric that you want to view.6. From the Statistic list, select a statistic.

For a list of relevant statistics for each metric, see the tables in Cluster Metrics (p. 47). Somestatistics are not relevant for a given metric. For example, the Sum statistic is not meaningful for theNodes metric.

7. Choose Update graph.

Cluster MetricsNoteTo check your cluster metrics if metrics are unavailable in the Amazon Elasticsearch Serviceconsole, use Amazon CloudWatch.

The AWS/ES namespace includes the following metrics for clusters.

Metric Description

ClusterStatus.green Indicates that all index shards are allocated to nodes in the cluster.

Relevant statistics: Minimum, Maximum

ClusterStatus.yellow Indicates that the primary shards for all indices are allocated to nodesin a cluster, but the replica shards for at least one index are not. Singlenode clusters always initialize with this cluster status because thereis no second node to which a replica can be assigned. You can eitherincrease your node count to obtain a green cluster status, or you canuse the Elasticsearch API to set the number_of_replicas settingfor your index to 0. For more information, see Configuring AmazonElasticsearch Service Domains and Update Indices Settings in theElasticsearch documentation.


ClusterStatus.red Indicates that the primary and replica shards of at least one index arenot allocated to nodes in a cluster. To recover, you must delete theindices or restore a snapsnot and then add EBS-based storage, uselarger instance types, or add instances. For more information, see RedCluster Status.




http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomains-configure-cluster

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomains-configure-cluster

https://www.elastic.co/guide/en/elasticsearch/reference/5.3/indices-update-settings.html

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-red-cluster-status

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-red-cluster-status


Metric Description

Nodes The number of nodes in the Amazon ES cluster.

Relevant Statistics: Minimum, Maximum, Average

SearchableDocuments The total number of searchable documents across all indices in thecluster.

Relevant statistics: Minimum, Maximum, Average

DeletedDocuments The total number of deleted documents across all indices in thecluster.


CPUUtilization The maximum percentage of CPU resources used for data nodes in thecluster.

Relevant statistics: Maximum, Average

FreeStorageSpace The free space, in megabytes, for all data nodes in the cluster. AmazonES throws a ClusterBlockException when this metric reaches 0.To recover, you must either delete indices, add larger instances, or addEBS-based storage to existing instances. To learn more, see Recoveringfrom a Lack of Free Storage Space

NoteFreeStorageSpace will always be lower than the value thatthe Elasticsearch _cluster/stats API provides. Amazon ESreserves a percentage of the storage space on each instancefor internal operations.

Relevant statistics: Minimum

ClusterUsedSpace The total used space, in megabytes, for a cluster. You can view thismetric in the Amazon CloudWatch console, but not in the Amazon ESconsole.


ClusterIndexWritesBlockedIndicates whether your cluster is accepting or blocking incoming writerequests. A value of 0 means that the cluster is accepting requests. Avalue of 1 means that it is blocking requests.

Many factors can cause a cluster to begin blocking requests. Somecommon factors include the following: FreeStorageSpace is toolow, JVMMemoryPressure is too high, or CPUUtilization is toohigh. To alleviate this issue, consider adding more disk space or scalingyour cluster.

Relevant statistics: Maximum

NoteYou can view this metric in the Amazon CloudWatch console,but not the Amazon ES console.


http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-red-cluster-status-lack-of-free-space

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-red-cluster-status-lack-of-free-space


Metric Description

JVMMemoryPressure The maximum percentage of the Java heap used for all data nodes inthe cluster.


AutomatedSnapshotFailure The number of failed automated snapshots for the cluster. A value of1 indicates that no automated snapshot was taken for the domain inthe previous 36 hours.


CPUCreditBalance The remaining CPU credits available for data nodes in thecluster. A CPU credit provides the performance of a full CPUcore for one minute. For more information, see CPU Creditsin the Amazon EC2 Developer Guide. This metric is availableonly for the t2.micro.elasticsearch, t2.small.elasticsearch, andt2.medium.elasticsearch instance types.


KibanaHealthyNodes A health check for Kibana. A value of 1 indicates normal behavior.A value of 0 indicates that Kibana is inaccessible. In most cases, thehealth of Kibana mirrors the health of the cluster.


NoteYou can view this metric on the Amazon CloudWatch console,but not the Amazon ES console.

KMSKeyError A value of 1 indicates that the KMS customer master key used toencrypt data at rest has been disabled. To restore the domain tonormal operations, re-enable the key. The console displays this metriconly for domains that encrypt data at rest..


KMSKeyInaccessible A value of 1 indicates that the KMS customer master key used toencrypt data at rest has been deleted or revoked its grants to AmazonES. You can't recover domains that are in this state. If you have amanual snapshot, though, you can use it to migrate the domain's datato a new domain. The console displays this metric only for domainsthat encrypt data at rest.


The following screenshot shows the cluster metrics that are described in the preceding table.


http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html#t2-instances-cpu-credits

Amazon Elasticsearch Service Developer GuideDedicated Master Node Metrics

Dedicated Master Node MetricsThe AWS/ES namespace includes the following metrics for dedicated master nodes.

Metric Description

MasterCPUUtilization The maximum percentage of CPU resources used by the dedicatedmaster nodes. We recommend increasing the size of the instance typewhen this metric reaches 60 percent.

Relevant statistics: Average

MasterFreeStorageSpace This metric is not relevant and can be ignored. The service does notuse master nodes as data nodes.


Amazon Elasticsearch Service Developer GuideDedicated Master Node Metrics

Metric Description

MasterJVMMemoryPressure The maximum percentage of the Java heap used for all dedicatedmaster nodes in the cluster. We recommend moving to a largerinstance type when this metric reaches 85 percent.


MasterCPUCreditBalance The remaining CPU credits available for dedicated master nodesin the cluster. A CPU credit provides the performance of a full CPUcore for one minute. For more information, see CPU Credits in theAmazon EC2 User Guide for Linux Instances. This metric is availableonly for the t2.micro.elasticsearch, t2.small.elasticsearch, andt2.medium.elasticsearch instance types.


MasterReachableFromNode A health check for MasterNotDiscovered exceptions. A value of 1indicates normal behavior. A value of 0 indicates that /_cluster/health/ is failing.

Failures mean that the master node stopped or is not reachable.They are usually the result of a network connectivity issue or AWSdependency problem.


NoteYou can view this metric on the Amazon CloudWatch console,but not the Amazon ES console.

The following screenshot shows the dedicated master nodes metrics that are described in the precedingtable.


http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html#t2-instances-cpu-credits

Amazon Elasticsearch Service Developer GuideEBS Volume Metrics

EBS Volume MetricsThe AWS/ES namespace includes the following metrics for EBS volumes.

Metric Description

ReadLatency The latency, in seconds, for read operations on EBS volumes.


WriteLatency The latency, in seconds, for write operations on EBS volumes.


ReadThroughput The throughput, in bytes per second, for read operations on EBS volumes.


WriteThroughput The throughput, in bytes per second, for write operations on EBS volumes.


DiskQueueDepth The number of pending input and output (I/O) requests for an EBS volume.


ReadIOPS The number of input and output (I/O) operations per second for readoperations on EBS volumes.



Amazon Elasticsearch Service Developer GuideLogging Configuration Service Calls Using AWS CloudTrail

Metric Description

WriteIOPS The number of input and output (I/O) operations per second for writeoperations on EBS volumes.


The following screenshot shows the EBS volume metrics that are described in the preceding table.

Auditing Amazon Elasticsearch Service Domainswith AWS CloudTrail

Amazon Elasticsearch Service (Amazon ES) is integrated with AWS CloudTrail, a service that logs all AWSAPI calls made by, or on behalf of, your AWS account. The log files are delivered to an Amazon S3 bucketthat you create and configure with a bucket policy that grants CloudTrail permissions to write log files tothe bucket. CloudTrail captures all Amazon ES configuration service API calls, including those submittedby the Amazon Elasticsearch Service console.

You can use the information collected by CloudTrail to monitor activity for your search domains. Youcan determine the request that was made to Amazon ES, the source IP address from which the requestwas made, who made the request, and when it was made. To learn more about CloudTrail, including howto configure and enable it, see the AWS CloudTrail User Guide. To learn more about how to create andconfigure an S3 bucket for CloudTrail, see Amazon S3 Bucket Policy for CloudTrail.


http://docs.aws.amazon.com/awscloudtrail/latest/userguide/

https://docs.aws.amazon.com/awscloudtrail/latest/userguide/create-s3-bucket-policy-for-cloudtrail.html

Amazon Elasticsearch Service Developer GuideLogging Configuration Service Calls Using AWS CloudTrail

NoteCloudTrail logs events only for configuration-related API calls to Amazon Elasticsearch Service.Data-related APIs are not logged.

The following example shows a sample CloudTrail log for Amazon ES:

{ "Records": [ { "eventVersion": "1.03", "userIdentity": { "type": "Root", "principalId": "000000000000", "arn": "arn:aws:iam::000000000000:root", "accountId": "000000000000", "accessKeyId": "A*****************A" }, "eventTime": "2015-07-31T21:28:06Z", "eventSource": "es.amazonaws.com", "eventName": "CreateElasticsearchDomain", "awsRegion": "us-east-1", "sourceIPAddress": "Your IP", "userAgent": "es/test", "requestParameters": { "elasticsearchClusterConfig": {}, "snapshotOptions": { "automatedSnapshotStartHour": "0" }, "domainName": "your-domain-name", "eBSOptions": { "eBSEnabled": false } }, "responseElements": { "domainStatus": { "created": true, "processing": true, "aRN": "arn:aws:es:us-east-1:000000000000:domain/your-domain-name", "domainId": "000000000000/your-domain-name", "elasticsearchClusterConfig": { "zoneAwarenessEnabled": false, "instanceType": "m3.medium.elasticsearch", "dedicatedMasterEnabled": false, "instanceCount": 1 }, "deleted": false, "domainName": "your-domain-name", "domainVersion": "1.5", "accessPolicies": "", "advancedOptions": { "rest.action.multi.allow_explicit_index": "true" }, "snapshotOptions": { "automatedSnapshotStartHour": "0" }, "eBSOptions": { "eBSEnabled": false } } }, "requestID": "05dbfc84-37cb-11e5-a2cd-fbc77a4aae72", "eventID": "c21da94e-f5ed-41a4-8703-9a5f49e2ec85", "eventType": "AwsApiCall", "recipientAccountId": "000000000000" }


Amazon Elasticsearch Service Developer GuideAmazon ES Information in CloudTrail

]}

Amazon Elasticsearch Service Information inCloudTrailWhen CloudTrail logging is enabled in your AWS account, API calls made to Amazon Elasticsearch Service(Amazon ES) operations are tracked in log files. Amazon ES records are written together with other AWSservice records in a log file. CloudTrail determines when to create and write to a new file based on a timeperiod and file size.

All Amazon ES configuration service operations are logged. For example, callsto CreateElasticsearchDomain, DescribeElasticsearchDomain, andUpdateElasticsearchDomainConfig generate entries in the CloudTrail log files. Every log entrycontains information about who generated the request. The user identity information in the log helpsyou determine whether the request was made with root or IAM user credentials, with temporary securitycredentials for a role or federated user, or by another AWS service. For more information, see theuserIdentity field in the CloudTrail Event Reference.

You can store your log files in your bucket indefinitely, or you can define Amazon S3 lifecycle rules toarchive or delete log files automatically. By default, your log files are encrypted using Amazon S3 server-side encryption (SSE). You can choose to have CloudTrail publish Amazon SNS notifications when newlog files are delivered if you want to take quick action upon log file delivery. For more information, seeConfiguring Amazon SNS Notifications for CloudTrail. You also can aggregate Amazon ES log files frommultiple AWS Regions and multiple AWS accounts into a single Amazon S3 bucket. For more information,see Receiving CloudTrail Log Files from Multiple Regions.

Understanding Amazon Elasticsearch Service Log FileEntriesCloudTrail log files contain one or more log entries where each entry is made up of multiple JSON-formatted events. A log entry represents a single request from any source and includes informationabout the requested action, any parameters, the date and time of the action, and so on. The log entriesare not guaranteed to be in any particular order—they are not an ordered stack trace of the public APIcalls. CloudTrail log files include events for all AWS API calls for your AWS account, not just calls to theAmazon ES configuration service API. However, you can read the log files and scan for eventSourcees.amazonaws.com. The eventName element contains the name of the configuration service actionthat was called.

Tagging Amazon Elasticsearch Service DomainsYou can use Amazon ES tags to add metadata to your Amazon ES domains. AWS does not apply anysemantic meaning to your tags. Tags are interpreted strictly as character strings. All tags have thefollowing elements.

Tag Element Description

Tag key The tag key is the required name of the tag. Tag keys must be unique for the AmazonES domain to which they are attached. For a list of basic restrictions on tag keys andvalues, see User-Defined Tag Restrictions.


http://docs.aws.amazon.com/awscloudtrail/latest/userguide/event_reference_top_level.html

http://docs.aws.amazon.com/awscloudtrail/latest/userguide/getting_notifications_top_level.html

http://docs.aws.amazon.com/awscloudtrail/latest/userguide/aggregating_logs_top_level.html

http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html

Amazon Elasticsearch Service Developer GuideWorking with Tags (Console)

Tag Element Description

Tag value The tag value is an optional string value of the tag. Tag values can be null and do nothave to be unique in a tag set. For example, you can have a key-value pair in a tag setof project/Trinity and cost-center/Trinity. For a list of basic restrictions on tag keysand values, see User-Defined Tag Restrictions.

Each Amazon ES domain has a tag set, which contains all the tags that are assigned to that Amazon ESdomain. AWS does not automatically set any tags on Amazon ES domains. A tag set can contain up to 50tags, or it can be empty. If you add a tag to an Amazon ES domain that has the same key as an existingtag for a resource, the new value overwrites the old value.

You can use these tags to track costs by grouping expenses for similarly tagged resources. An AmazonES domain tag is a name-value pair that you define and associate with an Amazon ES domain. The nameis referred to as the key. You can use tags to assign arbitrary information to an Amazon ES domain. Atag key could be used, for example, to define a category, and the tag value could be an item in thatcategory. For example, you could define a tag key of “project” and a tag value of “Salix,” indicatingthat the Amazon ES domain is assigned to the Salix project. You could also use tags to designateAmazon ES domains as being used for test or production by using a key such as environment=test orenvironment=production. We recommend that you use a consistent set of tag keys to make it easier totrack metadata that is associated with Amazon ES domains.

You also can use tags to organize your AWS bill to reflect your own cost structure. To do this, sign up toget your AWS account bill with tag key values included. Then, organize your billing information accordingto resources with the same tag key values to see the cost of combined resources. For example, you cantag several Amazon ES domains with key-value pairs, and then organize your billing information to seethe total cost for each domain across several services. For more information, see Using Cost AllocationTags in the AWS Billing and Cost Management documentation.

NoteTags are cached for authorization purposes. Because of this, additions and updates to tags onAmazon ES domains might take several minutes before they are available.

Working with Tags (Console)Use the following procedure to create a resource tag.

To create a tag (console)

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. In the navigation pane, choose your Amazon ES domain.4. On the domain dashboard, choose Manage tags.5. In the Key column, type a tag key.6. (Optional) In the Value column, type a tag value.7. Choose Submit.

To delete a tag (console)

Use the following procedure to delete a resource tag.

1. Go to https://aws.amazon.com, and then choose Sign In to the Console.2. Under Analytics, choose Elasticsearch Service.3. In the navigation pane, choose your Amazon ES domain.


http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html

http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html

http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html



Amazon Elasticsearch Service Developer GuideWorking with Tags (AWS CLI)

4. On the domain dashboard, choose Manage tags.5. Next to the tag that you want to delete, choose Remove.6. Choose Submit.

For more information about using the console to work with tags, see Working with Tag Editor in the AWSManagement Console Getting Started Guide.

Working with Tags (AWS CLI)You can create resource tags using the AWS CLI with the --add-tags command.

Syntax

add-tags --arn=<domain_arn> --tag-list Key=<key>,Value=<value>

Parameter Description

--arn Amazon resource name for the Amazon ES domain to which the tag isattached.

--tag-list Set of space-separated key-value pairs in the following format:Key=<key>,Value=<value>

Example

The following example creates two tags for the logs domain:

aws es add-tags --arn arn:aws:es:us-east-1:379931976431:domain/logs --tag-list Key=service,Value=Elasticsearch Key=instances,Value=m3.2xlarge

You can remove tags from an Amazon ES domain using the remove-tags command.

Syntax

remove-tags --arn=<domain_arn> --tag-keys Key=<key>,Value=<value>


--arn Amazon Resource Name (ARN) for the Amazon ES domain to which thetag is attached.

--tag-keys Set of space-separated key-value pairs that you want to remove from theAmazon ES domain.

Example

The following example removes two tags from the logs domain that were created in the precedingexample:

aws es remove-tags --arn arn:aws:es:us-east-1:379931976431:domain/logs --tag-keys service instances

You can view the existing tags for an Amazon ES domain with the list-tags command:


https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/tag-editor.html

Amazon Elasticsearch Service Developer GuideWorking with Tags (AWS SDKs)

Syntax

list-tags --arn=<domain_arn>


--arn Amazon Resource Name (ARN) for the Amazon ES domain to which thetags are attached.

Example

The following example lists all resource tags for the logs domain:

aws es list-tags --arn arn:aws:es:us-east-1:379931976431:domain/logs

Working with Tags (AWS SDKs)The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the Amazon ESConfiguration API Reference (p. 124), including the AddTags, ListTags, and RemoveTags operations.For more information about installing and using the AWS SDKs, see AWS Software Development Kits.



Amazon Elasticsearch Service Developer GuideIntroduction to Indexing

Indexing Data in AmazonElasticsearch Service

Because Elasticsearch uses a REST API, numerous methods exist for indexing documents. You can usestandard clients like curl or any programming language that can send HTTP requests. To further simplifythe process of interacting with it, Elasticsearch has low-level clients for many programming languages.Advanced users can skip directly to the section called “Programmatic Indexing” (p. 60).

For situations in which new data arrives incrementally (for example, customer orders from a smallbusiness), you might use the _index API to index documents as they arrive. For situations in which theflow of data is less frequent (for example, weekly updates to a marketing website), you might preferto generate a file and send it to the _bulk API. For large numbers of documents, lumping requeststogether and using the _bulk API offers superior performance. If your documents are enormous,however, you might need to index them individually using the _index API.

For information about integrating data from other AWS services, see Loading Streaming Data intoAmazon ES (p. 66).

Introduction to IndexingBefore you can search data, you must index it. Indexing is the method by which search engines organizedata for fast retrieval. The resulting structure is called, fittingly, an index.

In Elasticsearch, the basic unit of data is a JSON document. Within an index, Elasticsearch organizesdocuments into types (arbitrary data categories that you define) and identifies them using a unique ID.

A request to the _index API looks like the following:

PUT elasticsearch_domain_endpoint/index/type/id -d 'document'

A request to the _bulk API looks a little different, because you specify the index, type, and ID in the bulkdata file:

POST elasticsearch_domain_endpoint/_bulk --data-binary @bulk_movies.json

Bulk data files must conform to a specific file format, which requires a newline character (\n) at the endof every line, including the last line. This is the basic format:

action_and_metadata\noptional_document\naction_and_metadata\noptional_document\n...

For a short sample file, see the section called “Step 2: Uploading Data for Indexing” (p. 10).

Elasticsearch features automatic index creation when you add a document to an index that doesn'talready exist. It also features automatic ID generation if you don't specify an ID in the request. Thissimple example automatically creates the movies index, establishes the document type of movie,indexes the document, and assigns it a unique ID:



https://www.elastic.co/guide/en/elasticsearch/client/index.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#_automatic_id_generation

Amazon Elasticsearch Service Developer GuideProgrammatic Indexing

curl -XPOST elasticsearch_domain_endpoint/movies/movie -d '{"title": "Spirited Away"}' -H 'Content-Type: application/json'

ImportantTo use automatic ID generation, you must use the POST method instead of PUT.

To verify that the document exists, you can perform the following search:

curl -XGET elasticsearch_domain_endpoint/movies/_search?pretty

The response should contain the following:

"hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "movies", "_type" : "movie", "_id" : "AV4WaTnYxBoJaZkSFeX9", "_score" : 1.0, "_source" : { "title" : "Spirited Away" } } ]}

Automatic ID generation has a clear downside: because the indexing code didn't specify a document ID,you can't easily update the document at a later time.

Indices default to five primary shards and one replica. If you want to specify non-default settings, createthe index before adding documents:

curl -XPUT elasticsearch_domain_endpoint/movies -d '{"settings": {"number_of_shards": 6, "number_of_replicas": 2}}' -H 'Content-Type: application/json'

NoteRequests using curl are unauthenticated and rely on an IP-based access policy. For examples ofsigned requests, see the section called “Programmatic Indexing” (p. 60).

Elasticsearch indices have the following naming restrictions:

• All letters must be lowercase.• Index names cannot begin with _ or -.• Index names cannot contain spaces, commas, ", *, +, /, \, |, ?, >, or <.

Programmatic IndexingThis section includes examples of how to use popular Elasticsearch clients to index documents.

PythonYou can install elasticsearch-py, the official Elasticsearch client for Python, using pip. Instead of theclient, you might prefer requests. The requests-aws4auth package simplifies the authentication process,but is not strictly required. From the terminal, run the following commands:


https://www.elastic.co/guide/en/elasticsearch/client/index.html

https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html

https://pypi.python.org/pypi/pip

http://docs.python-requests.org/

https://pypi.python.org/pypi/requests-aws4auth

Amazon Elasticsearch Service Developer GuidePython

pip install elasticsearchpip install requestspip install requests-aws4auth

The following sample code establishes a secure connection to the specified Amazon ES domain andindexes a single document using the _index API. You must provide values for AWS_ACCESS_KEY,AWS_SECRET_KEY, region, and host:

from elasticsearch import Elasticsearch, RequestsHttpConnectionfrom requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY = ''AWS_SECRET_KEY = ''region = '' # For example, us-east-1service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY, AWS_SECRET_KEY, region, service)

host = '' # For example, my-test-domain.us-east-1.es.amazonaws.com

es = Elasticsearch( hosts = [{'host': host, 'port': 443}], http_auth = awsauth, use_ssl = True, verify_certs = True, connection_class = RequestsHttpConnection)

document = { "title": "Moneyball", "director": "Bennett Miller", "year": "2011"}

es.index(index="movies", doc_type="movie", id="5", body=document)

print(es.get(index="movies", doc_type="movie", id="5"))

ImportantThese samples are for testing purposes. We do not recommend storing your AWS access key andAWS secret key directly in code.

If you don't want to use the elasticsearch.py client, you can just make standard HTTP requests. Thissample creates a new index with seven shards and two replicas:

import requestsfrom requests_aws4auth import AWS4Auth



host = '' # The domain with https:// and trailing slash. For example, https://my-test-domain.us-east-1.es.amazonaws.com/path = 'my-index' # the Elasticsearch API endpoint

url = host + path

# The JSON body to accompany the request (if necessary)


Amazon Elasticsearch Service Developer GuidePython

payload = { "settings" : { "number_of_shards" : 7, "number_of_replicas" : 2 }}

r = requests.put(url, auth=awsauth, json=payload) # requests.get, post, and delete have similar syntax

print(r.text)

This next example uses the Beautiful Soup library to help build a bulk file from a local directory of HTMLfiles. Using the same client as the first example, you can send the file to the _bulk API for indexing. Youcould use this code as the basis for adding search functionality to a website:

from bs4 import BeautifulSoupfrom elasticsearch import Elasticsearch, RequestsHttpConnectionfrom requests_aws4auth import AWS4Authimport globimport json

bulk_file = ''id = 1

# This loop iterates through all HTML files in the current directory and# indexes two things: the contents of the first h1 tag and all other text.

for html_file in glob.glob('*.htm'):

with open(html_file) as f: soup = BeautifulSoup(f, 'html.parser')

title = soup.h1.string body = soup.get_text(" ", strip=True) # If get_text() is too noisy, you can do further processing on the string.

index = { 'title': title, 'body': body, 'link': html_file } # If running this script on a website, you probably need to prepend the URL and path to html_file.

# The action_and_metadata portion of the bulk file bulk_file += '{ "index" : { "_index" : "site", "_type" : "page", "_id" : "' + str(id) + '" } }\n'

# The optional_document portion of the bulk file bulk_file += json.dumps(index) + '\n'

id += 1



host = '' # For example, my-test-domain.us-east-1.es.amazonaws.com

es = Elasticsearch( hosts = [{'host': host, 'port': 443}], http_auth = awsauth, use_ssl = True, verify_certs = True,


https://www.crummy.com/software/BeautifulSoup/bs4/doc/

Amazon Elasticsearch Service Developer GuideJava

connection_class = RequestsHttpConnection)

es.bulk(bulk_file)

print(es.search(q='some test query'))

JavaThis first example uses the Elasticsearch Java REST Client, which you must configure as a dependency.The request is unauthenticated and relies on an IP-based access policy. You must provide a value forhost:

import java.io.IOException;import java.util.Collections;

import org.elasticsearch.client.Response;import org.elasticsearch.client.RestClient;import org.elasticsearch.client.http.entity.ContentType;import org.elasticsearch.client.http.HttpEntity;import org.elasticsearch.client.http.HttpHost;import org.elasticsearch.client.http.nio.entity.NStringEntity;

public class JavaRestClientExample {

public static void main(String[] args) throws IOException {

String host = ""; // For example, my-test-domain.us-east-1.es.amazonaws.com String index = "movies"; String type = "movie"; String id = "6";

String json = "{" + "\"title\":\"Walk the Line\"," + "\"director\":\"James Mangold\"," + "\"year\":\"2005\"" + "}";

RestClient client = RestClient.builder(new HttpHost(host, 443, "https")).build();

HttpEntity entity = new NStringEntity(json, ContentType.APPLICATION_JSON);

Response response = client.performRequest("PUT", "/" + index + "/" + type + "/" + id, Collections.<String, String>emptyMap(), entity);

System.out.println(response.toString()); }}

The easiest way of sending a signed request is to use the AWS Request Signing Interceptor. Therepository contains examples to help you get started. You must change the region string inSample.java and the Amazon ES endpoint in AmazonElasticsearchServiceSample.java.

A more complex way of sending a signed request to Amazon ES is to use the AWS SDK for Java. Note thetwo helper classes for response and error handling. You must provide values for host and region:

import java.io.ByteArrayInputStream;import java.io.IOException;import java.net.URI;import java.net.URISyntaxException;

import com.amazonaws.ClientConfiguration;


https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html

https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_maven_repository.html

https://github.com/awslabs/aws-request-signing-apache-interceptor

https://aws.amazon.com/sdk-for-java/


import com.amazonaws.DefaultRequest;import com.amazonaws.Request;import com.amazonaws.auth.AWS4Signer;import com.amazonaws.auth.AWSCredentials;import com.amazonaws.auth.AWSCredentialsProvider;import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;import com.amazonaws.http.AmazonHttpClient;import com.amazonaws.http.ExecutionContext;import com.amazonaws.http.HttpMethodName;

public class JavaClientExampleWithAuth {

public static void main(String[] args) throws IOException, URISyntaxException {

// Intentionally verbose. String prefix = "https://"; String host = ""; // For example, my-test-domain.us-east-1.es.amazonaws.com String index = "movies"; String type = "movie"; String id = "6"; String endpoint = prefix + host + "/" + index + "/" + type + "/" + id;

String service = "es"; String region = ""; // For example, us-east-1

// Libraries like Jackson simplify the conversion of objects to JSON. Here, we // just use a string.

String json = "{" + "\"title\":\"Walk the Line\"," + "\"director\":\"James Mangold\"," + "\"year\":\"2005\"" + "}";

// Builds the request. We need an AWS service, URI, HTTP method, and request // body (in this case, JSON).

Request<?> request = new DefaultRequest<Void>(service); request.setEndpoint(new URI(endpoint)); request.setHttpMethod(HttpMethodName.PUT); request.setContent(new ByteArrayInputStream(json.getBytes()));

// Retrieves our credentials from the computer. For more information on where // this class looks for credentials, see // http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html.

AWSCredentialsProvider credsProvider = new DefaultAWSCredentialsProviderChain(); AWSCredentials creds = credsProvider.getCredentials();

// Signs the request using our region, service, and credentials. AWS4Signer // modifies the original request rather than returning a new request.

AWS4Signer signer = new AWS4Signer(); signer.setRegionName(region); signer.setServiceName(service); signer.sign(request, creds); request.addHeader("Content-Type", "application/json");

// Creates and configures the HTTP client, creates the error and response // handlers, and finally executes the request.

ClientConfiguration config = new ClientConfiguration(); AmazonHttpClient client = new AmazonHttpClient(config); ExecutionContext context = new ExecutionContext(true); MyErrorHandler errorHandler = new MyErrorHandler(); MyHttpResponseHandler<Void> responseHandler = new MyHttpResponseHandler<Void>();



client.requestExecutionBuilder().executionContext(context).errorResponseHandler(errorHandler).request(request) .execute(responseHandler); }}

import com.amazonaws.AmazonWebServiceResponse;import com.amazonaws.http.HttpResponseHandler;

public class MyHttpResponseHandler<T> implements HttpResponseHandler<AmazonWebServiceResponse<T>> {

@Override public AmazonWebServiceResponse<T> handle(com.amazonaws.http.HttpResponse response) throws Exception { AmazonWebServiceResponse<T> awsResponse = new AmazonWebServiceResponse<T>(); return awsResponse; }

@Override public boolean needsConnectionLeftOpen() { return false; }}

import com.amazonaws.AmazonServiceException;import com.amazonaws.http.HttpResponseHandler;

public class MyErrorHandler implements HttpResponseHandler<AmazonServiceException> {

@Override public AmazonServiceException handle(com.amazonaws.http.HttpResponse response) throws Exception { AmazonServiceException ase = new AmazonServiceException(""); ase.setStatusCode(response.getStatusCode()); ase.setErrorCode(response.getStatusText()); return ase; }

@Override public boolean needsConnectionLeftOpen() { return false; }}


Amazon Elasticsearch Service Developer GuideLoading Streaming Data into Amazon ES from Amazon S3

Loading Streaming Data intoAmazon Elasticsearch Service

You can load streaming data into your Amazon ES domain from Amazon S3 buckets, Amazon Kinesisstreams, Amazon DynamoDB Streams, and Amazon CloudWatch metrics. For example, to load streamingdata from Amazon S3 and Amazon Kinesis, you use a Lambda function as an event handler in the AWSCloud. The Lambda function responds to new data by processing it and streaming the data to yourdomain.

Topics• Loading Streaming Data into Amazon ES from Amazon S3 (p. 66)• Loading Streaming Data into Amazon ES from Amazon Kinesis (p. 70)• Loading Streaming Data into Amazon ES from Amazon Kinesis Firehose (p. 73)• Loading Streaming Data into Amazon ES from Amazon DynamoDB (p. 73)• Loading Streaming Data into Amazon ES from Amazon CloudWatch (p. 73)• Loading Data into Amazon ES from AWS IoT (p. 73)

Streaming data provides fresh data for search and analytic queries. Amazon S3 pushes eventnotifications to AWS Lambda. For more information, see Using AWS Lambda with Amazon S3 in the AWSLambda Developer Guide. Amazon Kinesis requires AWS Lambda to poll for, or pull, event notifications.For more information, see Using AWS Lambda with Amazon Kinesis.

You should be familiar with these service integrations before attempting to use them to load streamingdata into your Amazon ES domain. For more information about these services, see the following AWSdocumentation:

• AWS Lambda Developer Guide• Amazon S3 Developer Guide• Amazon Kinesis Developer Guide• Amazon DynamoDB Developer Guide• Amazon CloudWatch User Guide• AWS IoT Developer Guide

NoteAWS Lambda is available in limited regions. For more information, see the list of AWS Lambdaregions in the AWS General Reference.

Loading Streaming Data into Amazon ES fromAmazon S3

You can integrate your Amazon ES domain with Amazon S3 and AWS Lambda. Any new data sent toan S3 bucket triggers an event notification to Lambda, which then runs your custom Java or Node.jsapplication code. After your application processes the data, it streams the data to your domain. At a highlevel, setting up to load streaming data to Amazon ES requires the following steps:

1. Creating a Lambda deployment package (p. 67)


http://aws.amazon.com/streaming-data/

http://docs.aws.amazon.com/lambda/latest/dg/with-s3.html

http://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html

http://docs.aws.amazon.com/lambda/latest/dg/welcome.html

http://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html

http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatch.html

http://docs.aws.amazon.com/iot/latest/developerguide/

http://docs.aws.amazon.com/general/latest/gr/rande.html#lambda_region

http://docs.aws.amazon.com/general/latest/gr/rande.html#lambda_region

Amazon Elasticsearch Service Developer GuideSetting Up to Load Streaming Datainto Amazon ES from Amazon S3

2. Configuring a Lambda function (p. 68)3. Granting authorization to add data to your Amazon ES domain (p. 69)

You also must create an Amazon S3 bucket and an Amazon ES domain. Setting up this integration pathhas the following prerequisites.

Prerequisite Description

Amazon S3 Bucket The event source that triggers your Lambda function. For moreinformation, see Create a Bucket in the Amazon Simple Storage ServiceGetting Started Guide. The bucket must reside in the same AWS Region asyour Amazon ES domain.

Amazon ES Domain The destination for data after it is processed by the application code inyour Lambda function. For more information, see Creating Amazon ESDomains (p. 13).

Lambda Function The Java or Node.js application code that runs when S3 pushes an eventnotification to Lambda. Amazon ES provides a sample application inNode.js, s3_lambda_es.js, that you can download to get started. See theLambda sample code for Amazon ES.

Lambda DeploymentPackage

A .zip file that consists of your Java or Node.js application code and anydependencies. For information about the required folder hierarchy, seeCreating a Lambda Deployment Package (p. 67). For information aboutcreating specific Lambda deployment packages, see Creating a DeploymentPackage (Node.js) and Creating a Deployment Package (Java).

Amazon ESAuthorization

An IAM access policy that permits Lambda to add data to your domain.Attach the policy to the Amazon S3 execution role that you create as partof your Lambda function. For details, see Granting Authorization to AddData to Your Amazon ES Domain (p. 69).

Setting Up to Load Streaming Data into Amazon ESfrom Amazon S3This section provides additional details about setting up the prerequisites for loading streaming data intoAmazon ES from Amazon S3. After you finish configuring the integration, data streams automatically toyour Amazon ES domain whenever new data is added to your Amazon S3 bucket.

Creating a Lambda Deployment PackageCreate a .zip file that contains your Lambda application code and any dependencies.

To create a deployment package:

1. Create a directory structure like the following:

eslambda \node_modules

This example uses eslambda for the name of the top-level folder, but you can use any name.However, the subfolder must be named node_modules.

2. Place your application source code in the eslambda folder.


http://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html

https://github.com/awslabs/amazon-elasticsearch-lambda-samples

http://docs.aws.amazon.com/lambda/latest/dg/nodejs-create-deployment-pkg.html


http://docs.aws.amazon.com/lambda/latest/dg/lambda-java-how-to-create-deployment-package.html


3. Add or edit the following four global variables:

• endpoint, the Amazon ES domain endpoint.

• region, the AWS Region in which you created your Amazon ES domain.

• index, the name of the Amazon ES index to use for data that is streamed from Amazon S3.

• doctype, the Amazon ES document type of the streamed data. For more information, see MappingTypes in the Elasticsearch documentation.

The following example from s3_lambda_es.js configures the sample application to use thestreaming-logs domain endpoint in the us-east-1 AWS Region:

/* Globals */var esDomain = { endpoint: 'search-streaming-logs-okga24ftzsbz2a2hzhsqw73jpy.us-east-1.es.example.com', region: 'us-east-1', index: 'streaming-logs', doctype: 'apache'};

4. Install any dependencies that are required by your application.

For example, if you use Node.js, you must execute the following command for each requirestatement in your application code:

npm install <dependency>

5. Verify that all runtime dependencies that are required by your application code are located in thenode_modules folder.

6. Execute the following command to package the application code and dependencies:

zip -r eslambda.zip *

The name of the zip file must match the top-level folder.

For more information about creating Lambda deployment packages, see Creating a Deployment Package(Node.js) and Creating a Deployment Package (Java).

Configuring a Lambda FunctionUse AWS Lambda to create and configure your Lambda function. To do that, you can use either the AWSCLI or the AWS Lambda console. For a tutorial about creating and configuring a Lambda function usingthe AWS CLI, see Using AWS Lambda with Amazon S3. For configuration settings on the AWS Lambdaconsole, see the following table.

NoteFor more information about creating and configuring a Lambda function, see the AWS LambdaDeveloper Guide.

FunctionConfiguration

Description

IAM Execution Role The name of the IAM role that is used to execute actions on Amazon S3. Whilecreating your Lambda function, the Lambda console automatically opens theIAM console to help you create the execution role. Later, you also must attachan IAM access policy to this role that permits Lambda to add data to your


https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#all-mapping-types





http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html




FunctionConfiguration

Description

domain. For details, see Granting Authorization to Add Data to Your Amazon ESDomain (p. 69).

Event Source Specifies the S3 bucket as the event source for the Lambda function. Forinstructions, see the Using AWS Lambda with Amazon S3 tutorial. AWS Lambdaautomatically adds the necessary permissions for Amazon S3 to invoke yourLambda function from this event source. Optionally, specify a file suffix to filterwhat kinds of files, such as .log, trigger the Lambda function.

Handler The name of the file that contains the application source code, but with the.handler file suffix. For example, if your application source code residesin a file named s3_lambda_es.js, you must configure the handler ass3_lambda_es.handler. For more information, see Getting Started in theAWS Lambda Developer Guide. Amazon ES provides a sample application inNode.js that you can download to get started: Lambda Sample Code for AmazonES.

Timeout The length of time that Lambda should wait before canceling an invocationrequest. The default value of three seconds is too short for the Amazon ES usecase. We recommend configuring your timeout for 10 seconds.

For more function configuration details, see Configuring a Lambda Function (p. 68) in this guide. Forgeneral information, see Lambda Functions in the AWS Lambda Developer Guide.

Granting Authorization to Add Data to Your Amazon ES Domain

When you choose S3 Execution Role as the IAM role to execute actions on S3, Lambda opens theIAM console and helps you to create a new execution role. Lambda automatically adds the necessarypermissions to invoke your Lambda function from this event source. After you create the role, open itin the IAM console and attach the following IAM access policy to the role. This grants permissions toLambda to stream data to Amazon ES:

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "es:*" ], "Effect": "Allow", "Resource": "arn:aws:es:us-west-2:123456789012:domain/streaming-logs/*" } ]}

For more information about attaching IAM access policies to roles, see Tutorial: Create and Attach YourFirst Customer Managed Policy in the IAM User Guide.


http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

http://docs.aws.amazon.com/lambda/latest/dg/getting-started.html



http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction-function.html

http://docs.aws.amazon.com/IAM/latest/UserGuide/walkthru_managed-policies.html


Amazon Elasticsearch Service Developer GuideLoading Streaming Data into

Amazon ES from Amazon Kinesis

Loading Streaming Data into Amazon ES fromAmazon Kinesis

You can load streaming data from Amazon Kinesis to Amazon ES. This integration relies on AWS Lambdaas an event handler in the cloud. Amazon Kinesis requires Lambda to poll your Amazon Kinesis streamto determine whether it has new data that will automatically invoke your Lambda function. After yourLambda function finishes processing any new data, it streams the data to your Amazon ES domain.

At a high level, setting up to streaming data to Amazon ES requires the following steps:

1. Creating a Lambda deployment package (p. 71).

2. Configuring a Lambda function (p. 71).

3. Granting authorization to add data to your Amazon ES domain (p. 72).

You also must create an Amazon Kinesis stream and an Amazon ES domain. Setting up this integrationpath has the following prerequisites.


Amazon Kinesis Stream The event source for your Lambda function. For instructions aboutcreating Amazon Kinesis streams, see Amazon Kinesis Streams.

Elasticsearch Domain The destination for data after it is processed by the application code inyour Lambda function. For more information, see Creating Amazon ESDomains (p. 13) in this guide.

Lambda Function The Java or Node.js application code that runs when Amazon Kinesispushes an event notification to Lambda. Amazon ES provides asample application in Node.js, kinesis_lambda_es.js, that you candownload to get started: Lambda Sample Code for Amazon ES.

Lambda DeploymentPackage

A .zip file that consists of your Java or Node.js application code andany dependencies. For information about the required folder hierarchy,see Creating a Lambda Deployment Package (p. 67). For generalinformation about creating Lambda deployment packages, see Creatinga Deployment Package (Node.js) and Creating a Deployment Package(Java).

Amazon ES Authorization An IAM access policy that permits Lambda to add data to your domain.Attach the policy to the Amazon Kinesis execution role that you createas part of your Lambda function. For details, see Granting Authorizationto Add Data to Your Amazon ES Domain (p. 69).

Setting Up to Load Streaming Data into Amazon ESfrom Amazon KinesisThis section provides more details about setting up the prerequisites for loading streaming data fromAmazon Kinesis into Amazon ES. After you finish configuring the integration, Lambda automaticallystreams data to your Amazon ES domain whenever new data is added to your Amazon Kinesis stream.


https://docs.aws.amazon.com/kinesis/latest/dev/amazon-kinesis-streams.html






Amazon Elasticsearch Service Developer GuideSetting Up to Load Streaming Data

into Amazon ES from Amazon Kinesis

Creating a Lambda Deployment PackageCreate a .zip file that contains your Lambda application code and any dependencies.

To create a deployment package:

1. Create a directory structure like the following:

eslambda \node_modules

You can use any name for the top-level folder rather than eslambda. However, you must name thesubfolder node_modules.

2. Place your application source code in the eslambda folder.3. Add or edit the following global variables in your sample application:

• endpoint, the Amazon ES domain endpoint.• region, the AWS Region in which you created your Amazon ES domain.• index, the name of the Amazon ES index to use for data that is streamed from Amazon Kinesis.• doctype, the Amazon ES document type of the streamed data. For more information, see Mapping

Types in the Elasticsearch documentation.

The following example from kinesis_lambda_es.js configures the sample application to use thestreaming-logs Amazon ES domain endpoint in the us-east-1 AWS Region.

/* Globals */var esDomain = { endpoint: 'search-streaming-logs-okga24ftzsbz2a2hzhsqw73jpy.us-east-1.es.example.com', region: 'us-east-1', index: 'streaming-logs', doctype: 'apache'};

4. Install any dependencies that are required by your application.

For example, if you use Node.js, you must execute the following command for each requirestatement in your application code:

npm install <dependency>

5. Verify that all runtime dependencies that are required by your application code are located in thenode_modules folder.

6. Execute the following command to package the application code and dependencies:

zip -r eslambda.zip *

The name of the zip file must match the top-level folder.

For more information about creating Lambda deployment packages, see Creating a Deployment Package(Node.js) and Creating a Deployment Package (Java).

Configuring a Lambda FunctionUse AWS Lambda to create and configure your Lambda function. To do that, you can use either theAWS CLI or the Creating a Deployment Package (Node.js) console. For a tutorial about creating and








Amazon Elasticsearch Service Developer GuideSetting Up to Load Streaming Data

into Amazon ES from Amazon Kinesis

configuring a Lambda function using the AWS CLI, see Using AWS Lambda with Amazon Kinesis. Forconfiguration settings on the AWS Lambda console, see the following table.

NoteFor more information about creating and configuring a Lambda function, see the GettingStarted tutorial in the AWS Lambda Developer Guide.

Configuration Description

Amazon Kinesisstream

The event source of your Lambda function. For instructions, see Amazon KinesisStreams.

IAM execution role The name of the IAM role that is used to execute actions on Amazon Kinesis.While configuring your Lambda function, the Lambda console automaticallyopens the IAM console to help you create the execution role. Later, you alsomust attach an IAM access policy to this role that permits Lambda to send datato your Amazon ES domain. For details, see Granting Authorization to Add Datato Your Amazon ES Domain (p. 72).

Handler The name of the file that contains the application source code, but withthe .handler file suffix. For example, if your application source code is ina file named kinesis_lambda_es.js, you must configure the handler askinesis_lambda_es.handler. For more information, see Lambda FunctionHandler. Amazon ES provides a sample application in Node.js that you candownload to get started: Lambda Sample Code for Amazon ES.

Timeout The length of time that Lambda should wait before canceling an invocationrequest. The default value of three seconds is too short for this use case. Werecommend configuring your timeout for 10 seconds.

For more information, see Lambda Functions in the AWS Lambda Developer Guide.

Granting Authorization to Add Data to Your Amazon ES Domain

When you choose Kinesis Execution Role as the IAM role to execute actions on Amazon Kinesis, Lambdaopens the IAM console and requires you to create a new execution role. Lambda automatically adds thenecessary permissions to invoke your Lambda function from this event source. After you create the role,open it in the IAM console and attach the following IAM access policy to the role so that Lambda haspermissions to stream data to Amazon ES:

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "es:*" ], "Effect": "Allow", "Resource": "arn:aws:es:us-west-2:123456789012:domain/streaming-logs/*" } ]}

For more information about attaching IAM access policies to roles, see Tutorial: Create and Attach YourFirst Customer Managed Policy in the IAM User Guide.


http://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html

http://docs.aws.amazon.com/lambda/latest/dg/java-gs.html#java-gs-create-lambda-function

http://docs.aws.amazon.com/lambda/latest/dg/java-gs.html#java-gs-create-lambda-function

http://docs.aws.amazon.com/kinesis/latest/dev/amazon-kinesis-streams.html

http://docs.aws.amazon.com/kinesis/latest/dev/amazon-kinesis-streams.html

http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html

http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html


http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction-function.html



Amazon Elasticsearch Service Developer GuideLoading Streaming Data into Amazon

ES from Amazon Kinesis Firehose

Loading Streaming Data into Amazon ES fromAmazon Kinesis Firehose

Amazon Kinesis supports Amazon ES a delivery destination. For instructions on how to load streamingdata into Amazon ES, see Creating an Amazon Kinesis Firehose Delivery Stream and Choose Amazon ESfor your destination in the Amazon Kinesis Firehose Developer Guide.

You might need to perform transforms on your data before loading it into Amazon ES. To learn moreabout using Lambda functions perform this task, see Data Transformation in the Amazon Kinesis FirehoseDeveloper Guide.

As you configure a delivery stream, Kinesis Firehose features a "one click" IAM role that gives it theresource access it needs to send data to Amazon ES, back up data on Amazon S3, and transform datausing Lambda. Because of the complexity involved in creating such a role manually, we recommend usingthe provided role.

Loading Streaming Data into Amazon ES fromAmazon DynamoDB

You can load streaming data from Amazon DynamoDB Streams to your Amazon ES domain. To do that,use the Logstash input plugin for Amazon DynamoDB and the logstash-output-amazon-es plugin, whichsigns and exports Logstash events to Amazon ES.

Loading Streaming Data into Amazon ES fromAmazon CloudWatch

You can load streaming data from CloudWatch Logs to your Amazon ES domain by using a CloudWatchLogs subscription. For information about Amazon CloudWatch subscriptions, see Real-time Processingof Log Data with Subscriptions. For configuration information, see Streaming CloudWatch Logs Data toAmazon Elasticsearch Service in the Amazon CloudWatch Developer Guide.

Loading Data into Amazon ES from AWS IoTYou can send data from AWS IoT using rules. To learn more, see Amazon ES Action in the AWS IoTDeveloper Guide.


http://docs.aws.amazon.com/firehose/latest/dev/basic-create.html

http://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-elasticsearch

http://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-elasticsearch

http://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html

https://github.com/awslabs/logstash-input-dynamodb

https://github.com/awslabs/logstash-output-amazon_es

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Subscriptions.html

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Subscriptions.html

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CWL_ES_Stream.html

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CWL_ES_Stream.html

http://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html

http://docs.aws.amazon.com/iot/latest/developerguide/elasticsearch-rule.html

Amazon Elasticsearch Service Developer GuideManual Snapshot Prerequisites

Working with Amazon ElasticsearchService Index Snapshots

Snapshots are backups of a cluster's data and state. They provide a convenient way to migrate dataacross Amazon ES domains and recover from failure. The service supports restoring from snapshotstaken on both Amazon ES domains and self-managed Elasticsearch clusters.

Amazon ES takes daily automated snapshots of the primary index shards in a domain, as describedin the section called “Configuring Automatic Snapshots” (p. 25). It stores these automated snapshotsin a preconfigured Amazon S3 bucket for 14 days at no additional charge to you. You can use thesesnapshots to restore the domain.

You cannot, however, use automated snapshots to migrate to new domains. Automated snapshots areread-only from within a given domain. For migrations, you must use manual snapshots stored in yourown repository (an S3 bucket). Standard S3 charges apply to manual snapshots.

TipSome users find tools like the Curator CLI convenient for index and snapshot management.The Curator CLI offers advanced filtering functionality that can help simplify tasks on complexclusters.

Topics• Manual Snapshot Prerequisites (p. 74)• Registering a Manual Snapshot Directory (p. 76)• Taking Manual Snapshots (p. 77)• Restoring Snapshots (p. 78)

Manual Snapshot PrerequisitesTo create and restore index snapshots manually, you must work with IAM and Amazon S3. Verify that youhave met the following prerequisites before you attempt to take a snapshot.


S3 bucket Stores manual snapshots for your Amazon ES domain.

IAM role Delegates permissions to Amazon Elasticsearch Service. The trust relationship for therole must specify Amazon Elasticsearch Service in the Principal statement. The IAMrole also is required to register your snapshot repository with Amazon ES. Only IAMusers with access to this role may register the snapshot repository.

IAM policy Specifies the actions that Amazon S3 may perform with your S3 bucket. The policymust be attached to the IAM role that delegates permissions to Amazon ElasticsearchService. The policy must specify an S3 bucket in a Resource statement.

S3 Bucket

You need an S3 bucket to store manual snapshots. Make a note of its Amazon Resource Name (ARN). Youneed it for the following:

• Resource statement of the IAM policy that is attached to your IAM role


https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

Amazon Elasticsearch Service Developer GuideManual Snapshot Prerequisites

• Python client that is used to register a snapshot repository

The following example shows an ARN for an S3 bucket:

arn:aws:s3:::es-index-backups

For more information, see Create a Bucket in the Amazon S3 Getting Started Guide.

IAM Role

You must have a role that specifies Amazon Elasticsearch Service, es.amazonaws.com, in a Servicestatement in its trust relationship, as shown in the following example:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "es.amazonaws.com" }, "Action": "sts:AssumeRole" } ]}

When you create an AWS service role using the IAM console, Amazon ES is not included in the Selectrole type list. However, you can still create the role by choosing Amazon EC2, following the stepsto create the role, and then editing the role's trust relationships to es.amazonaws.com instead ofec2.amazonaws.com. For instructions and additional information, see Creating a Role for an AWSService and Modifying a Role in the IAM User Guide.

NoteOnly IAM users or roles with access to this service role may register snapshotrepositories (p. 76). A common way to provide access is to attach the following policy to theuser or role:

{ "Version": "2012-10-17", "Statement": { "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::123456789012:role/TheServiceRole" }}

IAM Policy

You must attach an IAM policy to the IAM role. The policy specifies the S3 bucket that is used to storemanual snapshots for your Amazon ES domain. The following example specifies the ARN of the es-index-backups bucket:

{ "Version":"2012-10-17", "Statement":[ { "Action":[ "s3:ListBucket" ],


http://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html#roles-creatingrole-service-console

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html#roles-creatingrole-service-console

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_manage_modify.html#roles-managingrole-editing-console

Amazon Elasticsearch Service Developer GuideRegistering a Manual Snapshot Directory

"Effect":"Allow", "Resource":[ "arn:aws:s3:::es-index-backups" ] }, { "Action":[ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Effect":"Allow", "Resource":[ "arn:aws:s3:::es-index-backups/*" ] } ]}

For more information, see Creating Customer Managed Policies and Attaching Managed Policies in theIAM User Guide.

Registering a Manual Snapshot DirectoryYou must register the snapshot directory with Amazon Elasticsearch Service before you can take manualindex snapshots. This one-time operation requires that you sign your AWS request with credentials forone of the users or roles specified in the IAM role's trust relationship, as described in the section called“Manual Snapshot Prerequisites” (p. 74).

NoteYou can't use curl to perform this operation because it doesn't support AWS request signing.Instead, use the sample Python client to register your snapshot directory.If your domain resides within a VPC, your computer must be able to access the VPC in orderfor the Python client to successfully register the snapshot directory. Accessing a VPC variesby network configuration, but likely involves connecting to a VPN or corporate network.To check that you can reach the Amazon ES domain, navigate to https://your-vpc-domain.region.es.amazonaws.com in a web browser and verify that you receive the defaultJSON response.

Sample Python Client

Save the following sample Python code as a Python file, such as snapshot.py. Registering a snapshotdirectory is a one-time operation, but to migrate from one domain to another, you need to performthese steps on the old domain and the new domain.

You must update the following in your code:

region

AWS Region where you created the snapshot repositoryhost

Endpoint for your Amazon ES domainaws_access_key_id

IAM credentialaws_secret_access_key

IAM credential


http://docs.aws.amazon.com/IAM/latest/UserGuide/policies_using-managed.html#create-managed-policy-console

http://docs.aws.amazon.com/IAM/latest/UserGuide/policies_using-managed.html#attach-managed-policy-console

Amazon Elasticsearch Service Developer GuideTaking Manual Snapshots

path

Name of the snapshot repository

data

Must include the name of the S3 bucket and the ARN for the IAM role that you created in thesection called “Manual Snapshot Prerequisites” (p. 74). To enable server-side encryption withS3-managed keys for the snapshot repository, add "server_side_encryption": true to the"settings" JSON.

NoteThis sample Python client requires that you install version 2.x of the boto package on thecomputer where you register your snapshot repository.

from boto.connection import AWSAuthConnection

class ESConnection(AWSAuthConnection):

def __init__(self, region, **kwargs): super(ESConnection, self).__init__(**kwargs) self._set_auth_region_name(region) self._set_auth_service_name("es")

def _required_auth_capability(self): return ['hmac-v4']

if __name__ == "__main__":

client = ESConnection( region='us-west-1', host='search-weblogs-etrt4mbbu254nsfupy6oiytuz4.us-west-1.es.example.com', aws_access_key_id='my-access-key-id', aws_secret_access_key='my-access-key', is_secure=False)

print 'Registering Snapshot Repository' resp = client.make_request(method='PUT', path='/_snapshot/weblogs-index-backups', data='{"type": "s3","settings": { "bucket": "es-index-backups","region": "us-west-1","role_arn": "arn:aws:iam::123456789012:role/TheServiceRole"}}', headers={'Content-Type': 'application/json'}) body = resp.read() print body

ImportantIf the S3 bucket is in the us-east-1 region, you need to use "endpoint":"s3.amazonaws.com" in place of "region": "us-east-1".

Taking Manual SnapshotsYou specify two pieces of information when you create a snapshot:

• Name of your snapshot repository

• Name for the snapshot

The following example requests use curl, a common HTTP client, for convenience. If your access policiesspecify IAM users or roles, however, snapshot requests must be signed. Clients like curl can't perform


http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html


https://github.com/boto/boto#installation


Amazon Elasticsearch Service Developer GuideRestoring Snapshots

this signing. For sample code that demonstrates how to make signed requests, see the section called“Programmatic Indexing” (p. 60).

To manually take a snapshot

• Run the following command to manually take a snapshot:

curl -XPUT 'elasticsearch-domain-endpoint/_snapshot/repository/snapshot_name'

The following example takes a snapshot named snapshot_1 and stores it in the weblogs-index-backups snapshot repository:

curl -XPUT 'elasticsearch-domain-endpoint/_snapshot/weblogs-index-backups/snapshot_1'

NoteThe time required to take a snapshot increases with the size of the Amazon ES domain.Long-running snapshot operations commonly encounter the following error: 504GATEWAY_TIMEOUT. Typically, you can ignore these errors and wait for the operation tocomplete successfully. Use the following command to verify the state of all snapshots of yourdomain:

curl -XGET 'elasticsearch-domain-endpoint/_snapshot/repository/_all?pretty'

For more information about the options available to you when taking a snapshot, see Snapshot andRestore in the Elasticsearch documentation.

Restoring SnapshotsTo restore a snapshot, perform the following procedure:

1. Identify the snapshot that you want to restore. To see all snapshot repositories, run the followingcommand:

curl -XGET 'elasticsearch-domain-endpoint/_snapshot?pretty'

After you identify the repository, you run the following command to see all snapshots:

curl -XGET 'elasticsearch-domain-endpoint/_snapshot/repository/_all?pretty'

NoteMost automated snapshots are stored in the cs-automated repository. If your domainencrypts data at rest, they are stored in the cs-automated-enc repository. If you don'tsee the manual snapshot repository that you're looking for, make sure that you registeredit (p. 76) to the domain.

2. Delete or rename all open indices in the Amazon ES domain.

You can't restore a snapshot of your indices to an Elasticsearch cluster that already contains indiceswith the same names. Currently, Amazon ES does not support the Elasticsearch _close API, so youmust use one of the following alternatives:

• Delete the indices on the same Amazon ES domain, then restore the snapshot.• Restore the snapshot to a different Amazon ES domain (only possible with manual snapshots).


https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#_snapshot

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#_snapshot

Amazon Elasticsearch Service Developer GuideRestoring Snapshots

The following example shows how to delete all existing indices for a domain:

curl -XDELETE 'elasticsearch-domain-endpoint/_all'

If you don't plan to restore all indices, though, you might want to delete only one:

curl -XDELETE 'elasticsearch-domain-endpoint/index-name'

3. To restore a snapshot, run the following command:

curl -XPOST 'elasticsearch-domain-endpoint/_snapshot/repository/snapshot/_restore'

Due to special permissions on the .kibana index, attempts to restore all indices might fail,especially if you try to restore from an automated snapshot. The following example restores just oneindex, my-index, from 2017-snapshot in the cs-automated snapshot repository:

curl -XPOST 'elasticsearch-domain-endpoint/_snapshot/cs-automated/2017-snapshot/_restore' -d '{"indices": "my-index"}' -H 'Content-Type: application/json'

For more information about restoring only certain indices from a snapshot, see Snapshot and Restore inthe Elasticsearch documentation.

NoteIf not all primary shards were available for the indices involved, a snapshot might have a stateof PARTIAL. This value indicates that data from at least one shard was not stored successfully.You can still restore from a partial snapshot, but you might need to use older snapshots torestore any missing indices.


https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore


Migrating to a Different ElasticsearchVersion

The following table shows how to migrate your data to a newer Elasticsearch version. Most of the stepsrequire you to create and restore manual index snapshots. To learn more about this process, see Workingwith Index Snapshots (p. 74).

Migrating to a newer Elasticsearch version also requires creating a new domain. To learn more, seeCreating and Configuring Amazon ES Domains (p. 13).

From Version To Version Migration Process

5.x 6.0 1. Review Breaking changes in 6.0 to see if you need to makeadjustments to your indices or application.

ImportantIndices created in version 6.0 no longer supportmultiple mapping types. Indices created inversion 5.x still support multiple mapping typeswhen restored into a 6.0 cluster.If you use services like Amazon Kinesis Firehoseor Amazon CloudWatch Logs to stream data toAmazon ES, check that these services supportElasticsearch 6.0 before migrating.If you use AWS Lambda, check that your codeonly creates a single mapping type per index.

2. Create a manual snapshot of the 5.x domain.3. Create a new 6.0 domain.4. Restore the snapshot from the original domain to the 6.0

domain.5. If you no longer need your original domain, delete it.

Otherwise, you continue to incur charges for the domain.

2.3 6.0 Elasticsearch 2.3 snapshots are not compatible with 6.0.To migrate your data directly from 2.3 to 6.0, you mustmanually recreate your indices in the new domain.

Alternately, you can follow the 2.3 to 5.x steps in this table,perform _reindex operations in the new 5.x domain toconvert your 2.3 indices to 5.x indices, and then follow the5.x to 6.0 steps.

5.1 or 5.3 5.5 1. Create a manual snapshot of the 5.1 or 5.3 domain.2. Create a new 5.5 domain.3. Restore the snapshot from the original domain to the 5.5

domain.4. If you no longer need your original domain, delete it.



https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.0.html

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs-reindex.html


From Version To Version Migration Process

2.3 5.x 1. Review Breaking changes in 5.0 in the Elasticsearchdocumentation to find out if you need to makeadjustments to your indexing or application.

NoteThe Elasticsearch migration plugin currently isnot available.

2. Create a manual snapshot of the 2.3 domain.3. Create a new 5.x domain.4. Restore the snapshot from the 2.3 domain to the 5.x

domain.5. If you no longer need your 2.3 domain, delete it.


1.5 5.x Elasticsearch 1.5 snapshots are not compatible with 5.x.To migrate your data from 1.5 to 5.x, you must manuallyrecreate your indices in the new domain.

Important1.5 snapshots are compatible with 2.3, but AmazonES 2.3 domains do not support the _reindexoperation. Because you cannot reindex them,indices that originated in a 1.5 domain still fail torestore from 2.3 snapshots to 5.x domains.

1.5 2.3 1. Use the _plugin/migration Elasticsearch plugin to findout if you can directly upgrade to version 2.3. You mightneed to make changes to your data before migration.a. In a web browser, open http://domain_endpoint/

_plugin/migration/.b. Choose Run checks now.c. Review the results, and if needed, follow the

instructions to make changes to your data. For moreinformation, see Elasticsearch breaking changes.

2. Create a manual snapshot of the 1.5 domain.3. Create a new 2.3 domain.4. Restore the snapshot from the 1.5 domain to the 2.3

domain.5. If you no longer need your 1.5 domain, delete it.



https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking-changes-5.0.html

https://www.elastic.co/guide/en/elasticsearch/reference/2.3/docs-reindex.html

https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking-changes.html

Amazon Elasticsearch Service Developer GuideKibana

Kibana and Logstash

This chapter describes some considerations for using Kibana and Logstash with Amazon ES.

Topics

• Kibana (p. 82)

• Loading Bulk Data with the Logstash Plugin (p. 85)

KibanaKibana is a popular open source visualization tool designed to work with Elasticsearch. AmazonElasticsearch Service (Amazon ES) provides a default installation of Kibana with every Amazon ESdomain. You can find a link to Kibana on your domain dashboard on the Amazon ES console. The URL ishttps://domain.region.es.amazonaws.com/_plugin/kibana/. Queries on this default Kibanainstallation have a 60-second timeout.

For information about using Kibana to visualize your data, see the Kibana User Guide.

NoteTo prevent public access to Kibana, you must configure an IP-based access policy. The defaultinstallation of Kibana does not support IAM user authentication at this time. To learn moreabout IP-based access policies, see Configuring Access Policies (p. 23).

The following sections address some common Kibana use cases:

• the section called “Using a Proxy to Access Amazon ES from Kibana” (p. 82)

• the section called “Configuring Kibana to Use a WMS Map Server” (p. 84)

• the section called “Connecting a Local Kibana Server to Amazon ES” (p. 84)

Using a Proxy to Access Amazon ES from KibanaBecause Kibana is a JavaScript application, requests originate from the user's IP address.Unauthenticated, IP-based access control might be impractical due to the sheer number of IP addressesyou would need to whitelist in order for each user to have access to Kibana. One workaround is to placea proxy server between Kibana and Amazon ES. Then you can add an IP-based access policy that allowsrequests from only one IP address, the proxy's. The following diagram shows this configuration.


https://www.elastic.co/guide/en/kibana/current/introduction.html

https://www.elastic.co/guide/en/kibana/current/index.html

Amazon Elasticsearch Service Developer GuideUsing a Proxy to Access Amazon ES from Kibana

1. This is your Amazon ES domain. IAM provides authorized access to this domain. An additional, IP-based access policy provides access to the proxy server.

2. This is the proxy server, residing in a VPC subnet and running on an Amazon EC2 instance.3. Other applications running on EC2 instances can use the Signature Version 4 signing process to send

authenticated requests to Amazon ES. For sample code that makes signed requests to Amazon ES, seethe section called “Programmatic Indexing” (p. 60).

4. Kibana clients connect to your Amazon ES domain through the proxy.

To enable this sort of configuration, you need a resource-based policy that specifies roles and IPaddresses. Here's a sample policy:

{ "Version": "2012-10-17", "Statement": [ { "Resource": "arn:aws:es:us-west-2:111111111111:domain/recipes1/analytics", "Principal": { "AWS": "arn:aws:iam::111111111111:role/allowedrole1" }, "Action": [ "es:ESHttpGet" ],


Amazon Elasticsearch Service Developer GuideConfiguring Kibana to Use a WMS Map Server

"Effect": "Allow" }, { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "es:*", "Condition": { "IpAddress": { "aws:SourceIp": [ "123.456.789.123" ] } }, "Resource": "arn:aws:es:us-west-2:111111111111:domain/recipes1/analytics" } ]}

We recommend that you configure the EC2 instance running the proxy server with an Elastic IP address.This way, you can replace the instance when necessary and still attach the same public IP address to it. Tolearn more, see Elastic IP Addresses in the Amazon EC2 User Guide for Linux Instances.

Configuring Kibana to Use a WMS Map ServerDue to licensing restrictions, the default installation of Kibana on Amazon ES domains that useElasticsearch 5.x or greater does not include a map server for tile map visualizations. Use the followingprocedure to configure Kibana to use a Web Map Service (WMS) map server.

To configure Kibana to use a WMS map server:

1. Open Kibana. You can find a link to Kibana in the domain summary at https://console.aws.amazon.com/es/.

2. Choose Management.3. Choose Advanced Settings.4. Locate visualization:tileMap:WMSdefaults, and then choose the edit button to modify the default

value.5. Change enabled to true and url to the URL of a valid WMS map server.6. (Optional) Locate visualization:tileMap:WMSdefaults, and then choose the edit button to modify

the default value.7. (Optional) Change "layers": "0" to a comma-separated list of map layers that you want to

display. Layers vary by map service. The default value of 0 is often appropriate.8. Choose the save button.

To apply the new default value to visualizations, you might need to reload Kibana.

NoteMap services often have licensing fees or restrictions. You are responsible for all suchconsiderations on any map server that you specify. You might find the map services from theU.S. Geological Survey useful for testing.

Connecting a Local Kibana Server to Amazon ESIf you have invested significant time into configuring your own Kibana instance, you can use it instead of(or in addition to) the default Kibana instance that Amazon ES provides.


http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html



https://viewer.nationalmap.gov/services/

Amazon Elasticsearch Service Developer GuideLoading Bulk Data with the Logstash Plugin

To connect a local Kibana server to Amazon ES:

• Make the following changes to config/kibana.yml:

kibana_index: ".kibana-5"elasticsearch_url: "http://elasticsearch_domain_endpoint:80"

You must use the http prefix and explicitly specify port 80.

Loading Bulk Data with the Logstash PluginLogstash provides a convenient way to use the bulk API to upload data into your Amazon ES domainwith the S3 plugin. The service also supports all other standard Logstash input plugins that are providedby Elasticsearch. Amazon ES also supports two Logstash output plugins: the standard Elasticsearchplugin and the logstash-output-amazon-es plugin, which signs and exports Logstash events to AmazonES.

You must install your own local instance of Logstash and make the following changes in the Logstashconfiguration file to enable interaction with Amazon ES.

Configuration Field Input | OutputPlugin

Description

bucket Input Specifies the Amazon S3 bucket containing the datathat you want to load into an Amazon ES domain.You can find this service endpoint in the AmazonElasticsearch Service console dashboard.

region Input Specifies the AWS Region where the Amazon S3 bucketresides.

hosts Output Specifies the service endpoint for the target AmazonES domain.

ssl Output Specifies whether to use SSL to connect to Amazon ES.

flush_size Output By default, Logstash fills a buffer with 5,000 eventsbefore sending the entire batch onward. However, ifyour documents are large, approaching 100 MB in size,we recommend configuring flush_size option toa larger value to prevent the buffer from filling tooquickly.

If you increase flush_size, we recommend alsosetting the Logstash LS_HEAP_SIZE environmentvariable to 2048 MB to prevent running out ofmemory.

For more information, see flush_size in theElasticsearch documentation.

This example configures Logstash to do the following:

• Point the output plugin to an Amazon ES endpoint



https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html


https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-flush_size

Amazon Elasticsearch Service Developer GuideLoading Bulk Data with the Logstash Plugin

• Point to the input plugin to the wikipedia-stats-log bucket in S3• Use SSL to connect to Amazon ES

input{ s3 { bucket => "wikipedia-stats-log" access_key_id => "lizards" secret_access_key => "lollipops" region => "us-east-1" }}output{ elasticsearch { hosts => "search-logs-demo0-cpxczkdpi4bkb4c44g3csyln5a.us-east-1.es.example.com" ssl => true flush_size => 250000 }}

The following example demonstrates the same configuration, but connects to Amazon ES without SSL:

input{ s3 { bucket => "wikipedia-stats-log" access_key_id => "lizards" secret_access_key => "lollipops" region => "us-east-1" }}output{ elasticsearch { hosts => "search-logs-demo0-cpxczkdpi4bkb4c44g3csyln5a.us-east-1.es.example.com" ssl => false flush_size => 250000 }}

NoteThe service request in the preceding example must be signed. For more information aboutsigning requests, see the section called “Signing Amazon ES Requests” (p. 35). Use the logstash-output-amazon-es output plugin to sign and export Logstash events to Amazon ES. Forinstructions, see README.md.




https://github.com/awslabs/logstash-output-amazon_es/blob/master/README.md

Amazon Elasticsearch Service Developer GuideSizing Amazon ES Domains

Amazon Elasticsearch Service BestPractices

These topics address some considerations for operating Amazon Elasticsearch Service domains andprovide general guidelines that apply to many use cases.

Topics• Sizing Amazon ES Domains (p. 87)• Dedicated Master Nodes (p. 90)• Recommended CloudWatch Alarms (p. 92)

Sizing Amazon ES DomainsNo surefire method of sizing Amazon ES domains exists, but by starting with an understanding of yourstorage needs, the service, and Elasticsearch itself, you can make an educated initial estimate on yourhardware needs. This estimate can serve as a useful starting point for the most critical aspect of sizingdomains: testing them with representative workloads and monitoring their performance.

Topics• Calculating Storage Requirements (p. 87)• Choosing the Number of Shards (p. 88)• Choosing Instance Types and Testing (p. 89)

Calculating Storage RequirementsMost Elasticsearch workloads fall into one of two broad categories:

• Long-lived index: You write code that processes data into one or more Elasticsearch indices and thenupdates those indices periodically as the source data changes. Some common examples are website,document, and e-commerce search.

• Rolling indices: Data continuously flows into a set of temporary indices, with an indexing periodand retention window, such as a set of daily indices that is retained for two weeks. Some commonexamples are log analytics, time-series processing, and clickstream analytics.

For long-lived index workloads, you can examine the source data on disk and easily determine how muchstorage space it consumes. If the data comes from multiple sources, just add those sources together.

For rolling indices, you can multiply the amount of data generated during a representative time periodby the retention period. For example, if you generate 200 MB of log data per hour, that's 4.8 GB per day,which is 67 GB of data at any given time if you have a two-week retention period.

The size of your source data, however, is just one aspect of your storage requirements. You also have toconsider the following:

1. Number of replicas: Each replica is a full copy of an index and needs the same amount of disk space.By default, each Elasticsearch index has one replica. We recommend at least one to prevent againstdata loss. Replicas also improve search performance, so you might want more if you have a read-heavyworkload.


https://www.elastic.co/guide/en/elasticsearch/reference/5.6/_basic_concepts.html#getting-started-shards-and-replicas

Amazon Elasticsearch Service Developer GuideChoosing the Number of Shards

2. Elasticsearch indexing overhead: The on-disk size of an index varies, but is often 10% larger than thesource data. After indexing your data, you can use the _cat/indices API and pri.store.sizevalue to calculate the exact overhead. The _cat/allocation API also provides a useful summary.

3. Operating system reserved space: By default, Linux reserves 5% of the file system for the root userfor critical processes, system recovery, and to safeguard against disk fragmentation problems.

4. Elasticsearch watermarks: Elasticsearch has a default "low watermark" of 85%, meaning that whendisk usage exceeds 85%, Elasticsearch no longer allocates shards to that node. Elasticsearch also hasa default "high watermark" of 90%, at which point it attempts to relocate shards to other nodes. If nonodes have enough storage space to accommodate shard relocation, basic write operations like addingdocuments and creating indices might begin to fail.

5. Amazon ES overhead: Amazon ES reserves 20% of the storage space of each instance (up to 20 GB) forsegment merges, logs, and other internal operations.

Because of this 20 GB maximum, the total amount of reserved space can vary dramaticallydepending on the number of instances in your domain. For example, a domain might havethree m4.xlarge.elasticsearch instances, each with 500 GB of storage space, for a totalof 1.5 TB. In this case, the total reserved space is only 60 GB. Another domain might have 10m3.medium.elasticsearch instances, each with 100 GB of storage space, for a total of 1 TB. Here,the total reserved space is 200 GB, even though the first domain is 50% larger.

Amazon ES overhead does not stack with the Elasticsearch watermarks. You need to consider onlythe larger of the two. In the first example, the Amazon ES overhead is only 60 / 1500 = 4%, so youneed to be aware of the Elasticsearch watermarks. In the second example, the overhead is 200 / 1000= 20%, so you shouldn't encounter watermark issues. In the following formula, we apply a "worst-case" estimate for overhead that is accurate for domains with less than 100 GB of storage space perinstance and over-allocates for larger instances.

In summary, if you have 67 GB of data at any given time and want one replica, your minimum storagerequirement is closer to 67 * 2 * 1.1 / 0.95 / 0.8 = 194 GB. You can generalize this calculation as follows:

Source Data * (1 + Number of Replicas) * (1 + Indexing Overhead) / (1 - Linux Reserved Space) / (1 -Amazon ES Overhead) = Minimum Storage Requirement

Or you can use this simplified version:

Source Data * (1 + Number of Replicas) * 1.45 = Minimum Storage Requirement

Insufficient storage space is one of the most common causes of cluster instability, so you should cross-check the numbers when you choose instance types, instance counts, and storage volumes (p. 89).

NoteIf your minimum storage requirement exceeds 1 PB, see Petabyte Scale (p. 94).

Choosing the Number of ShardsAfter you understand your storage requirements, you can investigate your indexing strategy. EachElasticsearch index is split into some number of shards. Because you can't change the number of primaryshards for an existing index, you should decide about shard count before indexing your first document.

The overarching goal of choosing a number of shards is to distribute an index evenly across all datanodes in the cluster. However, these shards shouldn't be too large or too numerous. A good rule ofthumb is to try to keep shard size between 10–50 GB. Large shards can make it difficult for Elasticsearchto recover from failure, but because each shard uses some amount of CPU and memory, having too manysmall shards can cause performance issues and out of memory errors. In other words, shards should besmall enough that the underlying Amazon ES instance can handle them, but not so small that they placeneedless strain on the hardware.


https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html#cat-indices

https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-allocation.html

Amazon Elasticsearch Service Developer GuideChoosing Instance Types and Testing

For example, suppose you have 67 GB of data. You don't expect that number to increase over time,and you want to keep your shards around 30 GB each. Your number of shards therefore should beapproximately 67 * 1.1 / 30 = 3. You can generalize this calculation as follows:

(Source Data + Room to Grow) * (1 + Indexing Overhead) / Desired Shard Size = Approximate Numberof Primary Shards

This equation helps compensate for growth over time. If you expect those same 67 GB of data toquadruple over the next year, the approximate number of shards is (67 + 201) * 1.1 / 30 = 10. Remember,though, you don't have those extra 201 GB of data yet. Check to make sure this preparation for thefuture doesn't create unnecessarily tiny shards that consume huge amounts of CPU and memory in thepresent. In this case, 67 * 1.1 / 10 shards = 7.4 GB per shard, which will consume extra resources and isbelow the recommended size range. You might consider the more middle-of-the-road approach of sixshards, which leaves you with 12 GB shards today and 49 GB shards in the future. Then again, you mightprefer to start with three 30 GB shards and reindex your data when the shards exceed 50 GB.

NoteBy default, Elasticsearch indices are split into five primary shards. You can specify differentsettings when you create an index (p. 59).

Choosing Instance Types and TestingAfter you calculate your storage requirements and choose the number of shards that you need, you canstart to make hardware decisions. Hardware requirements vary dramatically by workload, but we can stilloffer some basic recommendations.

In general, the storage limits (p. 167) for each instance type map to the amount of CPU and memoryyou might need for light workloads. For example, an m4.large.elasticsearch instance has amaximum EBS volume size of 512 GB, 2 vCPU cores, and 8 GB of memory. If your cluster has manyshards, performs taxing aggregations, updates documents frequently, or processes a large number ofqueries, those resources might be insufficient for your needs. If you believe your cluster falls into one ofthese categories, try starting with a configuration closer to 2 vCPU cores and 8 GB of memory for every100 GB of your storage requirement.

TipFor a summary of the hardware resources that are allocated to each instance type, see AmazonElasticsearch Service Pricing.

Still, even those resources might be insufficient. Some Elasticsearch users report that they need manytimes those resources to fulfill their requirements. Finding the right hardware for your workload meansmaking an educated initial estimate, testing with representative workloads, adjusting, and testing again:

1. To start, we recommend a minimum of three instances to avoid potential Elasticsearch issues, suchas the split brain issue. If you have three dedicated master nodes (p. 90), we still recommend aminimum of two data nodes for replication.

2. If you have a 184 GB storage requirement and the recommended minimum number of three instances,you use the equation 184 / 3 = 61 GB to find the amount of storage that each instance needs. In thisexample, you might select three m3.medium.elasticsearch instances for your cluster, each usinga 90 GB EBS storage volume so that you have a safety net and some room for growth over time. Thisconfiguration provides 3 vCPU cores and 12 GB of memory, so it's suited to lighter workloads.

For a more substantial example, consider a 14 TB storage requirement and a heavy workload. In thiscase, you might choose to begin testing with 2 * 140 = 280 vCPU cores and 8 * 140 = 1120 GB ofmemory. These numbers work out to approximately 18 m4.4xlarge.elasticsearch instances,each using a 900 GB EBS storage volume.

3. After configuring the cluster, you can add your index (p. 59), perform some representative clienttesting using a realistic dataset, and monitor CloudWatch metrics (p. 46) to see how the clusterhandles the workload.





Amazon Elasticsearch Service Developer GuideDedicated Master Nodes

4. If performance satisfies your needs, tests succeed, and CloudWatch metrics are normal, the cluster isready to use. Remember to set CloudWatch alarms (p. 92) to detect unhealthy resource usage.

If performance isn't acceptable, tests fail, or CPUUtilization or JVMMemoryPressure are high, youmight need to choose a different instance type (or add instances) and continue testing. As you addinstances, Elasticsearch automatically rebalances the distribution of shards throughout the cluster.

Because it is easier to measure the excess capacity in an overpowered cluster than the deficit in anunderpowered one, we recommend starting with a larger cluster than you think you need, testing, andscaling down to an efficient cluster that has the resources to ensure stable operations.

Production clusters or clusters with complex states benefit from dedicated master nodes (p. 90),which improve performance and cluster reliability.

Dedicated Master NodesAmazon Elasticsearch Service uses dedicated master nodes to increase cluster stability. A dedicatedmaster node performs cluster management tasks, but does not hold data or respond to data uploadrequests. This offloading of cluster management tasks increases the stability of your domain.

We recommend that you allocate three dedicated master nodes for each production Amazon ES domain:

1. One dedicated master node means that you have no backup in the event of a failure.

2. Two dedicated master nodes means that your cluster does not have the necessary quorum of nodes toelect a new master node in the event of a failure.

A quorum is Number of Dedicated Master Nodes / 2 + 1 (rounded down to the nearest whole number),which Amazon ES sets to discovery.zen.minimum_master_nodes when you create your domain.

In this case, 2 / 2 + 1 = 2. Because one dedicated master node has failed and only one backup exists,the cluster does not have a quorum and cannot elect a new master.

3. Three dedicated master nodes, the recommended number, provides two backup nodes in the event ofa master node failure and the necessary quorum (2) to elect a new master.

4. Four dedicated master nodes is no better than three and can cause issues if you use zoneawareness (p. 45).

• If one master node fails, you have the quorum (3) to elect a new master. If two nodes fail, you losethat quorum, just as you do with three dedicated master nodes.

• If each Availability Zone has two dedicated master nodes and the zones are unable to communicatewith each other, neither zone has the quorum to elect a new master.

5. Having five dedicated master nodes works as well as three and allows you to lose two nodes whilemaintaining a quorum, but because only one dedicated master node is active at any given time, thisconfiguration means paying for four idle nodes. Many customers find this level of failover protectionexcessive.

Dedicated master nodes perform the following cluster management tasks:

• Track all nodes in the cluster

• Track the number of indices in the cluster

• Track the number of shards belonging to each index

• Maintain routing information for nodes in the cluster

• Update the cluster state after state changes, such as creating an index and adding or removing nodesin the cluster


Amazon Elasticsearch Service Developer GuideDedicated Master Nodes

• Replicate changes to the cluster state across all nodes in the cluster• Monitor the health of all cluster nodes by sending heartbeat signals, periodic signals that monitor the

availability of the data nodes in the cluster

The following illustration shows an Amazon ES domain with ten instances. Seven of the instances aredata nodes and three are dedicated master nodes. Only one of the dedicated master nodes is active; thetwo gray dedicated master nodes wait as backup in case the active dedicated master node fails. All dataupload requests are served by the seven data nodes, and all cluster management tasks are offloaded tothe active dedicated master node.

Although dedicated master nodes do not process search and query requests, their size is highlycorrelated with the number of instances, indices, and shards that they can manage. For productionclusters, we recommend the following instance types for dedicated master nodes. Theserecommendations are based on typical workloads and can vary based on your needs.

Instance Count Recommended Minimum Dedicated MasterInstance Type

5–10 m3.medium.elasticsearch

10–20 m4.large.elasticsearch

20–50 c4.xlarge.elasticsearch


Amazon Elasticsearch Service Developer GuideRecommended CloudWatch Alarms

Instance Count Recommended Minimum Dedicated MasterInstance Type

50–100 c4.2xlarge.elasticsearch

• For recommendations on dedicated master nodes for large clusters, see Petabyte Scale (p. 94).• For information about how certain configuration changes can affect dedicated master nodes, see the

section called “About Configuration Changes” (p. 44).• For clarification on instance count limits, see the section called “Cluster and Instance Limits” (p. 167).• For more information about specific instance types, including vCPU, memory, and pricing, see Amazon

Elasticsearch Instance Prices.

Recommended CloudWatch AlarmsCloudWatch alarms perform an action when a CloudWatch metric exceeds a specified value for someamount of time. For example, you might want AWS to email you if your cluster health status is red forlonger than one minute. This section includes some recommended alarms and how to respond to them.

For more information about setting alarms, see Creating Amazon CloudWatch Alarms in the AmazonCloudWatch User Guide.

Alarm Issue

ClusterStatus.redmaximum is >= 1 for 1minute, 1 consecutivetime

At least one primary shard and its replicas are not allocated to a node. Seethe section called “Red Cluster Status” (p. 105).

ClusterStatus.yellowmaximum is >= 1 for 1minute, 1 consecutivetime

At least one replica shard is not allocated to a node. See the section called“Yellow Cluster Status” (p. 107).

FreeStorageSpaceminimum is <= 30000for 1 minute, 1consecutive time

Your cluster is down to 30 GB of free storage space. See the section called“Lack of Available Storage Space” (p. 107). This value is in MB, so ratherthan 30000, we recommend setting it to 25% of your total storage.

ClusterIndexWritesBlockedis >= 1 for 1 minute, 1consecutive time

Your cluster is blocking write requests. See the section called“ClusterBlockException” (p. 107).

Nodes minimum is < xfor 1 day, 1 consecutivetime

x is the number of nodes in your cluster. This alarm indicates that at leastone node in your cluster has been unreachable for one day. See the sectioncalled “Failed Cluster Nodes” (p. 105).

AutomatedSnapshotFailuremaximum is >= 1 for 1minute, 1 consecutivetime

An automated snapshot failed. This failure is often the result of a redcluster health status. See the section called “Red Cluster Status” (p. 105).

For a summary of all automated snapshots and some information aboutfailures, you can also try the following:

GET domain_endpoint/_snapshot/cs-automated/_all




http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html

Amazon Elasticsearch Service Developer GuideRecommended CloudWatch Alarms

Alarm Issue

CPUUtilizationaverage is >= 80%for 15 minutes, 3consecutive times

100% CPU utilization isn't uncommon, but sustained high averages areproblematic. Consider using larger instance types or adding instances.

JVMMemoryPressuremaximum is >= 80%for 15 minutes, 1consecutive time

The cluster could encounter out of memory errors if usage increases.Consider scaling vertically. Amazon ES uses half of an instance's RAM forthe Java heap, up to a heap size of 32 GB. You can scale instances verticallyup to 64 GB of RAM, at which point you can scale horizontally by addinginstances.

MasterCPUUtilizationaverage is >= 50%for 15 minutes, 3consecutive times

MasterJVMMemoryPressureis >= 80% for 15minutes, 1 consecutivetime

Consider using larger instance types for your dedicated masternodes (p. 90). Because of their role in cluster stability and blue/greendeployments (p. 44), dedicated master nodes should have lower averageCPU usage than data nodes.

KMSKeyError is >=1 for 1 minute, 1consecutive time

The KMS encryption key that is used to encrypt data at rest in your domainis disabled. Re-enable it to restore normal operations. To learn more, seeEncryption at Rest (p. 103).

KMSKeyInaccessibleis >= 1 for 1 minute, 1consecutive time

The KMS encryption key that is used to encrypt data at rest in your domainhas been deleted or has revoked its grants to Amazon ES. You can't recoverdomains that are in this state, but if you have a manual snapshot, youcan use it to migrate to a new domain. To learn more, see Encryption atRest (p. 103).

NoteIf you just want to view metrics, see Monitoring CloudWatch Metrics (p. 46).



Petabyte Scale for AmazonElasticsearch Service

Amazon Elasticsearch Service offers domain storage of up to 1.5 PB. You can configure a domainwith 100 i3.16xlarge.elasticsearch instance types, each with 15 TB of storage. Becauseof the sheer difference in scale, recommendations for domains of this size differ from our generalrecommendations (p. 87). This section discusses considerations for creating domains, costs, storage,shard size, and dedicated master nodes. Despite frequent references to the i3 instance types, the shardsize and dedicated master node recommendations in this section apply to any domain approachingpetabyte scale.

Creating domains

Domains of this size exceed the default limit of 20 instances per domain. To request a service limitincrease of up to 100 instances per domain, open a case at the AWS Support Center.

Pricing

Before creating a domain of this size, check the Amazon Elasticsearch Service Pricing page to ensurethat the associated costs match your expectations.

Storage

The i3 instance types are specifically designed to provide fast, local non-volatile memory express(NVMe) storage. Because this local storage tends to offer considerable performance benefits whencompared to Amazon Elastic Block Store, EBS volumes are not an option when you select theseinstance types in Amazon ES.

Shard size

A common Elasticsearch guideline is not to exceed 50 GB per shard. Given the number of shardsnecessary to accommodate a 1.5 PB storage requirement, we recommend a shard size of at least 100GB.

For example, if you have 450 TB of source data and want one replica, your minimum storagerequirement is closer to 450 TB * 2 * 1.1 / 0.95 / 0.85 = 1.23 PB. For an explanation of thiscalculation, see the section called “Calculating Storage Requirements” (p. 87). Although 1.23 PB / 15TB = 82 instances, you might select 90 or more i3.16xlarge.elasticsearch instances to giveyourself a storage safety net and account for some variance in the amount of data over time.

To calculate the number of primary shards, use this formula: 450,000 GB * 1.1 / 150 GB per shard= 3,300 shards. As always, the most important step of sizing and configuring your domain is toperform representative client testing using a realistic data set.

Dedicated master nodes

We recommend that you allocate three dedicated master nodes to each production AmazonES domain. Rather than our usual guidelines for dedicated master nodes (p. 90), however, werecommend more powerful instance types for domains of this size. The following table showsrecommended instance types for dedicated master nodes for large domains.

Instance Count Recommended Minimum Dedicated Master Instance Type

5–10 m4.large.elasticsearch





Instance Count Recommended Minimum Dedicated Master Instance Type

10–20 c4.xlarge.elasticsearch





VPC Support for AmazonElasticsearch Service Domains

A virtual private cloud (VPC) is a virtual network that is dedicated to your AWS account. It's logicallyisolated from other virtual networks in the AWS Cloud. You can launch AWS resources, such as AmazonES domains, into your VPC.

Placing an Amazon ES domain within a VPC enables secure communication between Amazon ES andother services within the VPC without the need for an internet gateway, NAT device, or VPN connection.All traffic remains securely within the AWS Cloud. Because of their logical isolation, domains that residewithin a VPC have an extra layer of security when compared to domains that use public endpoints.

To support VPCs, Amazon ES places an endpoint into either one or two subnets of your VPC. A subnetis a range of IP addresses in your VPC. If you enable zone awareness (p. 45) for your domain, AmazonES places an endpoint into two subnets. The subnets must be in different Availability Zones in the sameregion. If you don't enable zone awareness, Amazon ES places an endpoint into only one subnet.

The following illustration shows the VPC architecture if zone awareness is not enabled.

The following illustration shows the VPC architecture if zone awareness is enabled.



Amazon ES also places an elastic network interface (ENI) in the VPC for each of your data nodes. AmazonES assigns each ENI a private IP address from the IPv4 address range of your subnet. The service alsoassigns a public DNS hostname (which is the domain endpoint) for the IP addresses. You must use apublic DNS service to resolve the endpoint (which is a DNS hostname) to the appropriate IP addresses forthe data nodes:

• If your VPC uses the Amazon-provided DNS server by setting the enableDnsSupport option to true(the default value), resolution for the Amazon ES endpoint will succeed.

• If your VPC uses a private DNS server and the server can reach the public authoritative DNS servers toresolve DNS hostnames, resolution for the Amazon ES endpoint will also succeed.

Because the IP addresses might change, you should resolve the domain endpoint periodically so that youcan always access the correct data nodes. We recommend that you set the DNS resolution interval to oneminute. If you’re using a client, you should also ensure that the DNS cache in the client is cleared.

NoteAmazon ES doesn't support IPv6 addresses with a VPC. You can use a VPC that has IPv6 enabled,but the domain will use IPv4 addresses.

Topics• Limitations (p. 98)• About Access Policies on VPC Domains (p. 98)• Before You Begin: Prerequisites for VPC Access (p. 99)• Creating a VPC (p. 99)


Amazon Elasticsearch Service Developer GuideLimitations

• Reserving IP Addresses in a VPC Subnet (p. 100)

• Service-Linked Role for VPC Access (p. 101)

• Migrating from Public Access to VPC Access (p. 101)

• Amazon VPC Documentation (p. 102)

LimitationsCurrently, operating an Amazon ES domain within a VPC has the following limitations:

• You can either launch your domain within a VPC or use a public endpoint, but you can't do both. Youmust choose one or the other when you create your domain.

• If you launch a new domain within a VPC, you can't later switch it to use a public endpoint. The reverseis also true: If you create a domain with a public endpoint, you can't later place it within a VPC. Instead,you must create a new domain and migrate your data.

• You can't launch your domain within a VPC that uses dedicated tenancy. You must use a VPC withtenancy set to Default.

• After you place a domain within a VPC, you can't move it to a different VPC. However, you can changethe subnets and security group settings.

• Compared to public domains, VPC domains display less information in the Amazon ES console.Specifically, the Cluster health tab does not include shard information, and the Indices tab is notpresent at all.

• Currently, Amazon ES does not support integration with Amazon Kinesis Data Firehose for domainsthat reside within a VPC. To use this service with Amazon ES, you must use a domain with publicaccess.

• To access the default installation of Kibana for a domain that resides within a VPC, users must haveaccess to the VPC. This process varies by network configuration, but likely involves connecting to aVPN or managed network or using a proxy server. To learn more, see the section called “About AccessPolicies on VPC Domains” (p. 98), the Amazon VPC User Guide, and the section called “Using a Proxyto Access Amazon ES from Kibana” (p. 82).

About Access Policies on VPC DomainsPlacing your Amazon ES domain within a VPC provides an inherent, strong layer of security. When youcreate a domain with public access, the endpoint takes the following form:

https://search-domain-name-identifier.region.es.amazonaws.com

As the "public" label suggests, this endpoint is accessible from any internet-connected device, thoughyou can (and should) control access to it (p. 32). If you access the endpoint in a web browser, you mightreceive a Not Authorized message, but the request reaches the domain.

When you create a domain with VPC access, the endpoint looks similar to a public endpoint:

https://vpc-domain-name-identifier.region.es.amazonaws.com

If you try to access the endpoint in a web browser, however, you might find that the request times out.To perform even basic GET requests, your computer must be able to connect to the VPC. This connectionoften takes the form of an internet gateway, VPN server, or proxy server. For details on the various formsit can take, see Scenarios and Examples in the Amazon VPC User Guide.


http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenarios.html

Amazon Elasticsearch Service Developer GuidePrerequisites

In addition to this connectivity requirement, VPCs let you manage access to the domain through securitygroups. For many use cases, this combination of security features is sufficient, and you might feelcomfortable applying an open access policy to the domain.

Operating with an open access policy does not mean that anyone on the internet can access the AmazonES domain. Rather, it means that if a request reaches the Amazon ES domain and the associated securitygroups permit it, the domain accepts the request without further security checks.

For an additional layer of security, we recommend using access policies that specify IAM users or roles.Applying these policies means that, for the domain to accept a request, the security groups must permitit and it must be signed with valid credentials.

NoteBecause security groups already enforce IP-based access policies, you can't apply IP-based accesspolicies to Amazon ES domains that reside within a VPC. If you use public access, IP-basedpolicies are still available.

Before You Begin: Prerequisites for VPC AccessBefore you can enable a connection between a VPC and your new Amazon ES domain, you must do thefollowing:

• Create a VPC

To create your VPC, you can use the Amazon VPC console, the AWS CLI, or one of the AWS SDKs.You must create a subnet in the VPC, or two subnets if you enable zone awareness (p. 45). For moreinformation, see Creating A VPC (p. 99). If you already have a VPC, you can skip this step.

• Reserve IP addresses

Amazon ES enables the connection of a VPC to a domain by placing network interfaces in a subnetof the VPC. Each network interface is associated with an IP address. You must reserve a sufficientnumber of IP addresses in the subnet for the network interfaces. For more information, see ReservingIP Addresses in a VPC Subnet (p. 100).

Creating a VPCTo create your VPC, you can use one of the following: the Amazon VPC console, the AWS CLI, or one ofthe AWS SDKs. The VPC must have a subnet, or two subnets if you enable zone awareness (p. 45). Thetwo subnets must be in different Availability Zones in the same region.

The following procedure shows how to use the Amazon VPC console to create a VPC with a publicsubnet, reserve IP addresses for the subnet, and create a security group to control access to your AmazonES domain. For other VPC configurations, see Scenarios and Examples in the Amazon VPC User Guide.

To create a VPC (console)

1. Sign in to the AWS Management Console, and open the Amazon VPC console at https://console.aws.amazon.com/vpc/.

2. In the navigation pane, choose VPC Dashboard.3. Choose Start VPC Wizard.4. On the Select a VPC Configuration page, select VPC with a Single Public Subnet.5. On the VPC with a Single Public Subnet page, keep the default options, and then choose Create

VPC.




http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenarios.html

https://console.aws.amazon.com/vpc/

https://console.aws.amazon.com/vpc/

Amazon Elasticsearch Service Developer GuideReserving IP Addresses in a VPC Subnet

6. In the confirmation message that appears, choose Close.

7. If you intend to enable zone awareness (p. 45) for your Amazon ES domain, you must create asecond subnet in a different Availability Zone in the same region. If you don't intend to enable zoneawareness, skip to step 8.

a. In the navigation pane, choose Subnets.

b. Choose Create Subnet.

c. In the Create Subnet dialog box, optionally create a name tag to help you identify the subnetlater.

d. For VPC, choose the VPC that you just created.

e. For Availability Zone, choose an Availability Zone that differs from that of the first subnet. TheAvailability Zones for both subnets must be in the same region.

f. For IPv4 CIDR block, configure a CIDR block large enough to provide sufficient IP addressesfor Amazon ES to use during maintenance activities. For more information, see Reserving IPAddresses in a VPC Subnet (p. 100).

NoteAmazon ES domains using VPC access don't support IPv6 addresses. You can use a VPCthat has IPv6 enabled, but the ENIs will have IPv4 addresses.

g. Choose Yes, Create.

8. In the navigation pane, choose Subnets.

9. In the list of subnets, find your subnet (or subnets, if you created a second subnet in step 7). In theAvailable IPv4 column, confirm that you have a sufficient number of IPv4 addresses.

10. Make a note of the subnet ID and Availability Zone. You need this information later when you launchyour Amazon ES domain and add an Amazon EC2 instance to your VPC.

11. Create an Amazon VPC security group. You use this security group to control access to your AmazonES domain.

a. In the navigation pane, choose Security Groups.

b. Choose Create Security Group.

c. In the Create Security Group dialog box, type a name tag, a group name, and a description. ForVPC, choose the ID of your VPC.

d. Choose Yes, Create.

12. Define a network ingress rule for your security group. This rule allows you to connect to yourAmazon ES domain.

a. In the navigation pane, choose Security Groups, and then select the security group that you justcreated.

b. At the bottom of the page, choose the Inbound Rules tab.

c. Choose Edit, and then choose HTTPS (443).

d. Choose Save.

Now you are ready to launch an Amazon ES domain (p. 13) in your Amazon VPC.

Reserving IP Addresses in a VPC SubnetAmazon ES connects a domain to a VPC by placing network interfaces in a subnet of the VPC (or twosubnets of the VPC if you enable zone awareness (p. 45)). Each network interface is associated with an IPaddress. Before you create your Amazon ES domain, you must have a sufficient number of IP addressesavailable in the VPC subnet to accommodate the network interfaces.


Amazon Elasticsearch Service Developer GuideService-Linked Role for VPC Access

The number of IP addresses that Amazon ES requires depends on the following:

• Number of data nodes in your domain. (Master nodes are not included in the number.)

• Whether you enable zone awareness (p. 45). If you enable zone awareness, you need only half thenumber of IP addresses per subnet that you need if you don't enable zone awareness.

Here is the basic formula: The number of IP addresses reserved in each subnet is three times the numberof nodes, divided by two if zone awareness is enabled.

Examples

• If a domain has 10 data nodes and zone awareness is enabled, the IP count is 10 / 2 * 3 = 15.

• If a domain has 10 data nodes and zone awareness is disabled, the IP count is 10 * 3 = 30.

When you create the domain, Amazon ES reserves the IP addresses. You can see the network interfacesand their associated IP addresses in the Network Interfaces section of the Amazon EC2 console athttps://console.aws.amazon.com/ec2/. The Description column shows which Amazon ES domain thenetwork interface is associated with.

TipWe recommend that you create dedicated subnets for the Amazon ES reserved IP addresses. Byusing dedicated subnets, you avoid overlap with other applications and services and ensure thatyou can reserve additional IP addresses if you need to scale your cluster in the future. To learnmore, see Creating a Subnet in Your VPC.

Service-Linked Role for VPC AccessA service-linked role is a unique type of IAM role that delegates permissions to a service so that it cancreate and manage resources on your behalf. Amazon ES requires a service-linked role to access yourVPC, create the domain endpoint, and place network interfaces in a subnet of your VPC.

Amazon ES automatically creates the role when you use the Amazon ES console to create adomain within a VPC. For this automatic creation to succeed, you must have permissions for theiam:CreateServiceLinkedRole action. To learn more, see Service-Linked Role Permissions in theIAM User Guide.

After Amazon ES creates the role, you can view it(AWSServiceRoleForAmazonElasticsearchService) using the IAM console.

NoteIf you create a domain that uses a public endpoint, Amazon ES doesn’t need the service-linkedrole and doesn't create it.

For full information on this role's permissions and how to delete it, see Using Service-LinkedRoles (p. 171).

Migrating from Public Access to VPC AccessWhen you create a domain, you specify whether it should have a public endpoint or reside within a VPC.Once created, you cannot switch from one to the other. Instead, you must create a new domain andeither manually reindex or migrate your data. Snapshots offer a convenient means of migrating data. Forinformation about taking and restoring snapshots, see Working with Index Snapshots (p. 74).


https://console.aws.amazon.com/ec2/

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/working-with-vpcs.html#AddaSubnet

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html#iam-term-service-linked-role

http://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#service-linked-role-permissions

Amazon Elasticsearch Service Developer GuideAmazon VPC Documentation

Amazon VPC DocumentationAmazon VPC has its own set of documentation to describe how to create and use your Amazon VPC. Thefollowing table provides links to the Amazon VPC guides.

Description Documentation

How to get started using Amazon VPC Amazon VPC Getting Started Guide

How to use Amazon VPC through the AWSManagement Console

Amazon VPC User Guide

Complete descriptions of all the Amazon VPCcommands

Amazon EC2 Command Line Reference (TheAmazon VPC commands are part of the AmazonEC2 reference.)

Complete descriptions of the Amazon VPC APIactions, data types, and errors

Amazon EC2 API Reference (The Amazon VPC APIactions are part of the Amazon EC2 reference.)

Information for the network administrator whoconfigures the gateway at your end of an optionalIPsec VPN connection

Amazon VPC Network Administrator Guide

For more detailed information about Amazon Virtual Private Cloud, see Amazon Virtual Private Cloud.


http://docs.aws.amazon.com/AmazonVPC/latest/GettingStartedGuide/

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/

http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/

http://docs.aws.amazon.com/AWSEC2/latest/APIReference/

http://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/

https://aws.amazon.com/vpc/

Amazon Elasticsearch Service Developer GuideEnabling Encryption of Data at Rest

Encryption of Data at Rest forAmazon Elasticsearch Service

Amazon ES domains offer encryption of data at rest, a security feature that helps prevent unauthorizedaccess to your data. The feature uses AWS Key Management Service (KMS) to store and manage yourencryption keys. If enabled, it encrypts the following aspects of a domain:

• Indices• Automated snapshots• Elasticsearch logs• Swap files• All other data in the application directory

The following are not encrypted when you enable encryption of data at rest, but you can take additionalsteps to protect them:

• Manual snapshots: Currently, you can't use KMS master keys to encrypt manual snapshots. Youcan, however, use server-side encryption with S3-managed keys to encrypt the bucket that you useas a snapshot repository. For instructions, see the section called “Registering a Manual SnapshotDirectory” (p. 76).

• Slow logs: If you publish slow logs (p. 28) and want to encrypt them, you can encrypt their CloudWatchLogs log group using the same KMS master key as the Amazon ES domain. To learn more, see EncryptLog Data in CloudWatch Logs Using AWS KMS in the Amazon CloudWatch Logs User Guide.

To learn how to create KMS master keys, see Creating Keys in the AWS Key Management Service DeveloperGuide.

Enabling Encryption of Data at RestBy default, domains do not encrypt data at rest, and you can't configure existing domains to usethe feature. To enable the feature, you must create another domain (p. 13) and migrate your data.Encryption of data at rest requires Elasticsearch 5.1 or newer.

Disabling Encryption of Data at RestAfter you configure a domain to encrypt data at rest, you can't disable the setting. Instead, you can takea manual snapshot (p. 74) of the existing domain, create another domain (p. 13), migrate your data, anddelete the old domain.

Monitoring Domains That Encrypt Data at RestDomains that encrypt data at rest have two additional metrics: KMSKeyError andKMSKeyInaccessible. For full descriptions of these metrics, see the section called “ClusterMetrics” (p. 47). You can view them using the Amazon ES console or Amazon CloudWatch.



http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/encrypt-log-data-kms.html

http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/encrypt-log-data-kms.html

http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html

Amazon Elasticsearch Service Developer GuideOther Considerations

TipEach metric represents a significant problem for a domain, so we recommend that you createCloudWatch alarms for both. For more information, see the section called “RecommendedCloudWatch Alarms” (p. 92).

Other Considerations• If you delete the key that you used to encrypt a domain, the domain becomes inaccessible. The

Amazon ES team can't help you recover your data. AWS Key Management Service deletes master keysonly after a waiting period of at least seven days, so the Amazon ES team might contact you if theydetect that your domain is at risk.

• Automatic key rotation preserves the properties of your KMS master keys, so the rotation has noeffect on your ability to access your Elasticsearch data. Encrypted Amazon ES domains do not supportmanual key rotation, which involves creating a new master key and updating any references to the oldkey. To learn more, see Rotating Customer Master Keys in the AWS Key Management Service DeveloperGuide.

• Certain instance types do not support encryption of data at rest. For details, see the section called“Supported Instance Types” (p. 114).

• Encryption of data at rest is not available in the cn-northwest-1 (Ningxia) region.• Kibana still works on domains that encrypt data at rest.• Domains that encrypt data at rest use a different repository name for their automated snapshots. To

learn more, see the section called “Restoring Snapshots” (p. 78).• Encrypting an Amazon ES domain requires two grants, and each encryption key has a limit of 500

grants per principal. This limit means that the maximum number of Amazon ES domains you canencrypt using a single key is 250. At present, Amazon ES supports a maximum of 100 domains peraccount, so this grant limit is of no consequence. If the domain limit per account increases, however,the grant limit might become relevant.

If you need to encrypt more than 250 domains at that time, you can create additional keys. Keys areregional, not global, so if you operate in more than one region, you already need multiple keys.


http://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html

http://docs.aws.amazon.com/kms/latest/developerguide/grants.html

http://docs.aws.amazon.com/kms/latest/developerguide/limits.html#grants-per-principal-per-key

Amazon Elasticsearch Service Developer GuideFailed Cluster Nodes

Handling AWS Service ErrorsThis section describes how to respond to common AWS service errors. Consult the information in thissection before contacting AWS Support.

Failed Cluster NodesEC2 instances might experience unexpected terminations and restarts. Typically, Amazon ES restartsthe nodes for you. However, it's possible for one or more nodes in an Elasticsearch cluster to remain in afailed condition.

To check for this condition, open your domain dashboard on the Amazon ES console. Choose theMonitoring tab, and then choose the Nodes metric. See if the reported number of nodes is fewer thanthe number that you configured for your cluster. If the metric shows that one or more nodes is down formore than one day, contact AWS Support.

You can also set a CloudWatch alarm (p. 92) to notify you when this issue occurs.

NoteThe Nodes metric is not accurate during changes to your cluster configuration and duringroutine maintenance for the service. This behavior is expected. The metric will report the correctnumber of cluster nodes soon. To learn more, see the section called “About ConfigurationChanges” (p. 44).

To protect your clusters from unexpected node terminations and restarts, create at least one replicafor each index in your Amazon ES domain. To learn more, see Shards and Replicas in the Elasticsearchdocumentation.

Red Cluster StatusA red cluster status means that at least one primary shard and its replicas are not allocated to a node.Amazon ES stops taking automatic snapshots, even of healthy indices, while the red cluster statuspersists.

The most common causes of a red cluster status are failed cluster nodes (p. 105) and the Elasticsearchprocess crashing due to a continuous heavy processing load.

NoteIf your Amazon ES domain enters a red cluster status, AWS Support might contact you to askwhether you want to address the problem yourself or you want the support team to restorethe latest automatic snapshot of the domain. If you don't respond or resolve the issue withinseven days, AWS Support restores the latest automatic snapshot. You can set a CloudWatchalarm (p. 92) to notify you when a red cluster status occurs.

Ultimately, red shards cause red clusters, and red indices cause red shards. To identity the indices causingthe red cluster status, Elasticsearch has some helpful APIs.

GET /_cluster/allocation/explain chooses the first unassigned shard that it finds and explainswhy it cannot be allocated to a node:

{


https://aws.amazon.com/premiumsupport/

https://aws.amazon.com/premiumsupport/

https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html#getting-started-shards-and-replicas

Amazon Elasticsearch Service Developer GuideRecovering from a Continuous Heavy Processing Load

"index": "test4", "shard": 0, "primary": true, "current_state": "unassigned", "can_allocate": "no", "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes"}

GET /_cat/indices?v shows the health status, number of documents, and disk usage for each index:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.sizegreen open test1 30h1EiMvS5uAFr2t5CEVoQ 5 0 820 0 14mb 14mbgreen open test2 sdIxs_WDT56afFGu5KPbFQ 1 0 0 0 233b 233bgreen open test3 GGRZp_TBRZuSaZpAGk2pmw 1 1 2 0 14.7kb 7.3kbred open test4 BJxfAErbTtu5HBjIXJV_7A 1 0 green open test5 _8C6MIXOSxCqVYicH3jsEA 1 0 7 0 24.3kb 24.3kb

Deleting red indices is the fastest way to fix a red cluster status. Depending on the reason for the redcluster status, you might then scale your Amazon ES domain to use larger instance types, more instances,or more EBS-based storage and try to recreate the problematic indices.

If deleting a problematic index isn't feasible, you can restore a snapshot (p. 78), delete documents fromthe index, change the index settings, reduce the number of replicas, or delete other indices to free updisk space. The important step is to resolve the red cluster status before reconfiguring your Amazon ESdomain. Reconfiguring a domain with a red cluster status can compound the problem and lead to thedomain being stuck in a configuration state of Processing until you resolve the status.

Recovering from a Continuous Heavy Processing LoadTo determine if a red cluster status is due to a continuous heavy processing load on a data node, monitorthe following cluster metrics.

Relevant Metric Description Recovery

JVMMemoryPressure Specifies the percentage of the Javaheap used for all data nodes in acluster. View the Maximum statisticfor this metric, and look for smallerand smaller drops in memory pressureas the Java garbage collector fails toreclaim sufficient memory. This patternlikely is due to complex queries or largedata fields.

At 75% memory usage, Elasticsearchtriggers the Concurrent Mark Sweep(CMS) garbage collector, which runsalongside other processes to keeppauses and disruptions to a minimum.If CMS fails to reclaim enough memoryand usage remains above 75%,

Set memory circuitbreakers for the JVM. Formore information, seethe section called “JVMOutOfMemoryError” (p. 108).

If the problem persists, deleteunnecessary indices, reducethe number or complexityof requests to the domain,add instances, or use largerinstance types.


Amazon Elasticsearch Service Developer GuideYellow Cluster Status

Relevant Metric Description Recovery

Elasticsearch triggers a differentgarbage collection algorithm that haltsor slows other processes in order tofree up sufficient memory to preventan out of memory error.

At 95% memory usage, Elasticsearchkills processes that attempt to allocatememory. It might kill a critical processand bring down one or more nodes inthe cluster.

The _nodes/stats/jvm API offersa useful summary of JVM statistics,memory pool usage, and garbagecollection information:

GET elasticsearch_domain_endpoint/_nodes/stats/jvm?pretty

CPUUtilization Specifies the percentage of CPUresources used for data nodes in acluster. View the Maximum statistic forthis metric, and look for a continuouspattern of high usage.

Add data nodes or increase thesize of the instance types ofexisting data nodes. For moreinformation, see the sectioncalled “Configuring AmazonES Domains” (p. 18).

Nodes Specifies the number of nodes in acluster. View the Minimum statisticfor this metric. This value fluctuateswhen the service deploys a new fleet ofinstances for a cluster.

Add data nodes. For moreinformation, see the sectioncalled “Configuring AmazonES Domains” (p. 18).

Yellow Cluster StatusA yellow cluster status means that the primary shards for all indices are allocated to nodes in a cluster,but the replica shards for at least one index are not. Single-node clusters always initialize with a yellowcluster status because there is no other node to which Amazon ES can assign a replica. To achieve greencluster status, increase your node count. For more information, see the section called “Sizing Amazon ESDomains” (p. 87) and the section called “Configuring Amazon ES Domains” (p. 18).

ClusterBlockExceptionYou might receive a ClusterBlockException error for the following reasons.

Lack of Available Storage SpaceElasticsearch has a default "low watermark" of 85%, meaning that once disk usage exceeds 85%,Elasticsearch no longer allocates shards to that node. Elasticsearch also has a default "high watermark"of 90%, at which point it attempts to relocate shards to other nodes. If no nodes have enoughstorage space to accommodate shard relocation, basic write operations like adding documents and


Amazon Elasticsearch Service Developer GuideBlock Disks Due to Low Memory

creating indices can begin to fail. To learn more, see Disk-based Shard Allocation in the Elasticsearchdocumentation.

To avoid these issues, you can monitor the FreeStorageSpace metric in the Amazon ES console andcreate CloudWatch alarms (p. 92) to trigger when FreeStorageSpace drops below a certain threshold.GET /_cat/allocation?v also provides a useful summary of shard allocation and disk usage. Toresolve issues associated with a lack of storage space, you can scale your Amazon ES domain to uselarger instance types, more instances, or more EBS-based storage. For instructions, see the section called“Configuring Amazon ES Domains” (p. 18).

Block Disks Due to Low MemoryWhen the JVMMemoryPressure metric exceeds 92% for 30 minutes, Amazon ES triggers a protectionmechanism and blocks all write operations to prevent the cluster from reaching red status. When theprotection is on, write operations fail with a ClusterBlockException error, new indexes can't becreated, and the IndexCreateBlockException error is thrown.

When the JVMMemoryPressure metric returns to 88% or lower for five minutes, the protection isdisabled, and write operations to the cluster are unblocked.

JVM OutOfMemoryErrorA JVM OutOfMemoryError typically means that one of the following JVM circuit breakers was reached.

Circuit Breaker Description Cluster Setting Property

Parent Breaker Total percentage of JVMheap memory allowed for allcircuit breakers. The defaultvalue is 70%.

indices.breaker.total.limit

For more information, see Cluster UpdateSettings in the Elasticsearch documentation.

Field Data Breaker Percentage of JVM heapmemory allowed to loada single data field intomemory. The default value is60%. If you upload data withlarge fields, we recommendraising this limit.

indices.breaker.fielddata.limit

For more information, see Field data in theElasticsearch documentation.

Request Breaker Percentage of JVM heapmemory allowed for datastructures used to respondto a service request. Thedefault value is 40%. If yourservice requests involvecalculating aggregations, werecommend raising this limit.

indices.breaker.request.limit

For more information, see Field data in theElasticsearch documentation.


https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-fielddata.html


Amazon Elasticsearch Service Developer GuideKibana: I Can't Sign AWS Service

Requests to the Kibana Service Endpoint

TroubleshootingThe following sections offer solutions to common problems that you might encounter when you useservices (such as Amazon S3, Kinesis, and IAM) and products (such as Kibana) that integrate with AmazonElasticsearch Service (Amazon ES):

Topics• Kibana: I Can't Sign AWS Service Requests to the Kibana Service Endpoint (p. 109)• Kibana: I Don't See the Indices for My Elasticsearch Domain in Kibana 4 (p. 110)• Kibana: I Get a Browser Error When I Use Kibana to View My Data (p. 111)• Integrations: I Don't See a Service Role for Amazon ES in the IAM Console (p. 111)• Domain Creation: Unauthorized Operation When Selecting VPC Access (p. 112)• Domain Creation: Stuck at Loading After Choosing VPC Access (p. 112)• SDKs: I Get Certificate Errors When I Try to Use an SDK (p. 113)

For information about service-specific errors, see Handling AWS Service Errors (p. 105) in this guide.

Kibana: I Can't Sign AWS Service Requests to theKibana Service Endpoint

The Kibana endpoint doesn't support signed AWS service requests. We recommend that you accessKibana with one of the configuration options described in the following table.

Kibana Configuration Description

Anonymous, IP-basedAccess

If the Kibana host is behind a firewall, configure the Kibana endpoint toaccept anonymous requests from the IP address of the firewall. Use CIDRnotation if you need to specify a range of IP addresses.

NAT Gateway withAmazon VPC

Amazon VPC supports NAT gateways. When you create a NAT gateway,you must specify an Elastic IP address to associate with the gateway. TheNAT gateway sends traffic to the public Internet gateway using the ElasticIP address as the source IP address. Specify this Elastic IP address in theaccess policy for Kibana to allow all requests from the gateway. For moreinformation, see NAT Gateway Basics and Creating a NAT Gateway in theAmazon Virtual Private Cloud User Guide.

Examples

The following example is an anonymous, IP-based access policy that specifies a range of IP addresses andan 18-bit routing prefix:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "*"


http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html#nat-gateway-basics

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html#nat-gateway-creating

Amazon Elasticsearch Service Developer GuideKibana: I Don't See the Indices for

My Elasticsearch Domain in Kibana 4

}, "Action": "es:*", "Resource": "arn:aws:es:us-west-2:123456789012:domain/mydomain/_plugin/kibana", "Condition": { "IpAddress": { "aws:SourceIp": "192.240.192.0/18" } } } ]}

The following example specifies anonymous access from the Elastic IP address associated with a NATgateway:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "es:*", "Resource": "arn:aws:es:us-west-2:123456789012:domain/mydomain/_plugin/kibana", "Condition": { "IpAddress": { "aws:SourceIp": "198.51.100.4" } } } ]}

Kibana: I Don't See the Indices for My ElasticsearchDomain in Kibana 4

Users familiar with Kibana 3 but new to Kibana 4 sometimes have difficulty finding their indices in theinterface. Unlike Kibana 3, which provides default dashboards to view your data, Kibana 4 requires youto first specify an index name or pattern that matches the name of an index in your Amazon ES domain.For example, if your Amazon ES domain contains an index named movies-2013, any of the followingpatterns would match this index:

• movies-2013• movies-*• mov*

You can configure visualizations for the data in your Amazon ES domain index after you specify the indexname or pattern. You can find the names of your domain indices on the Indices tab in the Amazon ESconsole. For more information about using Kibana 4, see the Kibana User Guide.

NoteAmazon ES also supports Kibana 3, so you can configure access to that version of the toolif you prefer to use it. Specify /_plugin/kibana3 as the resource in your access policyrather than /_plugin/kibana. After the service finishes processing the configurationchange, you can access Kibana 3 by manually editing the Kibana service endpoint provided


https://www.elastic.co/guide/en/kibana/4.0/index.html

Amazon Elasticsearch Service Developer GuideKibana: I Get a Browser Error When

I Use Kibana to View My Data

in the Amazon ES console. For example, if the console indicates that your Kibana endpointis mydomain-6w5y8xjt5ydwsrubmdk4m5kcpa.us-west-2.es.amazonaws.com/_plugin/kibana/, point your browser to mydomain-6w5y8xjt5ydwsrubmdk4m5kcpa.us-west-2.es.amazonaws.com/_plugin/kibana3/ instead.

Kibana: I Get a Browser Error When I Use Kibana toView My Data

Your browser wraps service error messages in HTTP response objects when you use Kibana to view datain your Amazon ES domain. You can use developer tools commonly available in web browsers, such asDeveloper Mode in Chrome, to view the underlying service errors and assist your debugging efforts.

To view service errors in Chrome

1. From the menu, choose View, Developer, Developer Tools.2. Choose the Network tab.3. In the Status column, choose any HTTP session with a status of 500.

For example, the following service error message indicates that a search request likely failed for oneof the reasons shown in the following table:

"Request to Elasticsearch failed: {"error":"SearchP…be larger than limit of[5143501209/4.7gb]]; }]"}"

Potential Cause Workaround

You reached the JVM request memory circuitbreaker.

The request breaker specifies the percentageof JVM memory used to respond to a servicerequest. You can configure JVM circuitbreakers to work around this failure. For moreinformation about configuring JVM circuitbreakers, see JVM OutOfMemoryError (p. 108)in Handling AWS Service Errors (p. 105) in thisguide.

You specified a generic regular expression inyour Kibana dashboard, such as logstash*.

Use a more restrictive regular expression, suchas limiting results to a subset of indices over atime period of the last seven days.

To view service errors in Firefox

1. From the menu, choose Tools, Web Developer, Network.2. Choose any HTTP session with a status of 500.3. Choose the Response tab to view the service response.

Integrations: I Don't See a Service Role for AmazonES in the IAM Console

You can integrate Amazon ES with other services, such as Amazon S3 and Kinesis, as described inLoading Streaming Data into Amazon ES from Amazon S3 (p. 66) and Loading Streaming Data into


Amazon Elasticsearch Service Developer GuideDomain Creation: Unauthorized

Operation When Selecting VPC Access

Amazon ES from Kinesis (p. 70) in this guide. Both of these integrations use AWS Lambda as an eventhandler in the cloud. When you create a Lambda function using the AWS Lambda console, the consoleautomatically opens the IAM console to help you create the required execution role. You don't need toopen the IAM console yourself and select a service role. However, you must open the IAM console afterAWS Lambda helps you to create the new role and attach the following IAM policy to it:

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "es:*" ], "Effect": "Allow", "Resource": "arn:aws:es:us-west-2:123456789012:domain/<my_domain_name>/*" } ]}

For step-by-step procedures, see Loading Streaming Data into Amazon ES from Amazon S3 (p. 66) andLoading Streaming Data into Amazon ES from Kinesis (p. 70) in this guide.

Domain Creation: Unauthorized Operation WhenSelecting VPC Access

When you create a new domain using the Amazon ES console, you have the option to select public accessor VPC access. If you select VPC access, Amazon ES queries for VPC information and fails if you don'thave the right policies associated with your user credentials. The error message follows:

You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation

To enable this query, you must have access to the ec2:DescribeVpcs, ec2:DescribeSubnets,and ec2:DescribeSecurityGroups operations. This requirement is only for the console. If you usethe AWS CLI to create and configure a domain with a VPC endpoint, you don't need access to thoseoperations.

Domain Creation: Stuck at Loading After ChoosingVPC Access

After creating a new domain that uses VPC access, the domain's Configuration state might neverprogress beyond Loading. If this issue occurs, you likely have AWS Security Token Service (AWS STS)disabled for your region.

To add VPC endpoints to your VPC, Amazon ES needs to assume theAWSServiceRoleForAmazonElasticsearchService role. Thus, AWS STS must be enabled to createnew domains that use VPC access in a given region. To learn more about enabling and disabling AWSSTS, see the IAM User Guide.


http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html

Amazon Elasticsearch Service Developer GuideSDKs: I Get Certificate Errors When I Try to Use an SDK

SDKs: I Get Certificate Errors When I Try to Use anSDK

Because AWS SDKs use the CA certificates from your computer, changes to the certificates on the AWSservers can cause connection failures when you attempt to use an SDK. Error messages vary, but typicallycontain the following text:

Failed to query Elasticsearch...SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

You can prevent these failures by keeping your computer's CA certificates and operating system up-to-date. If you encounter this issue in a corporate environment and do not manage your own computer, youmight need to ask an administrator to assist with the update process.

The following list shows minimum operating system and Java versions:

• Microsoft Windows versions that have updates from January 2005 or later installed contain at leastone of the required CAs in their trust list.

• Mac OS X 10.4 with Java for Mac OS X 10.4 Release 5 (February 2007), Mac OS X 10.5 (October 2007),and later versions contain at least one of the required CAs in their trust list.

• Red Hat Enterprise Linux 5 (March 2007), 6, and 7 and CentOS 5, 6, and 7 all contain at least one ofthe required CAs in their default trusted CA list.

• Java 1.4.2_12 (May 2006), 5 Update 2 (March 2005), and all later versions, including Java 6 (December2006), 7, and 8, contain at least one of the required CAs in their default trusted CA list.

The three certificate authorities are:

• Amazon Root CA 1• Starfield Services Root Certificate Authority - G2• Starfield Class 2 Certification Authority

Root certificates from the first two authorities are available from Amazon Trust Services, but keepingyour computer up-to-date is the more straightforward solution. To learn more about ACM-providedcertificates, see AWS Certificate Manager FAQs.

NoteCurrently, Amazon ES domains in the us-east-1 region use certificates from a different authority.We plan to update the region to use these new certificate authorities in the near future.


https://www.amazontrust.com/repository/

https://aws.amazon.com/certificate-manager/faqs/#certificates

Amazon Elasticsearch Service Developer GuideSupported Instance Types

Amazon Elasticsearch ServiceGeneral Reference

Amazon Elasticsearch Service (Amazon ES) supports a variety of instances, operations, plugins, and otherresources.

Topics• Supported Instance Types (p. 114)• Supported Elasticsearch Operations (p. 116)• Supported Plugins (p. 121)• Other Supported Resources (p. 123)

Supported Instance TypesAn instance is a virtual computing environment. An instance type is a specific configuration of CPU,memory, storage, and networking capacity. Choose an instance type for your Amazon ES domain that isbased on the requirements of the application or software that you plan to run on your instance. If youhave enabled dedicated master nodes (p. 90), you can choose an instance type for the master nodes thatdiffers from the instance type that you choose for the data nodes.

To learn more, see the section called “Sizing Amazon ES Domains” (p. 87), the section called “Cluster andInstance Limits” (p. 167), and the section called “EBS Volume Size Limits” (p. 167).

Amazon ES supports the following instance types.

T2 Instance Types

• t2.micro.elasticsearch

• t2.small.elasticsearch

• t2.medium.elasticsearch

Note

• You can use the t2 instance types only if the instance count for your domain is 10 or fewer.• The t2.micro.elasticsearch instance type only supports Elasticsearch 2.3 and 1.5.• The t2 instance types do not support encryption of data at rest.

M3 Instance Types

• m3.medium.elasticsearch

• m3.large.elasticsearch

• m3.xlarge.elasticsearch

• m3.2xlarge.elasticsearch

Note

• The m3 instance types are not available in the us-east-2, ca-central-1, eu-west-2, eu-west-3,ap-northeast-2, ap-south-1, and cn-northwest-1 regions.


Amazon Elasticsearch Service Developer GuideSupported Instance Types

• The m3 instance types do not support encryption of data at rest.

M4 Instance Types

• m4.large.elasticsearch

• m4.xlarge.elasticsearch




Note

• The m4 instance types are not available in the eu-west-3 region.

C4 Instance Types

• c4.large.elasticsearch

• c4.xlarge.elasticsearch

• c4.2xlarge.elasticsearch



Note

• The c4 instance types are not available in the eu-west-3 region.

R3 Instance Types

• r3.large.elasticsearch

• r3.xlarge.elasticsearch

• r3.2xlarge.elasticsearch



Note

• The r3 instance types are not available in the ca-central-1, eu-west-2, eu-west-3, sa-east-1,and cn-northwest-1 regions.

• The r3 instance types do not support encryption of data at rest.

R4 Instance Types

• r4.large.elasticsearch

• r4.xlarge.elasticsearch






Amazon Elasticsearch Service Developer GuideSupported Elasticsearch Operations

I2 Instance Types

• i2.xlarge.elasticsearch

• i2.2xlarge.elasticsearch

Note

• The i2 instance types are not available in the sa-east-1, ca-central-1, eu-west-2, eu-west-3,us-east-2, and cn-northwest-1 regions.

I3 Instance Types

• i3.large.elasticsearch

• i3.xlarge.elasticsearch





Note

• The i3 instance types do not support EBS storage volumes.• The i3 instance types do not support Elasticsearch 2.3 or 1.5.

Supported Elasticsearch OperationsAmazon ES currently supports several Elasticsearch versions. The following topics show the operationsthat Amazon ES supports for each version.

Topics• Notable API Differences (p. 116)• Version 6.0 (p. 117)• Version 5.5 (p. 118)• Version 5.3 (p. 119)• Version 5.1 (p. 119)• Version 2.3 (p. 120)• Version 1.5 (p. 121)

Notable API DifferencesPrior to Elasticsearch 5.3, the _cluster/settings API on Amazon ES domains supported only theHTTP PUT method, not the GET method. It now supports the GET method, as shown in the followingexample:

curl -XGET 'https://domain.region.es.amazonaws.com/_cluster/settings?pretty'

A sample return follows:

{


Amazon Elasticsearch Service Developer GuideVersion 6.0

"persistent" : { "cluster" : { "routing" : { "allocation" : { "cluster_concurrent_rebalance" : "2" } } }, "indices" : { "recovery" : { "max_bytes_per_sec" : "20mb" } } }, "transient" : { "cluster" : { "routing" : { "allocation" : { "exclude" : { "di_number" : "2" } } } } }}

• cluster_concurrent_rebalance specifies the number of shards that can be relocated to newnodes at any given time. For more information, see Shard Rebalancing Settings in the Elasticsearchdocumentation.

• max_bytes_per_sec is the maximum data transfer speed that Elasticsearch uses during a recoveryevent.

• di_number is an internal Amazon ES value that is used to copy shards to new domain instances afterconfiguration changes.

Version 6.0For Elasticsearch 6.0, Amazon ES supports the following operations.

• /_alias

• /_aliases

• /_all

• /_analyze

• /_bulk

• /_cache/clear (Index only)• /_cat

• /_cluster/allocation/explain

• /_cluster/health

• /_cluster/pending_tasks

• /_cluster/settings forthree properties4:• indices.breaker.fielddata.limit

• indices.breaker.request.limit

• /_delete_by_query1

• /_explain

• /_field_stats

• /_flush

• /_forcemerge (Index only)• /_ingest

• /_mapping

• /_mget

• /_msearch

• /_mtermvectors

• /_nodes

• /_plugin/kibana

• /_recovery (Index only)• /_refresh

• /_reindex1

• /_scripts3

/_search2

• /_search profile

• /_segments (Index only)• /_shard_stores

• /_shrink

• /_snapshot

• /_stats

• /_status

• /_tasks

• /_template

• /_termvectors (Index only)• _update3

• /_update_by_query1


https://www.elastic.co/guide/en/elasticsearch/reference/current/shards-allocation.html#_shard_rebalancing_settings

https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-recovery.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-recovery.html


• indices.breaker.total.limit

• /_cluster/state

• /_cluster/stats

• /_count

• /_rollover • /_validate

1. Cluster configuration changes might interrupt these operations before completion. We recommendthat you use the /_tasks operation along with these operations to verify that the requestscompleted successfully.

2. DELETE requests to /_search/scroll with a message body must specify "Content-Length" inthe HTTP header. Most clients add this header by default. To avoid a problem with = characters inscroll_id values, use the request body, not the query string, to pass scroll_id values to AmazonES.

3. For considerations about using scripts, see the section called “Other Supported Resources” (p. 123).4. Refers to the PUT method. For information about the GET method, see the section called “Notable API

Differences” (p. 116).

For more information about Elasticsearch 6.0 operations, see the Elasticsearch documentation.


• /_alias

• /_aliases

• /_all

• /_analyze

• /_bulk





• /_cluster/settings forthree properties4:• indices.breaker.fielddata.limit



• /_cluster/state

• /_cluster/stats

• /_count


• /_explain

• /_field_stats

• /_flush

• /_forcemerge (Index only)• /_ingest

• /_mapping

• /_mget

• /_msearch

• /_mtermvectors

• /_nodes

• /_plugin/kibana

• /_recovery (Index only)• /_refresh

• /_reindex1

• /_rollover

• /_scripts3

/_search2


• /_segments (Index only)• /_shard_stores

• /_shrink

• /_snapshot

• /_stats

• /_status

• /_tasks

• /_template

• /_termvectors (Index only)• _update3


• /_validate


2. DELETE requests to /_search/scroll with a message body must specify "Content-Length" inthe HTTP header. Most clients add this header by default. To avoid a problem with = characters in




scroll_id values, use the request body, not the query string, to pass scroll_id values to AmazonES.

3. For considerations about using scripts, see the section called “Other Supported Resources” (p. 123).

4. Refers to the PUT method. For information about the GET method, see the section called “Notable APIDifferences” (p. 116).



• /_alias

• /_aliases

• /_all

• /_analyze

• /_bulk





• /_cluster/settings for three properties4:• indices.breaker.fielddata.limit



• /_cluster/state

• /_cluster/stats

• /_count


• /_explain

• /_field_stats

• /_flush

• /_forcemerge (Indexonly)

• /_ingest

• /_mapping

• /_mget

• /_msearch

• /_mtermvectors

• /_nodes

• /_plugin/kibana

• /_recovery (Indexonly)

• /_refresh

• /_reindex1

• /_rollover

• /_search2


• /_segments (Indexonly)

• /_shard_stores

• /_shrink

• /_snapshot

• /_stats

• /_status

• /_tasks

• /_template

• /_termvectors(Index only)

• _update3


• /_validate




4. Refers to the PUT method. For information about the GET method, see the section called “Notable APIDifferences” (p. 116).







• /_alias

• /_aliases

• /_all

• /_analyze

• /_bulk





• /_cluster/settings for three properties(PUT only):• indices.breaker.fielddata.limit



• /_cluster/state

• /_cluster/stats

• /_count


• /_explain

• /_field_stats

• /_flush


• /_ingest

• /_mapping

• /_mget

• /_msearch

• /_mtermvectors

• /_nodes

• /_plugin/kibana


• /_refresh

• /_reindex1

• /_rollover

• /_search2



• /_shard_stores

• /_shrink

• /_snapshot

• /_stats

• /_status

• /_tasks

• /_template

• /_termvectors(Index only)

• _update3


• /_validate






• /_alias

• /_aliases

• /_all

• /_analyze

• /_bulk



• /_cluster/settings for four properties(PUT only):• indices.breaker.fielddata.limit


• /_cluster/stats

• /_count

• /_flush


• /_mapping

• /_mget

• /_msearch

• /_nodes

• /_percolate

• /_plugin/kibana


• /_refresh

• /_search


• /_snapshot

• /_stats

• /_status

• /_template





• threadpool.bulk.queue_size



• /_alias

• /_aliases

• /_all

• /_analyze

• /_bulk

• /_cat


• /_cluster/settings for four properties (PUTonly):• indices.breaker.fielddata.limit



• threadpool.bulk.queue_size

• /_cluster/stats

• /_count

• /_flush

• /_mapping

• /_mget

• /_msearch

• /_nodes

• /_percolate

• /_plugin/kibana

• /_plugin/kibana3

• /_refresh

• /_search

• /_snapshot

• /_stats

• /_status

• /_template


Supported PluginsAmazon ES comes prepackaged with several plugins that are available from the Elasticsearch community.Plugins are automatically deployed and managed for you.

Elasticsearch Version Plugins

6.0 • ICU Analysis• Ingest Attachment Processor• Ingest User Agent Processor• Japanese (kuromoji) Analysis• Mapper Murmur3• Mapper Size• Phonetic Analysis• Smart Chinese Analysis• Stempel Polish Analysis• Ukrainian Analysis

5.5 • ICU Analysis• Ingest Attachment Processor• Ingest User Agent Processor




https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/analysis-icu.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/ingest-attachment.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/ingest-user-agent.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/analysis-kuromoji.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/mapper-murmur3.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/mapper-size.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/analysis-phonetic.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/analysis-smartcn.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/analysis-stempel.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/analysis-ukrainian.html




Amazon Elasticsearch Service Developer GuideSupported Plugins


• Japanese (kuromoji) Analysis• Mapper Attachments• Mapper Murmur3• Mapper Size• Phonetic Analysis• Smart Chinese Analysis• Stempel Polish Analysis• Ukrainian Analysis

NoteElasticsearch 5.5 supports Kibana 5, but it runs as a Node.jsapplication (not as a plugin).

5.3 • ICU Analysis• Ingest Attachment Processor• Ingest User Agent Processor• Japanese (kuromoji) Analysis• Mapper Attachments• Mapper Murmur3• Mapper Size• Phonetic Analysis• Smart Chinese Analysis• Stempel Polish Analysis• Ukrainian Analysis


5.1 • ICU Analysis• Ingest Attachment Processor• Ingest User Agent Processor• Japanese (kuromoji) Analysis• Mapper Murmur3• Phonetic Analysis• Smart Chinese Analysis• Stempel Polish Analysis


2.3 • ICU Analysis• Japanese (kuromoji) Analysis• Kibana 4• Phonetic Analysis



https://www.elastic.co/guide/en/elasticsearch/plugins/5.5/mapper-attachments.html











https://www.elastic.co/guide/en/elasticsearch/plugins/5.3/mapper-attachments.html



















Amazon Elasticsearch Service Developer GuideOutput Plugins


1.5 • ICU Analysis• Japanese (kuromoji) Analysis• Kibana 3 (at the _plugin/kibana3/ endpoint)• Kibana 4

Output PluginsAmazon ES supports two Logstash output plugins to stream data into Amazon ES: the standardElasticsearch output plugin and the logstash-output-amazon-es plugin, which signs and exportsLogstash events to Amazon ES.

For more information about Logstash, see the section called “Loading Bulk Data with the LogstashPlugin” (p. 85).

Other Supported Resourcesbootstrap.mlockall

The service enables bootstrap.mlockall in elasticsearch.yml, which locks JVM memory andprevents the operating system from swapping it to disk. This applies to all supported instance typesexcept for the following:• t2.micro.elasticsearch

• t2.small.elasticsearch

• t2.medium.elasticsearch

Scripting module

The service supports scripting for Elasticsearch 5.x domains. The service does not support scriptingfor 1.5 or 2.3.

Supported scripting options include the following:• Painless• Lucene Expressions• Mustache

For Elasticsearch 5.5 and newer domains, Amazon ES supports stored scripts using the _scriptsendpoint. Elasticsearch 5.3 and 5.1 domains only support inline scripts. To learn more, see How touse scripts in the Elasticsearch documentation.

TCP transport

The service supports HTTP on port 80, but does not support TCP transport.


https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html

https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html



https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html


https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-scripting-using.html#modules-scripting-stored-scripts

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-scripting-using.html#modules-scripting-stored-scripts

Amazon Elasticsearch Service Developer GuideActions

Amazon Elasticsearch ServiceConfiguration API Reference

This reference describes the actions, data types, and errors in the Amazon Elasticsearch ServiceConfiguration API. The Configuration API is a REST API that you can use to create and configure AmazonES domains over HTTP. You also can use the AWS CLI and the console to configure Amazon ES domains.For more information, see Creating and Configuring Amazon ES Domains (p. 13).

• Actions (p. 124)• Data Types (p. 154)• Errors (p. 166)

ActionsThe following table provides a quick reference to the HTTP method required for each operation for theREST interface to the Amazon Elasticsearch Service Configuration API. The description of each operationalso includes the required HTTP method.

NoteAll configuration service requests must be signed. For more information, see Signing AmazonElasticsearch Service Requests (p. 35) in this guide and Signature Version 4 Signing Process inthe AWS General Reference.

Action HTTP Method

AddTags (p. 125) POST

CreateElasticsearchDomain (p. 126) POST

DeleteElasticsearchDomain (p. 130) DELETE

DeleteElasticsearchServiceRole (p. 132) DELETE

DescribeElasticsearchDomain (p. 133) GET

DescribeElasticsearchDomainConfig (p. 134)GET

DescribeElasticsearchDomains (p. 137) POST

DescribeElasticsearchInstanceTypeLimits (p. 140)GET

ListDomainNames (p. 143) GET

ListElasticsearchInstanceTypes (p. 144) GET

ListElasticsearchVersions (p. 146) GET

ListTags (p. 147) GET

RemoveTags (p. 148) POST

UpdateElasticsearchDomainConfig (p. 149) POST



Amazon Elasticsearch Service Developer GuideAddTags

AddTagsAttaches resource tags to an Amazon ES domain. For more information, see Tagging Amazon ESDomains (p. 55).

Syntax

POST /2015-01-01/tags{ "ARN": "<DOMAIN_ARN>", "TagList": [ { "Key": "<TAG_KEY>", "Value": "<TAG_VALUE>" } ]}

Request ParametersThis operation does not use request parameters.

Request Body

Parameter Data Type Required? Description

TagList TagList (p. 164) Yes List of resource tags

ARN ARN (p. 155) Yes Amazon Resource Name (ARN)for the Amazon ES domainto which you want to attachresource tags.

Response ElementsNot applicable. The AddTags operation does not return a data structure.

ErrorsThe AddTags operation can return any of the following errors:

• BaseException (p. )• LimitExceededException (p. )• ValidationException (p. )• InternalException (p. )

ExampleThe following example attaches a single resource tag with a tag key of project to the logs Amazon ESdomain:

Request

POST es.<AWS_REGION>.amazonaws.com/2015-01-01/tags


Amazon Elasticsearch Service Developer GuideCreateElasticsearchDomain

{ "ARN": "<DOMAIN_ARN>", "TagList": [ { "Key": "project", "Value": "trident" } ]}

Response

HTTP/1.1 200 OKx-amzn-RequestId: 5a6a5790-536c-11e5-9cd2-b36dbf43d89eContent-Type: application/jsonContent-Length: 0Date: Sat, 05 Sep 2015 01:20:55 GMT

CreateElasticsearchDomainCreates a new Amazon ES domain. For more information, see the section called “ Creating Amazon ESDomains” (p. 13).

NoteIf you attempt to create an Amazon ES domain and a domain with the same name alreadyexists, the API does not report an error. Instead, it returns details for the existing domain.

Syntax

POST /2015-01-01/es/domain{ "DomainName": "<DOMAIN_NAME>", "ElasticsearchVersion": "<VERSION>", "ElasticsearchClusterConfig": { "InstanceType": "<INSTANCE_TYPE>", "InstanceCount": <INSTANCE_COUNT>, "DedicatedMasterEnabled": "<TRUE|FALSE>", "DedicatedMasterCount": <INSTANCE_COUNT>, "DedicatedMasterType": "<INSTANCE_TYPE>", "ZoneAwarenessEnabled": "<TRUE|FALSE>" }, "EBSOptions": { "EBSEnabled": "<TRUE|FALSE>", "VolumeType": "<VOLUME_TYPE>", "VolumeSize": "<VOLUME_SIZE>", "Iops": "<VALUE>" }, "VPCOptions": { "SubnetIds": [ "<SUBNET_ID>" ], "SecurityGroupIds": [ "<SECURITY_GROUP_ID>" ] }, "AccessPolicies": "<ACCESS_POLICY_DOCUMENT>", "SnapshotOptions": { "AutomatedSnapshotStartHour": <START_HOUR> }, "LogPublishingOptions": { "SEARCH_SLOW_LOGS": { "CloudWatchLogsLogGroupArn":"<ARN>",



"Enabled":true }, "INDEX_SLOW_LOGS": { "CloudWatchLogsLogGroupArn":"<ARN>", "Enabled":true } }, "EncryptionAtRestOptions": { "Enabled": true, "KmsKeyId": "<KEY_ID>" }, "AdvancedOptions": { "rest.action.multi.allow_explicit_index": "<TRUE|FALSE>", "indices.fielddata.cache.size": "<PERCENTAGE_OF_HEAP>" }}

Request ParametersThis operation does not use HTTP request parameters.

Request Body


DomainName DomainName (p. 156) Yes Name of the Amazon ES domainto create.

ElasticsearchVersionString No Version of Elasticsearch. Ifnot specified, 1.5 is used asthe default. For the full listof supported versions, seethe section called “SupportedElasticsearch Versions” (p. 2).

ElasticsearchClusterConfigElasticsearchClusterConfig (p. 157)No Container for the clusterconfiguration of an Amazon ESdomain.

EBSOptions EBSOptions (p. 157) No Container for the parametersrequired to enable EBS-basedstorage for an Amazon ESdomain. For more information,see Configuring EBS-basedStorage (p. 21).

VPCOptions VPCOptions (p. 165) No Container for the values requiredto configure Amazon ES towork with a VPC. To learnmore, see VPC Support forAmazon Elasticsearch ServiceDomains (p. 96).

AccessPolicies String No IAM policy document specifyingthe access policies for the newAmazon ES domain. For moreinformation, see ConfiguringAccess Policies (p. 23).




SnapshotOptions SnapshotOptions (p. 163) No Container for parameters requiredto configure automated snapshotsof domain indices. For moreinformation, see ConfiguringSnapshots (p. 25).

AdvancedOptions AdvancedOptions (p. 154) No Key-value pairs to specifyadvanced configurationoptions. For more information,see Configuring AdvancedOptions (p. 26).

LogPublishingOptionsLogPublishingOptions (p. 161)No Key-value pairs to configure slowlog publishing.

EncryptionAtRestOptionsEncryptionAtRestOptions (p. 160)No Key-value pairs to enableencryption at rest.

Response Elements

Field Data Type Description

DomainStatus ElasticsearchDomainStatus (p. 158)Specifies the status and configuration of anew Amazon ES domain.

ErrorsCreateElasticsearchDomain can return any of the following errors:

• BaseException (p. )• DisabledOperationException (p. )• InternalException (p. )• InvalidTypeException (p. )• LimitExceededException (p. )• ResourceAlreadyExistsException (p. )• ValidationException (p. )

ExampleThis example demonstrates the following:

• Creates an Amazon ES domain named streaming-logs• Configures a cluster with six instances of the m3.medium.elasticsearch instance type and three

dedicated master nodes of the same type• Enables zone awareness• Configures VPC endpoints for the domain

Request



POST es.<AWS_REGION>.amazonaws.com/2015-01-01/es/domain{ "DomainName": "streaming-logs", "ElasticsearchVersion": "5.5", "ElasticsearchClusterConfig": { "InstanceType": "m3.medium.elasticsearch", "InstanceCount": 6, "DedicatedMasterEnabled": "true", "DedicatedMasterCount": 3, "DedicatedMasterType": "m3.medium.elasticsearch", "ZoneAwarenessEnabled": "true" }, "VPCOptions": { "SubnetIds": [ "subnet-87654321", "subnet-12345678" ] }, "EncryptionAtRestOptions": { "Enabled": true, "KmsKeyId": "1a2a3a4-1a2a-3a4a-5a6a-1a2a3a4a5a6a" }}

Response

HTTP/1.1 200 OKx-amzn-RequestId: 30b03e92-536f-11e5-9cd2-b36dbf43d89eContent-Type: application/jsonContent-Length: 645Date: Sat, 05 Sep 2015 01:41:15 GMT{ "DomainStatus": { "ARN": "arn:aws:es:us-west-1:123456789012:domain/streaming-logs", "AccessPolicies": "", "AdvancedOptions": { "rest.action.multi.allow_explicit_index": "true" }, "Created": true, "Deleted": false, "DomainId": "123456789012/streaming-logs", "DomainName": "streaming-logs", "EBSOptions": { "EBSEnabled": false, "EncryptionEnabled": null, "Iops": null, "VolumeSize": null, "VolumeType": null }, "ElasticsearchClusterConfig": { "DedicatedMasterCount": 3, "DedicatedMasterEnabled": true, "DedicatedMasterType": "m3.medium.elasticsearch", "InstanceCount": 6, "InstanceType": "m3.medium.elasticsearch", "ZoneAwarenessEnabled": true }, "ElasticsearchVersion": "5.5", "EncryptionAtRestOptions": { "Enabled": true, "KmsKeyId": "arn:aws:kms:us-west-1:123456789012:key/1a2a3a4-1a2a-3a4a-5a6a-1a2a3a4a5a6a" }, "Endpoint": null,


Amazon Elasticsearch Service Developer GuideDeleteElasticsearchDomain

"Endpoints": null, "Processing": true, "SnapshotOptions": { "AutomatedSnapshotStartHour": 0 }, "VPCOptions": { "AvailabilityZones": [ "us-west-1b", "us-west-1c" ], "SecurityGroupIds": [ "sg-12345678" ], "SubnetIds": [ "subnet-87654321", "subnet-12345678" ], "VPCId": "vpc-12345678" } }}

DeleteElasticsearchDomainDeletes an Amazon ES domain and all of its data. A domain cannot be recovered after it is deleted.

Syntax

DELETE /2015-01-01/es/domain/<DOMAIN_NAME>

Request Parameters


DomainName DomainName (p. 156) Yes Name of the Amazon ES domainthat you want to delete.

Request BodyThis operation does not use the HTTP request body.

Response Elements


DomainStatus ElasticsearchDomainStatus (p. 158)Specifies the configuration of thespecified Amazon ES domain.

ErrorsThe DeleteElasticsearchDomain operation can return any of the following errors:

• BaseException (p. )


Amazon Elasticsearch Service Developer GuideDeleteElasticsearchDomain

• InternalException (p. )• ResourceNotFoundException (p. )• ValidationException (p. )

ExampleThe following example deletes the weblogs domain:

Request

DELETE es.<AWS_REGION>.amazonaws.com/2015-01-01/es/domain/weblogs

Response

HTTP/1.1 200 OK{ "DomainStatus": { "ARN": "arn:aws:es:us-west-1:123456789012:domain/weblogs", "AccessPolicies": "", "AdvancedOptions": { "rest.action.multi.allow_explicit_index": "true" }, "Created": true, "Deleted": true, "DomainId": "123456789012/weblogs", "DomainName": "weblogs", "EBSOptions": { "EBSEnabled": false, "EncryptionEnabled": null, "Iops": null, "VolumeSize": null, "VolumeType": null }, "ElasticsearchClusterConfig": { "DedicatedMasterCount": 3, "DedicatedMasterEnabled": true, "DedicatedMasterType": "m3.medium.elasticsearch", "InstanceCount": 6, "InstanceType": "m3.medium.elasticsearch", "ZoneAwarenessEnabled": true }, "ElasticsearchVersion": "5.5", "EncryptionAtRestOptions": { "Enabled": true, "KmsKeyId": "arn:aws:kms:us-west-1:123456789012:key/1a2a3a4-1a2a-3a4a-5a6a-1a2a3a4a5a6a" }, "Endpoint": null, "Endpoints": null, "Processing": true, "SnapshotOptions": { "AutomatedSnapshotStartHour": 0 }, "VPCOptions": { "AvailabilityZones": [ "us-west-1b", "us-west-1c" ], "SecurityGroupIds": [ "sg-12345678" ],


Amazon Elasticsearch Service Developer GuideDeleteElasticsearchServiceRole

"SubnetIds": [ "subnet-87654321", "subnet-12345678" ], "VPCId": "vpc-12345678" } }}

DeleteElasticsearchServiceRoleDeletes the service-linked role between Amazon ES and Amazon EC2. This role gives Amazon ESpermissions to place VPC endpoints into your VPC. A service-linked role must be in place for domainswith VPC endpoints to be created or function properly.

NoteThis action only succeeds if no domains are using the service-linked role.

Syntax

DELETE /2015-01-01/es/role



Response ElementsNot applicable. The DeleteElasticsearchServiceRole operation does not return a data structure.

ErrorsDeleteElasticsearchServiceRole can return any of the following errors:

• BaseException (p. )• InternalException (p. )• ValidationException (p. )

ExampleThe following example demonstrates deletion of the service-linked role:

Request

DELETE es.<AWS_REGION>.amazonaws.com/2015-01-01/es/role

Response

If successful, this action provides no response.


Amazon Elasticsearch Service Developer GuideDescribeElasticsearchDomain

DescribeElasticsearchDomainDescribes the domain configuration for the specified Amazon ES domain, including the domain ID,domain service endpoint, and domain ARN.

Syntax

GET /2015-01-01/es/domain/<DOMAIN_NAME>

Request Parameters


DomainName DomainName (p. 156) Yes Name of theAmazon ESdomain that youwant to describe.


Response Elements


DomainStatus ElasticsearchDomainStatus (p. 158)Configuration of the specified AmazonES domain.

ErrorsDescribeElasticsearchDomain can return any of the following errors:

• BaseException (p. )• InternalException (p. )• ResourceNotFoundException (p. )• ValidationException (p. )

ExampleThe following example returns a description of the streaming-logs domain:

Request

GET es.<AWS_REGION>.amazonaws.com/2015-01-01/es/domain/streaming-logs

Response

{


Amazon Elasticsearch Service Developer GuideDescribeElasticsearchDomainConfig

"DomainStatus": { "ARN": "arn:aws:es:us-west-1:123456789012:domain/streaming-logs", "AccessPolicies": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:us-west-1:123456789012:domain/streaming-logs/*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"11.222.333.11\",\"11.222.333.12\",\"11.222.333.13\",\"11.222.333.14\",\"11.222.333.15\"]}}}]}", "AdvancedOptions": { "rest.action.multi.allow_explicit_index": "true" }, "Created": true, "Deleted": false, "DomainId": "123456789012/streaming-logs", "DomainName": "streaming-logs", "EBSOptions": { "EBSEnabled": true, "EncryptionEnabled": false, "Iops": null, "VolumeSize": 11, "VolumeType": "gp2" }, "ElasticsearchClusterConfig": { "DedicatedMasterCount": 2, "DedicatedMasterEnabled": false, "DedicatedMasterType": "m4.large.elasticsearch", "InstanceCount": 2, "InstanceType": "t2.small.elasticsearch", "ZoneAwarenessEnabled": false }, "ElasticsearchVersion": "5.5", "EncryptionAtRestOptions": { "Enabled": true, "KmsKeyId": "arn:aws:kms:us-west-1:123456789012:key/1a2a3a4-1a2a-3a4a-5a6a-1a2a3a4a5a6a" }, "Endpoint": "search-streaming-logs-oojmrbhufr27n44zdri52wukdy.us-west-1.es.amazonaws.com", "Endpoints": null, "Processing": false, "SnapshotOptions": { "AutomatedSnapshotStartHour": 8 }, "VPCOptions": null }}

DescribeElasticsearchDomainConfigDisplays the configuration of an Amazon ES domain.

Syntax

GET /2015-01-01/es/domain/<DOMAIN_NAME>/config

Request Parameters


DomainName DomainName (p. 156) Yes Name of the Amazon ESdomain.




Response Elements


DomainConfig ElasticsearchDomainConfig (p. 158)Configuration of the Amazon ESdomain.

ErrorsThe DescribeElasticsearchDomainConfig operation can return any of the following errors:

• BaseException (p. )• InternalException (p. )• ResourceNotFoundException (p. )

ExampleThe following example returns a description of the configuration of the logs domain:

Request

GET es.<AWS_REGION>.amazonaws.com/2015-01-01/es/domain/logs/config

Response

HTTP/1.1 200 OK{ "DomainConfig": { "AccessPolicies": { "Options": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::123456789012:root\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:us-west-1:123456789012:domain/logs/*\"}]}", "Status": { "CreationDate": 1499817484.04, "PendingDeletion": false, "State": "Active", "UpdateDate": 1500308955.652, "UpdateVersion": 17 } }, "AdvancedOptions": { "Options": { "indices.fielddata.cache.size": "", "rest.action.multi.allow_explicit_index": "true" }, "Status": { "CreationDate": 1499817484.04, "PendingDeletion": false, "State": "Active", "UpdateDate": 1499818054.108, "UpdateVersion": 5 }



}, "EBSOptions": { "Options": { "EBSEnabled": true, "EncryptionEnabled": false, "Iops": 0, "VolumeSize": 10, "VolumeType": "gp2" }, "Status": { "CreationDate": 1499817484.04, "PendingDeletion": false, "State": "Active", "UpdateDate": 1499818054.108, "UpdateVersion": 5 } }, "ElasticsearchClusterConfig": { "Options": { "DedicatedMasterCount": 2, "DedicatedMasterEnabled": false, "DedicatedMasterType": "m4.large.elasticsearch", "InstanceCount": 2, "InstanceType": "m4.large.elasticsearch", "ZoneAwarenessEnabled": false }, "Status": { "CreationDate": 1499817484.04, "PendingDeletion": false, "State": "Active", "UpdateDate": 1499966854.612, "UpdateVersion": 13 } }, "ElasticsearchVersion": { "Options": "5.5", "Status": { "PendingDeletion": false, "State": "Active", "CreationDate": 1436913638.995, "UpdateVersion": 6, "UpdateDate": 1436914324.278 }, "Options": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:us-east-1:123456789012:domain/logs/*\"}]}" }, "EncryptionAtRestOptions": { "Options": { "Enabled": true, "KmsKeyId": "arn:aws:kms:us-west-1:123456789012:key/1a2a3a4-1a2a-3a4a-5a6a-1a2a3a4a5a6a" }, "Status": { "CreationDate": 1509490412.757, "PendingDeletion": false, "State": "Active", "UpdateDate": 1509490953.717, "UpdateVersion": 6 } }, "LogPublishingOptions":{ "Status":{ "CreationDate":1502774634.546, "PendingDeletion":false, "State":"Processing",


Amazon Elasticsearch Service Developer GuideDescribeElasticsearchDomains

"UpdateDate":1502779590.448, "UpdateVersion":60 }, "Options":{ "INDEX_SLOW_LOGS":{ "CloudWatchLogsLogGroupArn":"arn:aws:logs:us-east-1:123456789012:log-group:sample-domain", "Enabled":true }, "SEARCH_SLOW_LOGS":{ "CloudWatchLogsLogGroupArn":"arn:aws:logs:us-east-1:123456789012:log-group:sample-domain", "Enabled":true } } }, "SnapshotOptions": { "Options": { "AutomatedSnapshotStartHour": 6 }, "Status": { "CreationDate": 1499817484.04, "PendingDeletion": false, "State": "Active", "UpdateDate": 1499818054.108, "UpdateVersion": 5 } }, "VPCOptions": { "Options": { "AvailabilityZones": [ "us-west-1b" ], "SecurityGroupIds": [ "sg-12345678" ], "SubnetIds": [ "subnet-12345678" ], "VPCId": "vpc-12345678" }, "Status": { "CreationDate": 1499817484.04, "PendingDeletion": false, "State": "Active", "UpdateDate": 1499818054.108, "UpdateVersion": 5 } } }}

DescribeElasticsearchDomainsDescribes the domain configuration for up to five specified Amazon ES domains. Information includesthe domain ID, domain service endpoint, and domain ARN.

Syntax

POST /2015-01-01/es/domain-info { "DomainNames": [ "<DOMAIN_NAME>",



"<DOMAIN_NAME>", ]}


Request Body

Field Data Type Required?Description

DomainNames DomainNameList (p. 156)Yes Array of Amazon ES domains in the followingformat:

{"DomainNames":["<Domain_Name>","<Domain_Name>"...]

Response Elements


DomainStatusList ElasticsearchDomainStatusList (p. 160)List that contains the status of eachrequested Amazon ES domain.

ErrorsThe DescribeElasticsearchDomains operation can return any of the following errors:

• BaseException (p. )• InternalException (p. )• ValidationException (p. )

ExampleThe following example returns a description of the logs and streaming-logs domains:

Request

POST es.<AWS_REGION>.amazonaws.com/2015-01-01/es/domain-info/{ "DomainNames": [ "logs", "streaming-logs" ]}

Response

HTTP/1.1 200 OK{



"DomainStatusList": [ { "ElasticsearchClusterConfig": { "DedicatedMasterEnabled": true, "InstanceCount": 3, "ZoneAwarenessEnabled": false, "DedicatedMasterType": "m3.medium.elasticsearch", "InstanceType": "m3.medium.elasticsearch", "DedicatedMasterCount": 3 }, "ElasticsearchVersion": "5.5", "EncryptionAtRestOptions": { "Enabled": true, "KmsKeyId": "arn:aws:kms:us-west-1:123456789012:key/1a2a3a4-1a2a-3a4a-5a6a-1a2a3a4a5a6a" }, "Endpoint": "search-streaming-logs-okga24ftzsbz2a2hzhsqw73jpy.us-east-1.es.example.com", "Created": true, "Deleted": false, "DomainName": "streaming-logs", "EBSOptions": { "EBSEnabled": false }, "VPCOptions": { "SubnetIds": [ "subnet-d1234567" ], "VPCId": "vpc-12345678", "SecurityGroupIds": [ "sg-123456789" ], "AvailabilityZones": [ "us-east-1" ] }, "SnapshotOptions": { "AutomatedSnapshotStartHour": 0 }, "DomainId": "123456789012/streaming-logs", "AccessPolicies": "", "Processing": false, "AdvancedOptions": { "rest.action.multi.allow_explicit_index": "true", "indices.fielddata.cache.size": "" }, "ARN": "arn:aws:es:us-east-1:123456789012:domain/streaming-logs" }, { "ElasticsearchClusterConfig": { "DedicatedMasterEnabled": true, "InstanceCount": 1, "ZoneAwarenessEnabled": false, "DedicatedMasterType": "search.m3.medium", "InstanceType": "search.m3.xlarge", "DedicatedMasterCount": 3 }, "ElasticsearchVersion": "5.5", "EncryptionAtRestOptions": { "Enabled": true, "KmsKeyId": "arn:aws:kms:us-west-1:123456789012:key/1a2a3a4-1a2a-3a4a-5a6a-1a2a3a4a5a6a" }, "Endpoint": "search-logs-p5st2kbt77diuihoqi6omd7jiu.us-east-1.es.example.com", "Created": true, "Deleted": false,


Amazon Elasticsearch Service Developer GuideDescribeElasticsearchInstanceTypeLimits

"DomainName": "logs", "EBSOptions": { "Iops": 4000, "VolumeSize": 512, "VolumeType": "io1", "EBSEnabled": true }, "VPCOptions": { "SubnetIds": [ "subnet-d1234567" ], "VPCId": "vpc-12345678", "SecurityGroupIds": [ "sg-123456789" ], "AvailabilityZones": [ "us-east-1" ] }, "SnapshotOptions": { "AutomatedSnapshotStartHour": 0 }, "DomainId": "123456789012/logs", "AccessPolicies": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:us-east-1:123456789012:domain/logs/*\"}]}", "Processing": false, "AdvancedOptions": { "rest.action.multi.allow_explicit_index": "true" }, "ARN": "arn:aws:es:us-east-1:123456789012:domain/logs" } ]}

DescribeElasticsearchInstanceTypeLimitsDescribes the instance count, storage, and master node limits for a given Elasticsearch version andinstance type.

Syntax

GET 2015-01-01/es/instanceTypeLimits/{ElasticsearchVersion}/{InstanceType}?domainName={DomainName}

Request Parameters


ElasticsearchVersionString Yes Elasticsearch version. For alist of supported versions, seethe section called “SupportedElasticsearch Versions” (p. 2).

InstanceType String Yes Instance type. To viewinstance types by region, seeAmazon Elasticsearch ServicePricing.






DomainName DomainName (p. 156) No The name of an existingdomain. Only specify if youneed the limits for an existingdomain.


Response Elements


LimitsByRole Map Map containing all applicable instancetype limits. "data" refers to data nodes."master" refers to dedicated masternodes.

ErrorsThe DescribeElasticsearchInstanceTypeLimits operation can return any of the following errors:

• BaseException (p. )• InternalException (p. )• InvalidTypeException (p. )• LimitExceededException (p. )• ResourceNotFoundException (p. )• ValidationException (p. )

ExampleThe following example returns a description of the logs and streaming-logs domains:

Request

GET es.<AWS_REGION>.amazonaws.com/2015-01-01/es/instanceTypeLimits/6.0/m4.large.elasticsearch

Response

HTTP/1.1 200 OK{ "LimitsByRole": { "data": { "AdditionalLimits": [ { "LimitName": "MaximumNumberOfDataNodesWithoutMasterNode", "LimitValues": [ "10" ] }



], "InstanceLimits": { "InstanceCountLimits": { "MaximumInstanceCount": 20, "MinimumInstanceCount": 1 } }, "StorageTypes": [ { "StorageSubTypeName": "standard", "StorageTypeLimits": [ { "LimitName": "MaximumVolumeSize", "LimitValues": [ "100" ] }, { "LimitName": "MinimumVolumeSize", "LimitValues": [ "10" ] } ], "StorageTypeName": "ebs" }, { "StorageSubTypeName": "io1", "StorageTypeLimits": [ { "LimitName": "MaximumVolumeSize", "LimitValues": [ "512" ] }, { "LimitName": "MinimumVolumeSize", "LimitValues": [ "35" ] }, { "LimitName": "MaximumIops", "LimitValues": [ "16000" ] }, { "LimitName": "MinimumIops", "LimitValues": [ "1000" ] } ], "StorageTypeName": "ebs" }, { "StorageSubTypeName": "gp2", "StorageTypeLimits": [ { "LimitName": "MaximumVolumeSize", "LimitValues": [ "512" ] }, {


Amazon Elasticsearch Service Developer GuideListDomainNames

"LimitName": "MinimumVolumeSize", "LimitValues": [ "10" ] } ], "StorageTypeName": "ebs" } ] }, "master": { "AdditionalLimits": [ { "LimitName": "MaximumNumberOfDataNodesSupported", "LimitValues": [ "100" ] } ], "InstanceLimits": { "InstanceCountLimits": { "MaximumInstanceCount": 5, "MinimumInstanceCount": 2 } }, "StorageTypes": null } }}

ListDomainNamesDisplays the names of all Amazon ES domains owned by the current user in the active region.

Syntax

GET /2015-01-01/domain



Response Elements


DomainNameList DomainNameList (p. 156) The names of all Amazon ESdomains owned by the currentuser.

ErrorsThe ListDomainNames operation can return any of the following errors:


Amazon Elasticsearch Service Developer GuideListElasticsearchInstanceTypes

• BaseException (p. )• ValidationException (p. )

ExampleThe following example lists all three domains owned by the current user:

Request

GET es.<AWS_REGION>.amazonaws.com/2015-01-01/domain

Response

{ "DomainNames": [ { "DomainName": "logs" }, { "DomainName": "streaming-logs" } ]}

ListElasticsearchInstanceTypesLists all Elasticsearch instance types that are supported for a given Elasticsearch version.

Syntax

GET 2015-01-01/es/instanceTypes/{ElasticsearchVersion}?domainName={DomainName}&maxResults={MaxResults}&nextToken={NextToken}

Request Parameters


ElasticsearchVersionString Yes The Elasticsearch version.

DomainName String No The Amazon ES domain name.

MaxResults Integer No Limits the number of results. Mustbe between 30 and 100.

NextToken String No Used for pagination. Onlynecessary if a previous API callproduced a result containingNextToken. Accepts a next-tokeninput to return results for the nextpage and provides a next-tokenoutput in the response, whichclients can use to retrieve moreresults.


Amazon Elasticsearch Service Developer GuideListElasticsearchInstanceTypes


Response Elements


ElasticsearchInstanceTypesList List of supported instance types for thegiven Elasticsearch version.

NextToken String Used for pagination. Only necessary ifa previous API call produced a resultcontaining NextToken. Accepts a next-token input to return results for the nextpage and provides a next-token outputin the response, which clients can use toretrieve more results.

ErrorsListElasticsearchInstanceTypes can return any of the following errors:


ExampleRequest

GET es.<AWS_REGION>.amazonaws.com/2015-01-01/es/instanceTypes/6.0

Response

{ "ElasticsearchInstanceTypes": [ "t2.small.elasticsearch", "t2.medium.elasticsearch", "r4.large.elasticsearch", "r4.xlarge.elasticsearch", "r4.2xlarge.elasticsearch", "r4.4xlarge.elasticsearch", "r4.8xlarge.elasticsearch", "r4.16xlarge.elasticsearch", "m4.large.elasticsearch", "m4.xlarge.elasticsearch", "m4.2xlarge.elasticsearch", "m4.4xlarge.elasticsearch", "m4.10xlarge.elasticsearch", "c4.large.elasticsearch", "c4.xlarge.elasticsearch", "c4.2xlarge.elasticsearch", "c4.4xlarge.elasticsearch",


Amazon Elasticsearch Service Developer GuideListElasticsearchVersions

"c4.8xlarge.elasticsearch" ], "NextToken": null}

ListElasticsearchVersionsLists all supported Elasticsearch versions on Amazon ES.

Syntax

GET 2015-01-01/es/versions?maxResults={MaxResults}&nextToken={NextToken}

Request Parameters


MaxResults Integer No Limits the number of results. Mustbe between 30 and 100.

NextToken String No Used for pagination. Onlynecessary if a previous API callproduced a result containingNextToken. Accepts a next-tokeninput to return results for the nextpage and provides a next-tokenoutput in the response, whichclients can use to retrieve moreresults.


Response Elements


ElasticsearchVersions List Lists all supported Elasticsearch versions.

NextToken String Used for pagination. Only necessary ifa previous API call produced a resultcontaining NextToken. Accepts a next-token input to return results for the nextpage and provides a next-token outputin the response, which clients can use toretrieve more results.

ErrorsListElasticsearchVersions can return any of the following errors:


Amazon Elasticsearch Service Developer GuideListTags


ExampleRequest

GET es.<AWS_REGION>.amazonaws.com/2015-01-01/es/versions

Response

{ "ElasticsearchVersions": [ "6.0", "5.5", "5.3", "5.1", "2.3", "1.5" ], "NextToken": null}

ListTagsDisplays all resource tags for an Amazon ES domain.

Syntax

GET /2015-01-01/tags?arn=<DOMAIN_ARN>

Request Parameters


ARN ARN (p. 155) Yes Amazon Resource Name (ARN) for theAmazon ES domain.


Response Elements


TagList TagList (p. 164) List of resource tags. For more information, see TaggingAmazon Elasticsearch Service Domains (p. 55).


Amazon Elasticsearch Service Developer GuideRemoveTags

ErrorsThe ListTags operation can return any of the following errors:


• ResourceNotFoundException (p. )

• ValidationException (p. )

• InternalException (p. )

ExampleThe following example lists the tags attached to the logs domain:

Request

GET es.<AWS_REGION>.amazonaws.com/2015-01-01/tags?arn=arn:aws:es:us-west-1:123456789012:domain/logs

Response

HTTP/1.1 200 OK{ "TagList": [ { "Key": "Environment", "Value": "MacOS" }, { "Key": "project", "Value": "trident" } ]}

RemoveTagsRemoves the specified resource tags from an Amazon ES domain.

Syntax

POST es.<AWS_REGION>.amazonaws.com/2015-01-01/tags-removal{ "ARN": "<DOMAIN_ARN>", "TagKeys": [ "<TAG_KEY>", "<TAG_KEY>", ... ]}



Amazon Elasticsearch Service Developer GuideUpdateElasticsearchDomainConfig

Request Body


ARN ARN (p. 155) Yes Amazon Resource Name (ARN) of anAmazon ES domain. For more information,see Identifiers for IAM Entities in Using AWSIdentity and Access Management.

TagKeys TagKey (p. 164) Yes List of tag keys for resource tags that youwant to remove from an Amazon ES domain.

Response ElementsNot applicable. The RemoveTags operation does not return a response element.

ErrorsThe RemoveTags operation can return any of the following errors:




ExampleThe following example deletes a resource tag with a tag key of project from the Amazon ES domain:

Request

POST /2015-01-01/tags-removal{ "ARN": "<DOMAIN_ARN>", "TagKeys": [ "project" ]}

This operation does not return a response element.

UpdateElasticsearchDomainConfigModifies the configuration of an Amazon ES domain, such as the instance type and the number ofinstances. You only need to specify the values that you want to update.

Syntax

POST /2015-01-01/es/domain/<DOMAIN_NAME>/config{ "ElasticsearchClusterConfig": { "InstanceType": "<INSTANCE_TYPE>",


http://docs.aws.amazon.com/IAM/latest/UserGuide/index.html?Using_Identifiers.html


"Instance_Count": <INSTANCE_COUNT>, "DedicatedMasterEnabled": "<TRUE|FALSE>", "DedicatedMasterCount": <INSTANCE_COUNT>, "DedicatedMasterType": "<INSTANCE_COUNT>", "ZoneAwarenessEnabled": "<TRUE|FALSE>" }, "EBSOptions": { "EBSEnabled": "<TRUE|FALSE>", "VolumeType": "<VOLUME_TYPE>", "VolumeSize": "<VOLUME_SIZE>", "Iops": "<VALUE>" }, "VPCOptions": { "SubnetIds": [ "<SUBNET_ID>" ], "SecurityGroupIds": [ "<SECURITY_GROUP_ID>" ] }, "AccessPolicies": "<ACCESS_POLICY_DOCUMENT>", "SnapshotOptions": { "AutomatedSnapshotStartHour": <START_HOUR>, "AdvancedOptions": { "rest.action.multi.allow_explicit_index": "<TRUE|FALSE>", "indices.fielddata.cache.size": "<PERCENTAGE_OF_HEAP>" } }, "LogPublishingOptions": { "SEARCH_SLOW_LOGS": { "CloudWatchLogsLogGroupArn":"<ARN>", "Enabled":true }, "INDEX_SLOW_LOGS": { "CloudWatchLogsLogGroupArn":"<ARN>", "Enabled":true } }}


Request Body

Parameter Data Type Required?Description

DomainName DomainName (p. 156) Yes Name of the Amazon ES domainfor which you want to update theconfiguration.

ElasticsearchClusterConfigElasticsearchClusterConfig (p. 157)No Desired changes to the clusterconfiguration, such as the instancetype and number of EC2 instances.

EBSOptions EBSOptions (p. 157) No Type and size of EBS volumesattached to data nodes.

VPCOptions VPCOptions (p. 165) No Container for the values required toconfigure Amazon ES to work with a



Parameter Data Type Required?Description

VPC. To learn more, see VPC Supportfor Amazon Elasticsearch ServiceDomains (p. 96).

SnapshotOptions SnapshotOptions (p. 163) No Hour during which the service takesan automated daily snapshot of theindices in the Amazon ES domain.

AdvancedOptions AdvancedOptions (p. 154) No Key-value pairs to specify advancedconfiguration options. For moreinformation, see ConfiguringAdvanced Options (p. 26).

AccessPolicies String No Specifies the access policies forthe Amazon ES domain. For moreinformation, see Configuring AccessPolicies (p. 23).

LogPublishingOptionsLogPublishingOptions (p. 161)No Key-value string pairs to configureslow log publishing.

Response Elements


DomainConfig String Status of the Amazon ES domain after updating itsconfiguration.

Errors

UpdateElasticsearchDomainConfig can return any of the following errors:



• InvalidTypeException (p. )

• LimitExceededException (p. )


Example

The following example configures the daily automatic snapshot for the streaming-logs domain tooccur during the hour starting at 3:00 AM GMT:

Request

POST es.<AWS_REGION>.amazonaws.com/2015-01-01/es/domain/streaming-logs/config{ "SnapshotOptions": { "AutomatedSnapshotStartHour": 3 }



}

Response

{ "DomainConfig": { "AccessPolicies": { "Options": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:us-west-1:123456789012:domain/streaming-logs/*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"11.222.333.11\",\"11.222.333.12\",\"11.222.333.13\",\"11.222.333.14\",\"11.222.333.15\"]}}}]}", "Status": { "CreationDate": 1502213150.329, "PendingDeletion": false, "State": "Active", "UpdateDate": 1502213466.93, "UpdateVersion": 6 } }, "AdvancedOptions": { "Options": { "rest.action.multi.allow_explicit_index": "true" }, "Status": { "CreationDate": 1502213150.329, "PendingDeletion": false, "State": "Active", "UpdateDate": 1502213466.93, "UpdateVersion": 6 } }, "EBSOptions": { "Options": { "EBSEnabled": true, "EncryptionEnabled": false, "Iops": null, "VolumeSize": 11, "VolumeType": "gp2" }, "Status": { "CreationDate": 1502213150.329, "PendingDeletion": false, "State": "Active", "UpdateDate": 1502929669.653, "UpdateVersion": 23 } }, "ElasticsearchClusterConfig": { "Options": { "DedicatedMasterCount": 2, "DedicatedMasterEnabled": false, "DedicatedMasterType": "m4.large.elasticsearch", "InstanceCount": 2, "InstanceType": "t2.small.elasticsearch", "ZoneAwarenessEnabled": false }, "Status": { "CreationDate": 1502213150.329, "PendingDeletion": false, "State": "Active", "UpdateDate": 1502929669.653, "UpdateVersion": 23 } },



"ElasticsearchVersion": { "Options": "5.5", "Status": { "CreationDate": 1502213150.329, "PendingDeletion": false, "State": "Active", "UpdateDate": 1502213466.93, "UpdateVersion": 6 } }, "EncryptionAtRestOptions": { "Options": { "Enabled": true, "KmsKeyId": "arn:aws:kms:us-west-1:123456789012:key/1a2a3a4-1a2a-3a4a-5a6a-1a2a3a4a5a6a" }, "Status": { "CreationDate": 1509490412.757, "PendingDeletion": false, "State": "Active", "UpdateDate": 1509490953.717, "UpdateVersion": 6 } }, "LogPublishingOptions":{ "Options":{ "INDEX_SLOW_LOGS":{ "CloudWatchLogsLogGroupArn":"arn:aws:logs:us-east-1:123456789012:log-group:sample-domain", "Enabled":true }, "SEARCH_SLOW_LOGS":{ "CloudWatchLogsLogGroupArn":"arn:aws:logs:us-east-1:123456789012:log-group:sample-domain", "Enabled":true } }, "Status":{ "CreationDate":1502774634.546, "PendingDeletion":false, "State":"Processing", "UpdateDate":1502779590.448, "UpdateVersion":60 } }, "SnapshotOptions": { "Options": { "AutomatedSnapshotStartHour": 3 }, "Status": { "CreationDate": 1502213150.329, "PendingDeletion": false, "State": "Active", "UpdateDate": 1503093165.447, "UpdateVersion": 25 } }, "VPCOptions": { "Options": { "AvailabilityZones": null, "SecurityGroupIds": null, "SubnetIds": null, "VPCId": null }, "Status": { "CreationDate": 1503093165.597,


Amazon Elasticsearch Service Developer GuideData Types

"PendingDeletion": false, "State": "Active", "UpdateDate": 1503093165.597, "UpdateVersion": 25 } } }}

Data TypesThis section describes the data types used by the REST Configuration API.

AdvancedOptionsKey-value string pairs to specify advanced Elasticsearch configuration options.


rest.action.multi.allow_explicit_indexKey-value pair:

rest.action.multi.allow_explicit_index=<true|false>

Specifies whetherexplicit references toindices are allowedinside the body ofHTTP requests. If youwant to configureaccess policies fordomain sub-resources,such as specific indicesand domain APIs,you must disable thisproperty. For moreinformation, see URL-based Access Control.For more informationabout access policiesfor sub-resources, seeConfiguring AccessPolicies (p. 23).

indices.fielddata.cache.sizeKey-value pair:

indices.fielddata.cache.size=<percentage_of_heap>

Specifies thepercentage of Javaheap space that isallocated to field data.By default, this settingis unbounded. Formore information,see Field Data inthe Elasticsearchdocumentation.

indices.query.bool.max_clause_countKey-value pair:

indices.query.bool.max_clause_count=<int>

Specifies the maximumnumber of clausesallowed in a Luceneboolean query. 1024is the default. Queries





Amazon Elasticsearch Service Developer GuideAdvancedOptionsStatus


with more than thepermitted numberof clauses result ina TooManyClauseserror. To learn more,see the Lucenedocumentation.

AdvancedOptionsStatusStatus of an update to advanced configuration options for an Amazon ES domain.


Options AdvancedOptions (p. 154) Key-value pairs to specifyadvanced Elasticsearchconfiguration options.

Status OptionStatus (p. 162) Status of an update to advancedconfiguration options for anAmazon ES domain.

ARN


ARN String Amazon Resource Name (ARN) ofan Amazon ES domain. For moreinformation, see IAM ARNs in theAWS Identity and Access Managementdocumentation.

CreateElasticsearchDomainRequestContainer for the parameters required by the CreateElasticsearchDomain service operation.


DomainName DomainName (p. 156) Name of the Amazon ES domain tocreate.

ElasticsearchClusterConfigElasticsearchClusterConfig (p. 157)Container for the cluster configuration ofan Amazon ES domain.

EBSOptions EBSOptions (p. 157) Container for the parameters required toenable EBS-based storage for an AmazonES domain. For more information, seeConfiguring EBS-based Storage (p. 21).




http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-arns

Amazon Elasticsearch Service Developer GuideDomainID


AccessPolicies String IAM policy document specifying theaccess policies for the new AmazonES domain. For more information, seeConfiguring Access Policies (p. 23).

SnapshotOptions SnapshotOptionsStatus (p. 163)Container for parameters required toconfigure automated snapshots ofdomain indices. For more information, seeConfiguring Snapshots (p. 25).

VPCOptions VPCOptions (p. 165) Container for the values required toconfigure Amazon ES to work with a VPC.

LogPublishingOptions LogPublishingOptions (p. 161)Key-value string pairs to configure slowlog publishing.

SnapshotOptions SnapshotOptionsStatus (p. 163)Container for parameters required toconfigure automated snapshots ofdomain indices. For more information, seeConfiguring Snapshots (p. 25).

AdvancedOptions AdvancedOptionsStatus (p. 155)Key-value pairs to specify advancedconfiguration options.

DomainID

Data Type Description

String Unique identifier for an Amazon ES domain

DomainNameName of an Amazon ES domain.


String Name of an Amazon ES domain. Domain names are unique across alldomains owned by the same account within an AWS region. Domainnames must start with a lowercase letter and must be between 3 and 28characters. Valid characters are a-z (lowercase only), 0-9, and – (hyphen).

DomainNameListString of Amazon ES domain names.


String Array Array of Amazon ES domains in the following format:

["<Domain_Name>","<Domain_Name>"...]


Amazon Elasticsearch Service Developer GuideEBSOptions

EBSOptionsContainer for the parameters required to enable EBS-based storage for an Amazon ES domain. For moreinformation, see Configuring EBS-based Storage (p. 21).


EBSEnabled Boolean Indicates whether EBS volumes are attached to datanodes in an Amazon ES domain.

VolumeType String Specifies the type of EBS volumes attached to datanodes.

VolumeSize String Specifies the size of EBS volumes attached to datanodes.

Iops String Specifies the baseline input/output (I/O) performanceof EBS volumes attached to data nodes. Applicableonly for the Provisioned IOPS EBS volume type.

ElasticsearchClusterConfigContainer for the cluster configuration of an Amazon ES domain.


InstanceType String Instance type of data nodes in the cluster.

InstanceCount Integer Number of instances in the cluster.

DedicatedMasterEnabled Boolean Indicates whether dedicated master nodes areenabled for the cluster. True if the cluster will usea dedicated master node. False if the cluster willnot. For more information, see About DedicatedMaster Nodes (p. 90).

DedicatedMasterType String Amazon ES instance type of the dedicated masternodes in the cluster.

DedicatedMasterCount Integer Number of dedicated master nodes in the cluster.

ZoneAwarenessEnabled Boolean Indicates whether zone awareness is enabled.Zone awareness allocates the nodes and replicaindex shards belonging to a cluster across twoAvailability Zones in the same region.

If you enable zone awareness, you must havean even number of instances in the instancecount, and you also must use the Amazon ESConfiguration API to replicate your data for yourElasticsearch cluster.For more information, see Enabling ZoneAwareness (p. 45).


Amazon Elasticsearch Service Developer GuideElasticsearchDomainConfig

ElasticsearchDomainConfigContainer for the configuration of an Amazon ES domain.


ElasticsearchVersion String Elasticsearch version.

ElasticsearchClusterConfig ElasticsearchClusterConfig (p. 157)Container for the clusterconfiguration of an Amazon ESdomain.

EBSOptions EBSOptions (p. 157) Container for EBS optionsconfigured for an Amazon ESdomain.

AccessPolicies String Specifies the access policiesfor the Amazon ES domain.For more information,see Configuring AccessPolicies (p. 23).

SnapshotOptions SnapshotOptionsStatus (p. 163)Hour during which the servicetakes an automated dailysnapshot of the indices in theAmazon ES domain. For moreinformation, see ConfiguringSnapshots (p. 25).

VPCOptions VPCDerivedInfoStatus (p. 165)The currentVPCOptions (p. 165) for thedomain and the status of anyupdates to their configuration.

LogPublishingOptions LogPublishingOptions (p. 161)Key-value pairs to configureslow log publishing.

AdvancedOptions AdvancedOptionsStatus (p. 154)Key-value pairs to specifyadvanced configurationoptions.

EncryptionAtRestOptions EncryptionAtRestOptionsStatus (p. 161)Key-value pairs to enableencryption at rest.

ElasticsearchDomainStatusContainer for the contents of a DomainStatus data structure.


DomainID DomainID (p. 156) Unique identifier for an AmazonES domain.

DomainName DomainName (p. 156) Name of an Amazon ES domain.Domain names are uniqueacross all domains owned by the


Amazon Elasticsearch Service Developer GuideElasticsearchDomainStatus


same account within an AWSRegion. Domain names muststart with a lowercase letterand must be between 3 and 28characters. Valid characters area-z (lowercase only), 0-9, and –(hyphen).

ARN ARN (p. 155) Amazon Resource Name (ARN)of an Amazon ES domain. Formore information, see Identifiersfor IAM Entities in Using AWSIdentity and Access Management.

Created Boolean Status of the creation of anAmazon ES domain. Trueif creation of the domain iscomplete. False if domaincreation is still in progress.

Deleted Boolean Status of the deletion of anAmazon ES domain. Trueif deletion of the domain iscomplete. False if domaindeletion is still in progress.

Endpoint ServiceUrl (p. 163) Domain-specific endpoint usedto submit index, search, anddata upload requests to anAmazon ES domain.

Endpoints EndpointsMap (p. 161) The key-value pair that exists ifthe Amazon ES domain uses VPCendpoints.

Processing Boolean Status of a change in theconfiguration of an AmazonES domain. True if theservice is still processing theconfiguration changes. Falseif the configuration changeis active. You must wait for adomain to reach active statusbefore submitting index, search,and data upload requests.

ElasticsearchVersion String Elasticsearch version.

ElasticsearchClusterConfig ElasticsearchClusterConfig (p. 157)Container for the clusterconfiguration of an Amazon ESdomain.




Amazon Elasticsearch Service Developer GuideElasticsearchDomainStatusList


EBSOptions EBSOptions (p. 157) Container for the parametersrequired to enable EBS-basedstorage for an Amazon ESdomain. For more information,see Configuring EBS-basedStorage (p. 21).

AccessPolicies String IAM policy document specifyingthe access policies for the newAmazon ES domain. For moreinformation, see ConfiguringAccess Policies (p. 23).

SnapshotOptions SnapshotOptions (p. 163) Container for parametersrequired to configure the timeof daily automated snapshots ofAmazon ES domain indices.

VPCOptions VPCDerivedInfo (p. 165) Information that AmazonES derives based onVPCOptions (p. 165) for thedomain.

LogPublishingOptions LogPublishingOptions (p. 161)Key-value pairs to configureslow log publishing.

AdvancedOptions AdvancedOptions (p. 154) Key-value pairs to specifyadvanced configuration options.

EncryptionAtRestOptions EncryptionAtRestOptions (p. 160)Key-value pairs to enableencryption at rest.

ElasticsearchDomainStatusListList that contains the status of each specified Amazon ES domain.


DomainStatusList ElasticsearchDomainStatus (p. 158)List that contains the statusof each specified Amazon ESdomain.

EncryptionAtRestOptionsSpecifies whether the domain should encrypt data at rest, and if so, the AWS Key Management Service(KMS) key to use. Can only be used to create a new domain, not update an existing one.


Enabled Boolean Specify true to enableencryption at rest.


Amazon Elasticsearch Service Developer GuideEncryptionAtRestOptionsStatus


KmsKeyId String The KMS key ID.Takes the form1a2a3a4-1a2a-3a4a-5a6a-1a2a3a4a5a6a.

EncryptionAtRestOptionsStatusStatus of the domain's encryption at rest options.


Options EncryptionAtRestOptions (p. 160) Encryption at rest options forthe domain.

Status OptionStatus (p. 162) Status of the domain'sencryption at rest options.

EndpointsMapThe key-value pair that contains the VPC endpoint. Only exists if the Amazon ES domain resides in a VPC.


Endpoints Key-value stringpair: "vpc":"<VPC_ENDPOINT>"

The VPC endpoint for the domain.

LogPublishingOptionsSpecifies whether the Amazon ES domain publishes the Elasticsearch slow logs to Amazon CloudWatch.You still have to enable the collection of slow logs using the Elasticsearch REST API. To learn more, seethe section called “Setting Elasticsearch Logging Thresholds” (p. 30).


Endpoints Key-value: "vpc":"<VPC_ENDPOINT>"

The VPC endpoint for thedomain.

INDEX_SLOW_LOGS Key-value Two key-value pairs thatdefine the CloudWatch loggroup and whether or not theElasticsearch index slow logshould be published there:

"CloudWatchLogsLogGroupArn":"arn:aws:logs:us-east-1:264071961897:log-group:sample-domain","Enabled":true



https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html#index-slow-log

Amazon Elasticsearch Service Developer GuideLogPublishingOptionsStatus


SEARCH_SLOW_LOGS Key-value Two key-value pairs thatdefine the CloudWatch loggroup and whether or not theElasticsearch search slow logshould be published there:

"CloudWatchLogsLogGroupArn":"arn:aws:logs:us-east-1:264071961897:log-group:sample-domain","Enabled":true

LogPublishingOptionsStatusStatus of an update to the configuration of the slow log publishing options for the Amazon ES domain.


Options LogPublishingOptions (p. 161) Slow log publishing optionsfor the domain

Status OptionStatus (p. 162) Status of an update tosnapshot options for anAmazon ES domain

OptionStateState of an update to advanced options for an Amazon ES domain.


OptionStatus String One of three valid values:

• RequiresIndexDocuments• Processing• Active

OptionStatusStatus of an update to configuration options for an Amazon ES domain.


CreationDate Timestamp Date and time when the Amazon ESdomain was created

UpdateDate Timestamp Date and time when the Amazon ESdomain was updated


https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html#search-slow-log

Amazon Elasticsearch Service Developer GuideServiceURL


UpdateVersion Integer Whole number that specifies the latestversion for the entity

State OptionState (p. 162) State of an update to configuration optionsfor an Amazon ES domain

PendingDeletion Boolean Indicates whether the service is processinga request to permanently delete theAmazon ES domain and all of its resources

ServiceURLDomain-specific endpoint used to submit index, search, and data upload requests to an Amazon ESdomain.


ServiceURL String Domain-specific endpoint used to submit index, search, anddata upload requests to an Amazon ES domain

SnapshotOptionsContainer for parameters required to configure the time of daily automated snapshots of the indices inan Amazon ES domain.


AutomatedSnapshotStartHour Integer Hour during which the service takes anautomated daily snapshot of the indicesin the Amazon ES domain

SnapshotOptionsStatusStatus of an update to the configuration of the daily automated snapshot for an Amazon ES domain.


Options SnapshotOptions (p. 163) Container for parameters required toconfigure the time of daily automatedsnapshots of indices in an Amazon ESdomain

Status OptionStatus (p. 162) Status of an update to snapshot optionsfor an Amazon ES domain


Amazon Elasticsearch Service Developer GuideTag

Tag


Key TagKey (p. 164) Required name of the tag. Tag keys must beunique for the Amazon ES domain to which theyare attached. For more information, see TaggingAmazon Elasticsearch Service Domains (p. 55).

Value TagValue (p. 164) Optional string value of the tag. Tag values canbe null and do not have to be unique in a tag set.For example, you can have a key-value pair in atag set of project/Trinity and cost-center/Trinity.

TagKey


Key String Name of the tag. String can have up to 128characters.

TagList


Tag Tag (p. 164) Resource tag attached to an Amazon ES domain.

TagValue


Value String Holds the value for a TagKey. String can have upto 256 characters.

VPCDerivedInfo


VPCId String The ID for your VPC. Amazon VPC generates thisvalue when you create a VPC.

SubnetIds StringList A list of subnet IDs associated with the VPCendpoints for the domain. To learn more, seeVPCs and Subnets in the Amazon VPC User Guide.


http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html

Amazon Elasticsearch Service Developer GuideVPCDerivedInfoStatus


AvailabilityZones StringList The list of availability zones associated with theVPC subnets. To learn more, see VPC and SubnetBasics in the Amazon VPC User Guide.

SecurityGroupIds StringList The list of security group IDs associated with theVPC endpoints for the domain. To learn more, seeSecurity Groups for your VPC in the Amazon VPCUser Guide.

VPCDerivedInfoStatus


Options VPCDerivedInfo (p. 164)Information that Amazon ES derives based onVPCOptions (p. 165) for the domain.

Status OptionStatus (p. 162)Status of an update to VPC configuration optionsfor an Amazon ES domain.

VPCOptions


SubnetIds StringList A list of subnet IDs associated with the VPCendpoints for the domain. If your domain haszone awareness enabled, you need to providetwo subnet IDs, one per zone. Otherwise, youonly need to provide one. To learn more, seeVPCs and Subnets in the Amazon VPC User Guide.

SecurityGroupIds StringList The list of security group IDs associated withthe VPC endpoints for the domain. If you do notprovide a security group ID, Amazon ES uses thedefault security group for the VPC. To learn more,see Security Groups for your VPC in the AmazonVPC User Guide.

VPCOptionsStatus


Options VPCOptions (p. 165) Container for the values required to configureAmazon ES to work with a VPC.

Status OptionStatus (p. 162)Status of an update to VPC configuration optionsfor an Amazon ES domain.


http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html#vpc-subnet-basics

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html#vpc-subnet-basics


http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html


Amazon Elasticsearch Service Developer GuideErrors

ErrorsAmazon ES throws the following errors:

Exception Description

BaseException Thrown for all service errors. Contains the HTTP status code ofthe error.

ValidationException Thrown when the HTTP request contains invalid input or ismissing required input. Returns HTTP status code 400.

DisabledOperationException Thrown when the client attempts to perform an unsupportedoperation. Returns HTTP status code 409.

InternalException Thrown when an error internal to the service occurs whileprocessing a request. Returns HTTP status code 500.

InvalidTypeException Thrown when trying to create or access an Amazon ES domainsub-resource that is either invalid or not supported. ReturnsHTTP status code 409.

LimitExceededException Thrown when trying to create more than the allowed numberand type of Amazon ES domain resources and sub-resources.Returns HTTP status code 409.

ResourceNotFoundException Thrown when accessing or deleting a resource that does notexist. Returns HTTP status code 400.

ResourceAlreadyExistsExceptionThrown when a client attempts to create a resource that alreadyexists in an Amazon ES domain. Returns HTTP status code 400.


Amazon Elasticsearch Service Developer GuideCluster and Instance Limits

Amazon Elasticsearch Service LimitsThe following tables show limits for Amazon ES resources, including the number of instances per cluster,the minimum and maximum sizes for EBS volumes, and network limits.

Cluster and Instance LimitsThe following table shows Amazon ES limits for clusters and instances.

Clusters and Instances Limit

Maximum Number of Instances(Instance Count) per Cluster

20 (except for T2 instance types, which have a maximum of 10)

NoteThe default limit is 20 instances per domain. Torequest an increase up to 100 instances per domain(for Elasticsearch 2.3 or later), create a case with theAWS Support Center.For more information about requesting an increase,see AWS Service Limits.

Maximum Number of DedicatedMaster Nodes

5

NoteYou can use a T2 instance type as a dedicated masternode only if the instance count is 10 or fewer.

Smallest Instance Type Supported byElasticsearch Version

t2.micro.elasticsearch (versions 1.5 and 2.3) andt2.small.elasticsearch (version 5.x and 6.0).

For a list of the instance types that Amazon ES supports, see Supported Instance Types (p. 114).

EBS Volume Size LimitsThe following table shows the minimum and maximum sizes for EBS volumes for each instance type thatAmazon ES supports. See Amazon Elasticsearch Service Pricing for information on which instance typesoffer instance storage.

NoteIf you select magnetic storage under EBS volume type when creating your domain, maximumvolume size is 100 GB for all instance types except t2.micro, t2.small, and t2.medium. For themaximum sizes listed in the following table, select one of the SSD options.

Instance Type Minimum EBS Size Maximum EBS Size

t2.micro.elasticsearch 10 GB 35 GB

t2.small.elasticsearch 10 GB 35 GB

t2.medium.elasticsearch 10 GB 35 GB

m3.medium.elasticsearch 10 GB 100 GB



http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html


Amazon Elasticsearch Service Developer GuideEBS Volume Size Limits


m3.large.elasticsearch 10 GB 512 GB

m3.xlarge.elasticsearch 10 GB 512 GB

m3.2xlarge.elasticsearch 10 GB 512 GB

m4.large.elasticsearch 10 GB 512 GB

m4.xlarge.elasticsearch 10 GB 1 TB*

m4.2xlarge.elasticsearch 10 GB 1.5 TB*



c4.large.elasticsearch 10 GB 100 GB

c4.xlarge.elasticsearch 10 GB 512 GB

c4.2xlarge.elasticsearch 10 GB 1 TB*

c4.4xlarge.elasticsearch 10 GB 1.5 TB*

c4.8xlarge.elasticsearch 10 GB 1.5 TB*

r3.large.elasticsearch 10 GB 512 GB

r3.xlarge.elasticsearch 10 GB 512 GB

r3.2xlarge.elasticsearch 10 GB 512 GB



r4.large.elasticsearch 10 GB 1 TB*

r4.xlarge.elasticsearch 10 GB 1.5 TB*

r4.2xlarge.elasticsearch 10 GB 1.5 TB*




i2.xlarge.elasticsearch 10 GB 512 GB

i2.2xlarge.elasticsearch 10 GB 512 GB

i3.large.elasticsearch N/A N/A

i3.xlarge.elasticsearch N/A N/A

i3.2xlarge.elasticsearch N/A N/A




Amazon Elasticsearch Service Developer GuideNetwork Limits



* 512 GB is the maximum volume size that is supported with Elasticsearch version 1.5.

Network LimitsThe following table shows the maximum size of HTTP request payloads.

Instance Type Maximum Size of HTTP Request Payloads

t2.micro.elasticsearch 10 MB

t2.small.elasticsearch 10 MB

t2.medium.elasticsearch10 MB

m3.medium.elasticsearch10 MB

m3.large.elasticsearch 10 MB

m3.xlarge.elasticsearch100 MB

m3.2xlarge.elasticsearch100 MB

m4.large.elasticsearch 10 MB

m4.xlarge.elasticsearch100 MB




c4.large.elasticsearch 10 MB

c4.xlarge.elasticsearch100 MB

c4.2xlarge.elasticsearch100 MB



r3.large.elasticsearch 10 MB

r3.xlarge.elasticsearch100 MB

r3.2xlarge.elasticsearch100 MB



r4.large.elasticsearch 100 MB

r4.xlarge.elasticsearch100 MB


Amazon Elasticsearch Service Developer GuideJava Process Limit

Instance Type Maximum Size of HTTP Request Payloads





i2.xlarge.elasticsearch100 MB

i2.2xlarge.elasticsearch100 MB

i3.large.elasticsearch 100 MB

i3.xlarge.elasticsearch100 MB





Java Process LimitAmazon ES limits Java processes to a heap size of 32 GB. Advanced users can specify the percentageof the heap used for field data. For more information, see the section called “Configuring AdvancedOptions” (p. 26) and the section called “JVM OutOfMemoryError” (p. 108).


https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html

Amazon Elasticsearch Service Developer GuideService-Linked Role Permissions for Amazon ES

Using Service-Linked Roles forAmazon ES

Amazon Elasticsearch Service uses AWS Identity and Access Management (IAM) service-linked roles. Aservice-linked role is a unique type of IAM role that is linked directly to Amazon ES. Service-linked rolesare predefined by Amazon ES and include all the permissions that the service requires to call other AWSservices on your behalf.

A service-linked role makes setting up Amazon ES easier because you don’t have to manually add thenecessary permissions. Amazon ES defines the permissions of its service-linked roles, and unless definedotherwise, only Amazon ES can assume its roles. The defined permissions include the trust policy and thepermissions policy, and that permissions policy cannot be attached to any other IAM entity.

You can delete a service-linked role only after first deleting its related resources. This protects yourAmazon ES resources because you can't inadvertently remove permission to access the resources.

For information about other services that support service-linked roles, see AWS Services That Work withIAM and look for the services that have Yes in the Service-Linked Role column. Choose a Yes with a linkto view the service-linked role documentation for that service.

Service-Linked Role Permissions for Amazon ESAmazon ES uses the service-linked role named AWSServiceRoleForAmazonElasticsearchService.

The AWSServiceRoleForAmazonElasticsearchService service-linked role trusts the following services toassume the role:

• es.amazonaws.com

The role permissions policy allows Amazon ES to complete the following actions on the specifiedresources:

• Action: ec2:CreateNetworkInterface on *• Action: ec2:DeleteNetworkInterface on *• Action: ec2:DescribeNetworkInterfaces on *• Action: ec2:ModifyNetworkInterfaceAttribute on *• Action: ec2:DescribeSecurityGroups on *• Action: ec2:DescribeSubnets on *

You must configure permissions to allow an IAM entity (such as a user, group, or role) to create, edit, ordelete a service-linked role. For more information, see Service-Linked Role Permissions in the IAM UserGuide.

Creating a Service-Linked Role for Amazon ESYou don't need to manually create a service-linked role. When you create a VPC access domain using theAWS Management Console, Amazon ES creates the service-linked role for you. In order for this automaticcreation to succeed, you must have permissions for the iam:CreateServiceLinkedRole action.


http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html#iam-term-service-linked-role

http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html

http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html

http://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#service-linked-role-permissions

Amazon Elasticsearch Service Developer GuideEditing a Service-Linked Role for Amazon ES

If you delete this service-linked role and then need to create it again, you can use the same process torecreate the role in your account.

You can also use the IAM console, the IAM CLI, or the IAM API to create a service-linked role manually.For more information, see Creating a Service-Linked Role in the IAM User Guide.

Editing a Service-Linked Role for Amazon ESAmazon ES does not allow you to edit the AWSServiceRoleForAmazonElasticsearchService service-linkedrole. After you create a service-linked role, you cannot change the name of the role because variousentities might reference the role. However, you can edit the description of the role using IAM. For moreinformation, see Editing a Service-Linked Role in the IAM User Guide.

Deleting a Service-Linked Role for Amazon ESIf you no longer need to use a feature or service that requires a service-linked role, we recommendthat you delete that role. That way you don’t have an unused entity that is not actively monitored ormaintained. However, you must clean up your service-linked role before you can manually delete it.

Cleaning Up a Service-Linked RoleBefore you can use IAM to delete a service-linked role, you must first confirm that the role has no activesessions and remove any resources used by the role.

To check whether the service-linked role has an active session in the IAM console

1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

2. In the navigation pane of the IAM console, choose Roles. Then choose the name (not the check box)of the AWSServiceRoleForAmazonElasticsearchService role.

3. On the Summary page for the selected role, choose the Access Advisor tab.4. On the Access Advisor tab, review recent activity for the service-linked role.

NoteIf you are unsure whether Amazon ES is using theAWSServiceRoleForAmazonElasticsearchService role, you can try to delete the role. If theservice is using the role, then the deletion fails and you can view the regions where the roleis being used. If the role is being used, then you must wait for the session to end before youcan delete the role. You cannot revoke the session for a service-linked role.

To remove Amazon ES resources used by the AWSServiceRoleForAmazonElasticsearchService

1. Sign in to the AWS Management Console and open the Amazon ES console.2. Delete any domains that list VPC under the Endpoint column.

Manually Delete a Service-Linked RoleUse the Amazon ES configuration API to delete the AWSServiceRoleForAmazonElasticsearchServiceservice-linked role. For more information, see the section called“DeleteElasticsearchServiceRole” (p. 132).


http://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#create-service-linked-role

http://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#edit-service-linked-role

https://console.aws.amazon.com/iam/

https://console.aws.amazon.com/iam/


Document History for AmazonElasticsearch Service

This topic describes important changes to the documentation for Amazon Elasticsearch Service (AmazonES).

Relevant Dates to this History:

• Current product version—2015-01-01• Latest product release—19 December 2017• Latest documentation update—9 January 2018

Change Description Release Date

Petabyte Scale Amazon ES now supports i3 instance types and totaldomain storage of up to 1.5 PB. To learn more, see PetabyteScale (p. 94).

19 December2017

Encryption of Data atRest

Amazon ES now supports encryption of data at rest. To learnmore, see Encryption at Rest (p. 103).

7 December2017

Elasticsearch 6.0Support

Amazon Elasticsearch Service now supports Elasticsearchversion 6.0. For a summary of changes, see the Elasticsearch6.0 release notes. For migration considerations andinstructions, see Migrating to a Different ElasticsearchVersion (p. 80).

6 December2017

VPC Support Amazon Elasticsearch Service now lets you launch domainswithin an Amazon Virtual Private Cloud. VPC support providesan additional layer of security and simplifies communicationsbetween Amazon ES and other services within a VPC. To learnmore, see VPC Support (p. 96).

17 October2017

Slow Logs Publishing Amazon Elasticsearch Service now supports the publishing ofslow logs to CloudWatch Logs. To learn more, see the sectioncalled “Configuring Slow Logs” (p. 28).

16 October2017


Amazon Elasticsearch Service now supports Elasticsearchversion 5.5.

For new feature summaries, see the Amazon announcement ofavailability, the Elasticsearch 5.5 release announcement, andthe Kibana 5.5 release announcement.

You can now restore automated snapshots without contactingAWS Support and store scripts using the Elasticsearch_scripts API.

7 September2017


Amazon Elasticsearch Service added support for Elasticsearchversion 5.3.

1 June 2017


https://www.elastic.co/guide/en/elasticsearch/reference/current/release-notes-6.0.0.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/release-notes-6.0.0.html

https://aws.amazon.com/about-aws/whats-new/2017/09/elasticsearch-5_5-now-available-on-amazon-elasticsearch-service/

https://www.elastic.co/blog/elasticsearch-5-5-0-released

https://www.elastic.co/blog/kibana-5-5-0-released


Change Description Release Date

More Instances andEBS Capacity perCluster

Amazon Elasticsearch Service now supports up to 100 nodesand 150 TB EBS capacity per cluster.

5 April 2017

Canada (Central) andEU (London) Support

Amazon Elasticsearch Service added support for the followingregions: Canada (Central), ca-central-1 and EU (London),eu-west-2.

20 March2017

More Instances andLarger EBS Volumes

Amazon Elasticsearch Service added support for moreinstances and larger EBS volumes.

21 February2017



30 January2017

Support for thePhonetic AnalysisPlugin

Amazon Elasticsearch Service now provides built-inintegration with the Phonetic Analysis plugin, which allowsyou to run “sounds-like” queries on your data.

22 December2016

US East (Ohio)Support

Amazon Elasticsearch Service added support for the followingregion: US East (Ohio), us-east-2. For a list of regionssupported by Amazon Elasticsearch Service, see AWS Regionsand Endpoints in the AWS General Reference.

17 October2016

New PerformanceMetric

Amazon Elasticsearch Service added a performance metric,ClusterUsedSpace.

29 July 2016



27 July 2016

Asia Pacific (Mumbai)Support

Amazon Elasticsearch Service added support for the followingregion: Asia Pacific (Mumbai), ap-south-1. For a list ofregions supported by Amazon Elasticsearch Service, see AWSRegions and Endpoints in the AWS General Reference.

27 June 2016

More Instances perCluster

Amazon Elasticsearch Service increased the maximum numberof instances (instance count) per cluster from 10 to 20.

18 May 2016

Asia Pacific (Seoul)Support

Amazon Elasticsearch Service added support for the followingregion: Asia Pacific (Seoul), ap-northeast-2. For a list ofregions supported by Amazon Elasticsearch Service, see AWSRegions and Endpoints in the AWS General Reference.

28 January2016

Amazon ElasticsearchService

Initial release. 1 October2015









AWS GlossaryFor the latest AWS terminology, see the AWS Glossary in the AWS General Reference.


http://docs.aws.amazon.com/general/latest/gr/glos-chap.html