Amazon Elasticsearch Service | Administration

Question 1

Is there a limit on the amount of EBS storage that can be allocated to an Amazon Elasticsearch domain?

Administration

Amazon Elasticsearch Service | Analytics

Answer 1

Yes. Amazon Elasticsearch Service supports one EBS volume (max size of 1.5 TB) per instance associated with a domain. With the default maximum of 20 data nodes allowed per Amazon Elasticsearch Service domain, you can allocate about 30 TB of EBS storage to a single domain. You can request a service limit increase up to 100 instances per domain by creating a case with the AWS Support Center. With 100 instances, you can allocate about 150 TB of EBS storage to a single domain.

Question 2

Can programs running on servers in my own data center access my Amazon Elasticsearch domains?

Administration

Amazon Elasticsearch Service | Analytics

Answer 2

Yes. The programs with public Internet access can access Amazon Elasticsearch Service domains through a public endpoint. If your data center is already connected to Amazon VPC through Direct Connect or SSH tunneling, you can also use VPC access. In both cases, you can configure IAM policies and security groups to allow programs running on servers outside of AWS to access your Amazon Elasticsearch domains. Click here for more information about signed requests.

Question 3

How can I migrate data from my existing Elasticsearch cluster to a new Amazon Elasticsearch domain?

Administration

Amazon Elasticsearch Service | Analytics

Answer 3

To migrate data from an existing Elasticsearch cluster you should create a snapshot of an existing Elasticsearch cluster, and store the snapshot in your Amazon S3 bucket. Then you can create a new Amazon Elasticsearch domain and load data from the snapshot into the newly created Amazon Elasticsearch domain using the Elasticsearch restore API.

Question 4

How can I scale an Amazon Elasticsearch domain?

Administration

Amazon Elasticsearch Service | Analytics

Answer 4

Amazon Elasticsearch Service allows you to control the scaling of your Amazon Elasticsearch domains using the console, API, and CLI. You can scale your Amazon Elasticsearch domain by adding, removing, or modifying instances or storage volumes depending on your application needs. Amazon Elasticsearch Service is integrated with Amazon CloudWatch to provide metrics about the state of your Amazon Elasticsearch domains to enable you to make appropriate scaling decisions for your domains.

Question 5

Does scaling my Amazon Elasticsearch domain require downtime?

Administration

Amazon Elasticsearch Service | Analytics

Answer 5

No. Scaling your Amazon Elasticsearch domain by adding or modifying instances, and storage volumes is an online operation that does not require any downtime.

Question 6

What options does Amazon Elasticsearch Service provide for node failures?

Administration

Amazon Elasticsearch Service | Analytics

Answer 6

Amazon Elasticsearch Service automatically detects node failures and replaces the node. The service will acquire new instances, and will then redirect Elasticsearch requests and document updates to the new instances. In the event that the node cannot be replaced, customers will be able to use any snapshots they have of their cluster to restart the domain with preloaded data.

Question 7

Does Amazon Elasticsearch Service support cross-zone replication?

Administration

Amazon Elasticsearch Service | Analytics

Answer 7

Yes. Customers can enable Zone Awareness for their Amazon Elasticsearch domains either at domain creation time or by modifying a live domain. When Zone Awareness is enabled, Amazon Elasticsearch Service will distribute the instances supporting the domain across two different Availability Zones. Then, if replication is enabled in the Elasticsearch engine, Elasticsearch will allocate replicas of the domain across these different instances enabling cross-zone replication.

Question 8

Does Amazon Elasticsearch Service expose any performance metrics through Amazon CloudWatch?

Administration

Amazon Elasticsearch Service | Analytics

Answer 8

Yes. Amazon Elasticsearch Service exposes several performance metrics through Amazon CloudWatch including number of nodes, cluster health, searchable documents, EBS metrics (if applicable), CPU, memory and disk utilization for data and master nodes. Please refer to the service documentation for a full listing of available CloudWatch metrics.

Question 9

I wish to perform security analysis or operational troubleshooting of my Amazon Elasticsearch Service deployment. Can I get a history of all the Amazon Elasticsearch Service API calls made on my account?

Administration

Amazon Elasticsearch Service | Analytics

Answer 9

Yes. AWS CloudTrail is a web service that records AWS API calls for your account and delivers log files to you. The AWS API call history produced by AWS CloudTrail enables security analysis, resource change tracking, and compliance auditing. Learn more about AWS CloudTrail at the AWS CloudTrail detail page, and turn it on via CloudTrail’s AWS Management Console home page.

Question 10

What is a snapshot?

Administration

Amazon Elasticsearch Service | Analytics

Answer 10

A snapshot is a copy of your Amazon Elasticsearch domain at a moment in time.

Question 11

Why would I need snapshots?

Administration

Amazon Elasticsearch Service | Analytics

Answer 11

Creating snapshots can be useful in case of data loss caused by node failure, as well as the unlikely event of a hardware failure. You can use snapshots to recover your Amazon Elasticsearch domain with preloaded data or to create a new Amazon Elasticsearch domain with preloaded data. Another common reason to use backups is for archiving purposes. Snapshots are stored in Amazon S3.

Question 12

Does Amazon Elasticsearch Service provide automated snapshots?

Administration

Amazon Elasticsearch Service | Analytics

Answer 12

Yes. By default, Amazon Elasticsearch Service will automatically create daily snapshots of each Amazon Elasticsearch domain. The daily snapshots are setup to occur between midnight and 1AM UTC. Customers will also be able to modify the timing of the automated snapshot to better suit their needs.

Question 13

Can I change the default settings for the automated daily snapshot provided by Amazon Elasticsearch Service?

Administration

Amazon Elasticsearch Service | Analytics

Answer 13

Yes. You will be able to change the timing of the automated daily snapshot to suit your application schedule.

Question 14

How long are the automated daily snapshots stored by Amazon Elasticsearch Service?

Administration

Amazon Elasticsearch Service | Analytics

Answer 14

Amazon Elasticsearch Service will retain the last 14 days worth of automated daily snapshots.

Question 15

Is there a charge for the automated daily snapshots?

Administration

Amazon Elasticsearch Service | Analytics

Answer 15

There is no additional charge for the automated daily snapshots. The snapshots are stored for free in an Amazon Elasticsearch Service S3 bucket and will be made available for node recovery purposes.

Question 16

Can I create additional snapshots of my Amazon Elasticsearch domains as needed?

Administration

Amazon Elasticsearch Service | Analytics

Answer 16

Yes. You can use the Elasticsearch snapshot API to create additional manual snapshots in addition to the daily-automated snapshots created by Amazon Elasticsearch Service. The manual snapshots are stored in your S3 bucket and will incur relevant Amazon S3 usage charges.

Question 17

Can snapshots created by the manual snapshot process be used to recover a domain in the event of a failure?

Administration

Amazon Elasticsearch Service | Analytics

Answer 17

Yes. Customers can create a new Amazon Elasticsearch domain and load data from the snapshot into the newly created Amazon Elasticsearch domain using the Elasticsearch restore API.

Question 18

What happens to my snapshots when I delete my Amazon Elasticsearch domain?

Administration

Amazon Elasticsearch Service | Analytics

Answer 18

The daily snapshots retained by Amazon Elasticsearch Service will be deleted as part of domain deletion. Before deleting a domain, you should consider creating a snapshot of the domain in your own S3 buckets using the manual snapshot process. The snapshots stored in your S3 bucket will not be affected if you delete your Amazon Elasticsearch domain.

Question 19

What are slow logs?

Administration

Amazon Elasticsearch Service | Analytics

Answer 19

Slow logs are log files that help track the performance of various stages in an operation. Elasticsearch exposes two kinds of slow logs:

Index Slow Logs – These logs provide insights into the indexing process and can be used to fine-tune the index setup.

Search Slow Logs – These logs provide insights into how fast or slow queries and fetches are performing. These logs help fine tune the performance of any kind of search operation on Elasticsearch.

For complete details on Elasticsearch slow logs, please refer to Elasticsearch documentation.

Question 20

How can I enable slow logs on Amazon ES?

Administration

Amazon Elasticsearch Service | Analytics

Answer 20

Slows logs can be enabled via the click of a button from the Console or via our CLI and APIs. For more details please refer to our documentation.

Question 21

Can I only enable slow logs for specific indices?

Administration

Amazon Elasticsearch Service | Analytics

Answer 21

Yes. You can update the settings for a specific index to enable or disable slow logs for it. For more details refer to our documentation.

Question 22

Does turning on slow logs in Amazon ES automatically enable logging for all indexes?

Administration

Amazon Elasticsearch Service | Analytics

Answer 22

No. Turning on slow logs in Amazon ES enables the option to publish the generated logs to Amazon CloudWatch Logs for indices in the given domain. However, in order to generate the logs you have to update the settings for one or more indices to start the logging process. For more details on setting the index configuration for enabling slow logs, please refer to our documentation.

Question 23

If I turn off the Slow Logs in Amazon ES, does it mean that log files are no longer being generated?

Administration

Amazon Elasticsearch Service | Analytics

Answer 23

No. The generation of log files are dependent on the index settings. To turn off generation of the log files you have to update the index configuration. For more details on setting the index configuration for enabling slow logs, see our documentation.

Question 24

Can I adjust the granularity of logging?

Administration

Amazon Elasticsearch Service | Analytics

Answer 24

Yes. Elasticsearch exposes multiple levels of logging. You need to set the appropriate level in the configuration for your index. For more details on setting the index configuration for enabling slow logs, see our documentation.

Question 25

Does enabling Slow Logs cost me anything?

Administration

Amazon Elasticsearch Service | Analytics

Answer 25

When Slow Logs are enabled, Amazon ES starts publishing the generated logs to Amazon CloudWatch Logs. Amazon ES does not charge for enabling slow logs. However, standard CloudWatch charges apply.

Question 26

Is there any limit on the size of each log entry?

Administration

Amazon Elasticsearch Service | Analytics

Answer 26

Yes. Each log entry made into CloudWatch will be limited to 255,000 characters. If your log entry is bigger than that, it will be truncated to 255,000 characters.

Question 27

What is the recommended best practice for using slow logs?

Administration

Amazon Elasticsearch Service | Analytics

Answer 27

Slow logs are only needed when you want to troubleshoot your indexes or fine-tune performance. The recommended approach is to only enable logging for those indexes for which you need additional performance insights. Also, once the investigation is done, you should turn off logging so that you don’t incur any additional costs on account of it. For more details, see our documentation.

Question 28

How can I consume logs from CloudWatch Logs?

Administration

Amazon Elasticsearch Service | Analytics

Answer 28

CloudWatch offers multiple ways to consume logs. You can view log data, export it to S3, or process it in real time. To learn more, see the CloudWatch Logs developer guide.

Question 29

Are slow logs available for all versions of Elasticsearch supported by Amazon ES?

Administration

Amazon Elasticsearch Service | Analytics

Answer 29

Yes. slow logs can be enabled for all versions of Elasticsearch supported by Amazon ES. However, there are slight differences in the way log settings can be specified for each version of Elasticsearch. Please refer to our documentation for more details.