Amazon Kinesis Data Analytics | Troubleshooting Kinesis Data Analytics Applications

Question 1

What are the best practices associated for building and managing my Kinesis Data Analytics applications?

Troubleshooting Kinesis Data Analytics Applications

Amazon Kinesis Data Analytics | Analytics

Answer 1

For information about best practices, see the Best Practices section of the Kinesis Data Analytics Developer Guide, which covers managing applications, defining input schema, connecting to outputs, and authoring application code.

Question 2

How do I get a particular SQL statement to work correctly?

Troubleshooting Kinesis Data Analytics Applications

Amazon Kinesis Data Analytics | Analytics

Answer 2

For details, see Example Applications in the Kinesis Data Analytics Developer Guide, which has a number of SQL examples that you can use. In addition, the Kinesis Data Analytics SQL Reference provides a detailed guide to authoring streaming SQL statements. If you are still running into issues, we recommend that you ask a question on the Amazon Kinesis Forums.

Question 3

Kinesis Data Analytics was unable to detect or discover my schema. How can I use Kinesis Data Analytics?

Troubleshooting Kinesis Data Analytics Applications

Amazon Kinesis Data Analytics | Analytics

Answer 3

For other UTF-8 encoded data that does not use a delimiter, uses a different delimiter than CSV, or in cases where the discovery API did not fully discover the schema, you can define a schema by hand or use string manipulation functions to structure your data. For more information, see Using the Schema Discovery Feature and Related Editing in the Kinesis Data Analytics Developer Guide.

Question 4

What are the important parameters I should monitor to make sure my application is running correctly?

Troubleshooting Kinesis Data Analytics Applications

Amazon Kinesis Data Analytics | Analytics

Answer 4

The most important parameter to monitor is the CloudWatch metric, MillisBehindLatest, which represents how far behind from the current time you are reading from the stream. This metric provides an effective mechanism to determine whether you are processing records from the source stream fast enough. You should set up a CloudWatch Alarm to trigger if you fall behind more than one hour (this number depends on your use case and can be adjusted as needed). You can learn more in the Best Practices section of the Kinesis Data Analytics Developer Guide.