Amazon Kinesis Data Analytics | Configuring Input

Question 1

What destinations are supported?

Configuring Input

Amazon Kinesis Data Analytics | Analytics

Answer 1

Kinesis Data Analytics supports up to four destinations per application. You can persist SQL results to Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service (through an Amazon Kinesis Data Firehose), and Amazon Kinesis Data Streams. You can write to a destination not directly supported by Kinesis Data Analytics by sending SQL results to Amazon Kinesis Data Streams, and leveraging its integration with AWS Lambda to send to a destination of your choice.

Question 2

How do I set up a streaming data source?

Configuring Input

Amazon Kinesis Data Analytics | Analytics

Answer 2

A streaming data source can be an Amazon Kinesis data stream or an Amazon Kinesis Data Firehose delivery stream. Your Kinesis Data Analytics application continuously reads new data from streaming data sources as it arrives in real time. The data is made accessible in your SQL code through an in-application stream. An in-application stream acts like a SQL table because you can create, insert, and select from it. However, the difference is that an in-application stream is continuously updated with new data from the streaming data source.

You can use the AWS Management Console to add a streaming data source. You can learn more about sources in the Configuring Application Input section of the Kinesis Data Analytics Developer Guide.

Question 3

How do I set up a reference data source?

Configuring Input

Amazon Kinesis Data Analytics | Analytics

Answer 3

A reference data source can be an Amazon S3 object. Your Kinesis Data Analytics application reads the S3 object in its entirety when it starts running. The data is made accessible in your SQL code through a table. The most common use case for using a reference data source is to enrich the data coming from the streaming data source using a SQL JOIN.

Using the AWS CLI, you can add a reference data source by specifying the S3 bucket, object, IAM role, and associated schema. Kinesis Data Analytics loads this data when you start the application, and reloads it each time you make any update API call.

Question 4

What data formats are supported?

Configuring Input

Amazon Kinesis Data Analytics | Analytics

Answer 4

Kinesis Data Analytics detects the schema and automatically parses UTF-8 encoded JSON and CSV records using the DiscoverInputSchema API. This schema is applied to the data read from the stream as part of the insertion into an in-application stream.

For other UTF-8 encoded data that does not use a delimiter, uses a different delimiter than CSV, or in cases were the discovery API did not fully discover the schema, you can define a schema using the interactive schema editor or use string manipulation functions to structure your data. For more information, see Using the Schema Discovery Feature and Related Editing in the Kinesis Data Analytics Developer Guide.