Amazon Kinesis Data Analytics | Authoring Application Code

Question 1

How is my input stream exposed to my SQL code?

Authoring Application Code

Amazon Kinesis Data Analytics | Analytics

Answer 1

Kinesis Data Analytics applies your specified schema and inserts your data into one or more in-application streams for streaming sources, and a single SQL table for reference sources. The default number of in-application streams is the one that meets the needs of most of your use cases. You should increase this if you find that your application is not keeping up with the latest data in your source stream as defined by CloudWatch metric MillisBehindLatest. The number of in-application streams required is impacted by both the amount of throughput in your source stream and your query complexity. The parameter for specifying the number of in-application streams that are mapped to your source stream is called input parallelism.

Question 2

What does my application code look like?

Authoring Application Code

Amazon Kinesis Data Analytics | Analytics

Answer 2

Application code is a series of SQL statements that process input and produce output. These SQL statements operate on in-application streams and reference tables. An in-application stream is like a continuously updating table on which you can perform the SELECT and INSERT SQL operations. Your configured sources and destinations are exposed to your SQL code through in-application streams. You can also create additional in-application streams to store intermediate query results.

You can use the following pattern to work with in-application streams:

Always use a SELECT statement in the context of an INSERT statement. When you select rows, you insert results into another in-application stream.

Use an INSERT statement in the context of a pump. You use a pump to make an INSERT statement continuous, and write to an in-application stream.

You use a pump to tie in-application streams together, selecting from one in-application stream and inserting into another in-application stream.

The following SQL code provides a simple, working application:

CREATE OR REPLACE STREAM “DESTINATION_SQL_STREAM” (

ticker_symbol VARCHAR(4),

change DOUBLE,

price DOUBLE);

CREATE OR REPLACE PUMP “STREAM_PUMP” AS INSERT INTO “DESTINATION_SQL_STREAM

SELECT STREAM ticker_symbol, change, price

FROM “SOURCE_SQL_STREAM_001”;

For more information about application code, see Application Code in the Kinesis Data Analytics Developer Guide.

Question 3

How does Kinesis Data Analytics help me with writing SQL code?

Authoring Application Code

Amazon Kinesis Data Analytics | Analytics

Answer 3

Kinesis Data Analytics includes a library of analytics templates for common use cases including streaming filters, tumbling time windows, and anomaly detection. You can access these templates from the SQL editor in the AWS Management Console. After you create an application and navigate to the SQL editor, the templates are available in the upper-left corner of the console.