Kinesis Producers

Question 1

What are the 4 options to send data into a Kinesis Stream?

Answer 1

• Kinesis SDK
• Kinesis Producer Library (KPL)
• Kinesis Agent
• 3rd party libraries

Question 2

What is the Kinesis Producer Library (KPL)?

Answer 2

It is a more advanced way to send data to the stream

Question 3

What is the Kinesis Agent?

Answer 3

It is an agent that runs on a Linux server

Question 4

What are some examples of 3rd party libraries to connect to Kinesis?

Answer 4

• Spark
• Log4J
• Appenders
• Flume
• Kafka Connect
• NiFi

Question 5

When would you decide to use Kinesis SDK?

Answer 5

A case with low throughput, higher latency, simple API using AWS Lambda etc

Question 6

What would a “ProvisionedThroughputExceeded” exception indicate?

Answer 6

It indicates you have exceeded you data sending limits

Question 7

What are the 3 possible solutions to resolve “ProvisionedThroughputExceeded”?

Answer 7

• Retry with backoff
• Increase shards (scaling)
• Ensure your partition key is good

Question 8

What are the 8 features of the Kinesis Producer Library (KPL)?

Answer 8

• Easy to use and highly configurable with C++/Java
• Used for building high performance, long-running producers
• Automated and configurable retry mechanism
• Synchronous or Asynchronous API
• Submits records to Cloudwatch for monitoring
• 2 Batch features enabled by default
• Compression must be enabled by the user
• KPL records must be decoded by KCL

Question 9

What are the 2 Kinesis Producer Library (KCL) Batch features enabled by default?

Answer 9

• Collect records and write to multiple shards in the same PutRecords API call
• Aggregate records

Question 10

What are the 2 things that happen when Kinesis Producer Library (KPL) Aggregates records?

Answer 10

• Store multiple records in 1 record, allowing you to go over the 1000 records per limit
• Increase payload size and improve throughput (max 1MB/s)

Question 11

What is an example of Kinesis Producer Library (KPL) Batching records?

Answer 11

The limit in Kinesis is 1MB. But if KPL finds 3 records that have a combined total less than that, it will combine them into 1 message.

Question 12

How does KPL know how long to wait before sending batched records?

Answer 12

You can set this with the “RecordMaxBufferedTime”, which defaults to 100ms

Question 13

What are the 4 features of Kinesis Agent?

Answer 13

• Write from multiple directories and write to multiple streams
• Routing feature based on directory/log file
• Pre-processing data before sending to streams (csv to json etc)
• Handles log rotation and retry