AWS Kinesis

Question 1

What is AWS Kinesis

Answer 1

Kinesis is a managed “data streaming” service. It is great for application logs, metrics, IoT, real time big data, streaming processing frameworks (Spark, NiFi)

The data streamed is automatically replicated synchronously to 3 AZ

Question 2

What are the 3 applications under Kinesis and what are they used for

Answer 2

• Kinesis Streams: low latency streaming ingest at scale
• Kinesis Analytics: perform real-time analytics on streams using SQL
• Kinesis Firehose: load streams into S3, Redshift, ElasticSearch & Splunk

Question 3

How does Kinesis Streams work

Answer 3

Streams are divided in ordered Shards / Partitions

Producers -> Shards (1,2,3) -> Consumers

Data retention is 24 hours by default, can go up to 365 days
Ability to reprocess / replay data
Multiple applications can consume the same stream
* Real-time processing with scale of throughput
* Once data is inserted in Kinesis, it can’t be deleted (immutability). This is the main difference between SQS and Kinesis. Messages in SQS can be deleted

Question 4

Explain Kinesis Streams Shards

Answer 4

Two modes for capacity:
* On-demand: no capacity planning, Kinesis scales shards automatically
* Provisioned: you manage the shards over time

• Batching available or send messages one at a time
• The number of shards can evolve over time (reshard / merge)
• Records are ordered per shard

Question 5

What are Kinesis producers and consumers

Answer 5

Kinesis Producers
* AWS SDK: simple producer
* Kinesis Producer Library (KPL):
batch, compression, retries, C++,
Java
* Kinesis Agent: For EC2s

Kinesis Consumers
* AWS SDK: simple consumer
* Lambda: (through Event source mapping)
* KCL: checkpointing, coordinated reads

Question 6

What are some limits with Kinesis Data Streams

Answer 6

Producer:
* 1MB/s or 1000 messages/s at write PER SHARD
* “ProvisionedThroughputException” otherwise

Consumer Classic:
* 2MB/s at read PER SHARD across all consumers
* 5 API calls per second PER SHARD across all consumers

Consumer Enhanced Fan-Out:
* 2MB/s at read PER SHARD, PER ENHANCED CONSUMER
* No API calls needed (push model)

Data Retention:
* 24 hours data retention by default
* Can be extended to 365 days