[DEVELOPER] Advanced Kinesis

Question 1

What is the retention period for Kinesis Data Streams?

Answer 1

1 - 365 days

Question 2

How can you delete data in Kinesis Data Streams without processing it?

Answer 2

You can’t. Kinesis Data Streams data is immutable

Question 3

How can you ensure record ordering in Kinesis Data Streams?

Answer 3

Use the same partition key. Data that shares the same parition key go to the same shard.

Question 4

What is the I/O performance of Kinesis Data Streams Provisioned Capacity Mode?

Answer 4

Each shard gets:
- 1 MB/s in (or 1000 records per second)
- 2 MB/s out

Question 5

What is the pricing model for Kinesis Data Streams Provisioned Capacity Mode?

Answer 5

You pay per shard provisioned per hour

Question 6

What info does Kinesis Data Streams On-Demand Capacity Mode use to determine how it sets the capacity?

Answer 6

You get automatic scaling based on observed throughput peak during the last 30 days.

Default is 4 MB/s

Question 7

What is pricing model for Kinesis Data Streams On-Demand Capacity Mode?

Answer 7

Pay per stream per hour & data in/out per GB

Question 8

In what use case would you use Kinesis Data Streams Provisioned Mode over On-Demand Mode?

Answer 8

When you know your capacity ahead of time.

Question 9

Suppose you want to send streaming data from a VPC endpoint but you don’t want to go through the internet. How can you accomplish this?

Answer 9

Use Kinesis! VPC endpoints are available for Kinesis to access within VPC

Question 10

How does encryption work for Kinesis Data Streams?

Answer 10

Encryption at rest using KMS
Encryption in flight using HTTPS

Question 11

What is the API Call for a producer to send a record to Kinesis Data Streams?

Answer 11

PutRecord

Question 12

You are using Kinesis Data Streams with multiple producers and multiple shards and repeatedly get ProvisionedThroughputExceeded errors on an individual shard. What can you do to address the problem?

Answer 12

• Use highly distributed partition keys, maybe you have a hot partition getting too many messages
• Implement exponential backoff with retries
• Increase the number of shard (shard splitting)

Question 13

You are using Kinesis Data Streams with 4 consumers all reading from the same shard in the Shared (Classic) Fan-out consumer pattern. What is the read throughput of each consumer?

Answer 13

0.5 MB/sec

Classic KDS Fan-out is 2MB/s per shard across all consumers

Question 14

You are using Kinesis Data Streams with 4 consumers all reading from the same shard in the Enhanced Fan-out consumer pattern. What is the read throughput of each consumer?

Answer 14

2 MB/secc

Enhanced KDS fan-out is 2MB/s per shard per consumer

Question 15

How is data transferred from shard to consumer in the Kinesis Data Streams Standard (Classic) Fan-out consumer pattern?

Answer 15

Consumers poll data from Kinesis using GetRecords API call

Question 16

How is data transferred from shard to consumer in the Kinesis Data Streams Enhanced Fan-out consumer pattern?

Answer 16

Consumers use SubscribeToShard API and Kinesis pushes data to consumers over HHTP/2

Question 17

When would you prefer the Enhanced Fan-Out Consumer pattern over the Standard Fan-Out Consumer pattern for Kinesis Data Streams?

Answer 17

Enhanced is better for
- Lots of consuming applications for the same shard
- Lower Latency (70ms vs. 200ms for standard)

Question 18

What is the default limit for the number of consumer applications for Kinesis Data Streams Enhanced Fan-Out Consumer pattern?

Answer 18

5 consumers per stream BUT you can raise this with an AWS support ticket.

Question 19

Does the Standard Fan-Out pattern for Kinesis Data Streams support Lambda consumers?

Answer 19

Yes

Question 20

Does the Enhanced Fan-Out pattern for Kinesis Data Streams support Lambda consumers?

Answer 20

Yes

Question 21

Does the Enhanced Fan-Out pattern for Kinesis Data Streams support batch reads for Lambda consumers?

Answer 21

Yes

Question 22

What does KCL stand for?

Answer 22

Kinesis Client Library

Question 23

You are using KCL to read from a Kinesis Data Stream with 4 shards into DynamoDB. What is the maximum number of KCL instances you can use?

Answer 23

4 (same as the number of shards)

Question 24

In the context of Kinesis Data Streams, what is shard splitting?

Answer 24

A method to increase KDS capacity (and cost) by splitting traffic to a shard into 2 new shards.

Question 25

What happens to data in the old shard once a shard is split in Kinesis Data Streams?

Answer 25

consumers can still read from the shard, the shard will not be deleted until all the data has expired.

Question 26

What is the maximum number of new shards you can create from a single shard in one shard-splitting operation?

Answer 26

2

Question 27

Suppose you have an application in Kinesis Data Streams that is getting less traffic than expected and you want to decrease capacity and cost. How might you do this?

Answer 27

Use shard merging to scale down KDS, merging two shards with low traffic

Question 28

What are the allowed AWS Destinations for Kinesis Data Firehose?

Answer 28

• S3
• Redshift (by copying through S3)
• Amazon OpenSearch

Question 29

What is the maximum size of the data blob of a single record in Kinesis Data Streams?

Answer 29

1 MB

Question 30

What is the pricing model for Kinesis Data Firehose?

Answer 30

Pay for data going through Firehose

Question 31

Suppose you are handling a lot of data and need to write it to Splunk in near real-time. This data is sensitive so you would also like to have a backup of all data in an S3 bucket. What AWS service would you use for this?

Answer 31

Kinesis Data Firehose

Question 32

Which Kinesis Service(s) allow(s) you to write your own custom producer and consumer code?

Answer 32

Kinesis Data Streams

(Firehose is Fully managed)

Question 33

Which Kinesis Service(s) allow(s) replay capability?

Answer 33

Kinesis Data Streams

(Firehose has no data storage and thus no replay)

Question 34

Which Kinesis Service(s) allow(s) for automatic scaling?

Answer 34

Kinesis Data Firehose

(Streams has to use manual splitting/merging)

Question 35

Which Kinesis Service(s) process their data in real time?

Answer 35

Kinesis Data Streams

(Firehose is near real-time)

Question 36

In Kinesis Data Streams, is data from the same shard guaranteed to all have the same partition key?

Answer 36

No

partition key is hashed to a shard, so you can have the multiple partition keys hash to the same shard

Question 37

Suppose you have data from multiple sources in a producer/consumer model. Data from each source must be processed in order. There is a lot of data so you want to maximize throughput. What is the best service to use?

Answer 37

Kinesis Data Streams

(SQS FIFO with groupIDs will work but may not have optimum performance)

Question 38

Suppose you have data from multiple sources in a producer/consumer model. Data from each source must be processed in order. You want to have as many consumers as possible to read the data so you can get maximum parallelism, and you want to keep costs low. What is the best service to use?

Answer 38

SQS FIFO

(Kinesis Data Streams will work but may be more expensive, and you are limited to the number of consumers <= number of shards)