Kinesis

Question 1

Kinesis

Answer 1

a managed alternative to Apache Kafka

a big data streaming tool which allows you to collect application logs, metrics, IoT, clickstream, basically anything that is real-time big data.

overall is you know associated with big data real time. (exam)

Question 2

compatible with

Answer 2

many streaming processing frameworks so you may have heard of Apache Spark, Apache NiFi, etc.

Basically these are frameworks allowing you to perform computations in real time on data that arrives through a stream.

Question 3

data replication

Answer 3

the data is automatically replicated to 3 Availability Zones.

Question 4

three sub-Kinesis products

Answer 4

1. Kinesis Stream: people also just call Kinesis which is how to ingest streams at scale with a low latency. (exam)
2. Kinesis Analytics: to perform real-time analytics on streams using SQL, perform filters, computation, aggregations in real-time
3. Firehose: load your stream into other parts of AWS such as S3, Redshift, ElasticSearch and so on.

Question 5

Now how do we get the streams?

Answer 5

our clickstreams, IoT devices and metrics and logs will be producing data directly into our Kinesis streams.

Question 6

Kinesis Analytics

Answer 6

Once we have the data into Kinesis streams, Kinesis wants you to process the data and maybe perform computations, metrics, monitoring, alerting, whatever you want and for this you will need to perform some computation in real-time.

Question 7

Kinesis Firehose

Answer 7

once these computations are done, it’s good to have them stored somewhere into you know, S3, database, Redshift, et cetera.
For example to put it in S3 or in Redshift.

Question 8

Streams are divided into

Answer 8

ordered Shards or Partitions.

Question 9

Shard

Answer 9

think of it as one little queue.

we have our producers and they are going to produce
to a Kinesis stream maybe this one has three shards.
And so the data is going to go into either shard and the consumers will be consuming from either shard as well.

Question 10

if we wanna scale up our stream

Answer 10

we just add shards

if wanted a higher throughput we would increase the number of shards.

Question 11

in this shard the data is not there forever

Answer 11

By default it’s here for one day. We can set it up so each shard can keep your data up to 7 days.

Question 12

why would you have such short data retention?

Answer 12

because Kinesis is just a massive highway, it’s a massive pipe. And so you want to process your data do something and put it somewhere else as soon as possible.

Question 13

difference with SQS

Answer 13

Kinesis is also awesome because it allows you to reprocess and replay data. SQS once the data was consumed it was gone.

with Kinesis the data is still there. And it will expire after some time.

Question 14

multiple consumers

Answer 14

You’re also able to have multiple applications consume

the same stream so sort of like an SNS of a mindset

We need to just have one stream with a stream of data
and we can have many applications, many consumers consume the same stream.

Question 15

Kinesis is not a database

Answer 15

Once the data is inserted into Kinesis you cannot delete it.

It’s called immutability. So you add data it’s called a log, you add data over time and then you process it using consumers.

The data will stay in Kinesis for one to seven days and then you do something with it.

Question 16

shards size

Answer 16

streams are made of many shards.

But a shard represents one megabyte per second or 1000 messages per second on the right side. So the producer can write up to 1000 messages per second
or one megabyte per second.

On the read side you have two megabyte per second
throughput per shard

Question 17

you’re going to pay for

Answer 17

how much shards you provisioned. And you can have as many shards as you want but if you over provision your shards and you’re not using them up to their full capacity you’re going to overpay. Similarly if you have more throughput than your shards then you’re going to have throughput issues.

Question 18

batch

Answer 18

You have ability to batch the messages and the calls.

And this allows you to efficiently push messages into Kinesis.

to reduce costs and increase throughput.

Question 19

the records will be ordered

Answer 19

per shard

Question 20

resharding

Answer 20

adding a shard

Question 21

merging

Answer 21

deleting a shard

Question 22

producers sending data

Answer 22

you need to send data in a partition key. So your data is the gray box and the message key is the orange box
and the message key is whatever you want as a string.

And this key will get hashed to determine the shard ID.
So the key is basically a way for you to root the data
to a specific shard.

the same key always goes to the same partition

if you want to get all your data in order for a same key then you would just provide that key to every data point and they will be in order for you.

when your data is produced, now the messages know where to go which shard because of the message key.

Question 23

sequence number

Answer 23

the messages when they’re sent to a shard, they get a sequence number and that sequence number is always increasing

Question 24

if you need to choose a partition key

Answer 24

you need to choose one that is going to be
highly distributed (exam)
that prevents the hot partition.

if your key wasn’t distributed, then all your data will go through the same shard and one shard will be overwhelmed.

Question 25

if we have an application and you have one million users

Answer 25

user ID is a great key right because we have one million users and so realistically all the users will do actions in different times but we’re gonna get ordering for that user ID which is our message key and so user ID is a good one.

Very distributed, very active and useful from a business perspective.

But if you have country ID as a field and it turns out that 90% of your users are in one country say
The United States then it’s not good because
all your country ID will go to one shard.

Question 26

ProvisionedThroughputExceeded

Answer 26

And if you get an exception called ProvisionedThroughputExceeded

that’s when you go over the limits.
when we send more data than what was provisioned.
we exceed the number of megabytes per second
or transactions per second.

Question 27

to produce messages you use

Answer 27

CLI, but you can use the SDK or producer libraries

from various frameworks.

Question 28

ProvisionedThroughputExceeded solutions

Answer 28

And for this you can just

1. use retries and exponential backoff.
2. to increase the number of shards
3. to ensure that the partition key is a good one.

Question 29

you can use a normal consumer using

Answer 29

the CLI, the SDK, or the Kinesis Client Library (available in Java, Node, Python, Ruby or .Net.)

Question 30

Kinesis Client Library

Answer 30

it uses also DynamoDB to checkpoint the offsets
and to track other workers and share the work amongst shards.

We’ll have a DynamoDB table and the Kinesis app
that uses KCL the client library. We’ll checkpoint the progress through Amazon DynamoDB and then they will synchronize their work between them to consume messages from different shards.

Question 31

Kinesis Security

Answer 31

1. we can control access and authorization to Kinesis
   using IAM policies.
2. encryption in flight using HTTPS endpoints.
3. encryption at rest using KMS.
4. There is a possibility to also encrypt and decrypt
   the data client side but it’s much harder to implement.
   You need to write your own code.
5. you can also have VPC Endpoints available for Kinesis to access privately within a VPC.

Question 32

Kinesis data firehose

Answer 32

a fullly managed service.

There is no administration needed, it scales automatically,

fully serverless.

We’re not going to prevision anything in advance.

It’s going to be near real time. (Kinesis streams was real time) = 60 seconds latency minimum for non full batches.

Question 33

Kinesis data firehose used for

Answer 33

to load data into Redshift, Amazon S3, ElasticSearch and Splunk. (exam)

Question 34

Kinesis data firehose minimum data

Answer 34

we will write about 32 MB of data as a minimum, at a time to load into these stores.

Question 35

Kinesis data firehose supports

Answer 35

many formats, conversions, transformations, and compression.

handy with CSV, JSON

Question 36

Kinesis data firehose ypu pay

Answer 36

the amount of data going through Firehose.

you don’t pay for provisioning Firehose. But you do first data streams.

Question 37

Kinesis data firehose data sources

Answer 37

could be Kinesis Producer Library, a kineses agent,
a kinesis data stream

Both the agent and the kinesis data streams can send data directly into the data firehose.

can even be CloudWatch Logs or CloudWatch Events.
And then you can do some transformations using a Lambda function.

Question 38

what is the difference between kinesis data streams

and Firehose?

Answer 38

Streams =
1. when you write custom code. You need to write your own producer, your own consumer most of the time.

2. And it’s going to be real time. About 200 ms latency.
3. You must manage scaling yourself so you must do something called shard splitting or shard merging.
   And so that means that you have to do capacity planing over time.
4. you can store data and it is going to expire between one to seven days. So if you need a place to just store data for three days.

kinesis data streams is a great way of doing it.

5. Thanks for this you can do replay capability
6. it’s multi consumers.

Firehose

1. fully managed, you’re only provision capacity,
2. you send data to S3, Splunk, Redshift and ElasticSearch.
3. serverless so data transfromations with Lambda.
4. near real time
5. automated scaling.
6. there is no data storage. So you cannot replay from Firehose.

Question 39

Kinesis Analytics

Answer 39

can take data from kinesis data streams and kinesis data firehose and perform some queries.

the output of these queries can be analyzed by your analytics tools, rear outputs.

performs a real time analytics using SQL.

auto scaling,

managed

no servers to provision,

continuous it’s going to be real time

out of these queries you can create new streams so they can be consumed again by consumers or by kinesis data firehose

Question 40

Kinesis Analytics you only pay for

Answer 40

the actual consumption rate of kinesis data analytics.

Question 41

Data Ordering for Kinesis vs SQS FIFO

Answer 41

in Kinesis if we have 5 shards and 100 IDs, more or less 20 trucks will be assigned to each shard (partition ID) based on the hashed ID of the objects we are processing

And we can have therefore only 5 parallel consumers

In SQS FIFO there is only one queue. But we can create groups with IDs. So we can have 100 groups based on the IDs of the objects. So we will be able to have 100 parallel consumers.

Question 42

SQS vs SNS vs Kinesis —– SQS

Answer 42

1. the consumers pull data and the data is going to be deleted right after being consumed.
2. You can have as many consumers as we want
3. you don’t need to provision throughput, it will scale automatically for you.
4. There is no ordering guarantee unless you use the FIFO queue, but if you use the FIFO que then you get limited throughputs
5. there is an individual message delay capability so you can take a message and say be consumed in 15 minutes.

Question 43

SQS vs SNS vs Kinesis —– SNS

Answer 43

1. pub/sub so you push data to many subscribers.
2. you can have up to 10 million subscribers to one topic, up to 10,000 topics.
3. the data is not persisted, so that means that it’s lost if not delivered.
4. you don’t need to provision the throughput in advance
5. if you wanted to persist the data out, deliver it to many SQS ques
6. you can use a fan-out architecture to integrate it with SQS

Question 44

SQS vs SNS vs Kinesis —– Kinesis

Answer 44

1. a pull of data, so like SQS we pull data, SNS was pushing data.
2. We can have as many consumers as we want, but we can only have one consumer per shard.
3. the possibility to replay data is available; we could reprocess a whole day of data
4. meant for real-time big data, analytics and ETL, (exam) we wanna do real-time ingestion of data of IOT

any time you real-time big data

5. There is ordering but it’s at the shard level.
6. the data expires after X number of days
7. there is some data retention but it’s temporary.
8. you must provision your throughput in advance