SQS, SNS, Kinesis, Active MQ Flashcards
What is SQS?
A service that connects 2 or more applications using queues between them.
What is a SQS Queue?
An SQS queue is what receives messages from “Producers” (apps), and sends them to “Consumers” (app that receives the messages).
Producers are apps that send messages to other apps through SQS. These other apps receive the message and process the order it gives them.
What is SQS useful for?
Use cases?
It’s a de-coupler of apps. Meaning you can separate an app in mutliple apps, and have each of those apps perform a function independently, but still be communicated between each other with SQS.
SQS is great for avoiding overloading an app with requests, since these requests go to the queue, and not directly to an app.
Imagine a website that has to upload content to S3. Instead of overloading the website and making it be slow because it has to upload the content to S3, you make the website send the message to upload content to an SQS queue, and another server will receive that message and do the job for you, making the website faster.
Other example if when an app needs to write to a database, if the database is overloaded with requests, it can lose data. But if you use SQS as a buffer between the app and the database, you fix this issue because SQS doesnt lose data. it scales infinitely.
Whats the max size of an SQS message?
256KB
How do producer apps send messages to SQS?
Through the SDK
What is the message retention time for SQS?
up to 14 days. Default is 4 days.
What are some examples of SQS messages?
Order id
Customer id
Address
etc.
How does the consumer interact with SQS?
The consumer polls the SQS for messages.
The SQS Queue responds with up to 10 messages.
Then the consumer has the responsibility to process these messages. And then ask the SQS queue to delete them.
What is the SQS throughput limit?
SQS throughput is unlimited. The limit is how fast the consumer can poll the messages from the sqs queue.
How can you scale consumer throughput in sqs?
You can scale consumer throughput horizontally by adding more servers to poll for messages from the sqs queue in parallel.
You can also scale vertically by adding more computing to your consumer server.
How do parallel consumers work with SQS?
You can set up multiple instances, servers, or lambda functions to poll the SQS queue. Each poll from each consumer should be followed by a delete command to the queue. This way each consumer will poll a different list of messages from the queue.
How can SQS work with Auto Scaling Group?
You can create EC2 instances in an ASG to act as consumer for an SQS queue.
Now, the metric that we will use to scale up or down the ASG will be a “Queue Length” CloudWatch Metric.
This way we will increase or decrease the number of consumers automatically depending of how many messages are in queue.
What is SQS security?
HTTPS encryption for in flight and SSE-SQS or SSE-KMS encryption for at rest.
IAM policies for access control to SQS API.
SQS access policies (Similar to S3 bucket policies)
How can you send and receive messages in sqs?
You can send messages and see them manually, but you can set up apps to send, poll, and delete messages automatically with the AWS SDK.
What is visibility timeout in SQS?
When a message is polled it instantly becomes invisible to other poll requests for 30 seconds. This is the time the app has to delete it before it becomes polled by other consumers too.
In this way we avoid messages being processed twice or more.
In case your app needs more time to process messages, you can change the visibility timeout. You shouldn’t make too long, for example hours, because if the app that polled a message crashed, you would have to wait hours before being able to poll it again. You should set a reasonable visibility timeout for your apps.
What is long polling?
In SQS, consumers can do a long poll, meaning if when the poll begins there are no messages, the longer poll waits for messages to arrive, and when messages arrive within the poll time, they are sent to the consumer.
This decreases the api calls made to sqs while increasing efficiency and latency of our app.
What is SQS FIFO queue?
It’s first in first out method of queueing messages.
Messages are only sent once thanks to the capability of FIFO of removing duplicate messages.
Messages are processed only once by the consumers
Messages are processed in order by the consumers
It has a limited throughput to guarantee the ordering
What is SNS?
Simple Notification Service.
It receives a message from a “Publisher” (The receiver of this message in SNS is an SNS Topic) and sends it to multiple “Subscribers”.
What could be an SNS subscriber?
A lambda function, an SQS queue, https endpoint apps, emails, sms and mobile notifications, kinesis data firehose.
What is SNS good for?
When you have a service app that needs to send a message to multiple destinations, instead of doing it multiple times to all those destinations, it does it only once to the SNS Topic, which will have all the destinations as subscribers, and the SNS will send the message to them. Thus reducing load from the service app.
What could be an SNS publisher?
CloudWatch alarms, ASG notifications, AWS Budgets, lambda, s4 bucket events, dynamodb, cloudformation, etc. (I dont need to remember them).
How do you publish messages to SNS?
Similar to SQS, you use the SDK with your app to publish to the SNS.
How is SNS security?
Same as SQS, HTTPS encryption for in flight and SSE-SQS or SSE-KMS encryption for at rest.
IAM policies for access control to SNS API.
SNS access policies (Similar to S3 bucket policies)
How is SNS needed for S3 Notifications?
You can only have one S3 event rule for an s3 prefix. You need to use fan out with SNS if you want to send this event to multiple SQS queues or other services like lambda.
How could SNS integrate with SQS?
When you need to send messages to multiple SQS queues, you do it with SNS. (Its called Fan-Out)
Set up an SNS topic with the SQS queues as subscribers. And you send the message only once to SNS.
If you were to not use SNS, you would need to send the message multiple times to each SQS queue, and you could experience app failures that could make you send a specific message to only one of the 2 SQS queues. But by using SNS, you send each message only once, making it impossible to send some messages to only one of 2 SQS queues.
This model with SNS + SQS is fully decoupled, and there is no data loss.
How can you integrate SNS with SQS FIFO?
SNS has FIFO capability, which if enabled lets you use SQS FIFO queues as subscribers AND standard queues too. The idea is first in first out, same as SQS FIFO.
FIFO features: Ordering, deduplication,
How do you send messages with Fanout + ordering + deduplication features?
With SNS FIFO + SQS FIFO.
What is SNS Message filtering?
It’s a JSON policy that filters messages sent to an SNS topics’ subscribers.
For example you can apply a filter so that only one subscriber of the SNS topic receives certain messages, for example a message for an order that has State = Placed. So all orders that have the state “Placed” will go only to that specific subscriber (SQS Queue for example).
In the same way, in this policy, only messages that have the “Placed” state will go to this SQS subscriber.
In the same way, we can create another Filter Policy that sends messages with the “Cancelled” state to another subscriber. Etc.
What could be a subscriber in an SNS FIFO configuration?
Only SQS.
What is kinesis for?
Makes it easy to collect, process and analyze data in real time.
Data such as app logs, metrics, website clickstreams
What are the different kinesis services?
Kinesis Data Streams
Kinesis Data Firehose
Kinesis Data Analytics
Kinesis Video Streams
What does Kinesis Data Streams do?
Capture and process big data streams
What does Kinesis Data Firehose do?
Load data streams into AWS Datastores
What does Kinesis Data Analytics do?
Analyze data streams with SQL or Apache Flink
What are Kinesis Data Streams Shards?
A Kinesis Data Stream is made of multiple shards. From 1 to n. These shards can be provisioned or on demand. With Shards you define the capacity of your stream, and also the transfer speed.
The transfer rate in which producers can send data to kinesis is 1MB/sec or 1000msg/sec PER SHARD. So scaling shards also scales transfer speeds.
What are kinesis data streams producers?
Producers could be EC2 applications, Desktop or Mobile Clients, or a server with the kinesis agent installed.
The producers send the data to the shards. Producers use the SDK or the KPL to send their data to kinesis.
Data is sent to kinesis in “Records”. Records are made of a partition key and the data blob which is up to 1MB.
What is the partition key in kinesis?
The partition key is the part of the producer-sent record that defines in which shard the record is placed.
You set what is the partition key for all the data uploaded to kinesis.
What are kinesis data streams consumers?
The consumers consume the data stored in kinesis data streams.
The data consumed is in records, just as with producers, with a partition key and a data blob. But also a sequence number, that indicates in which part of the shard the record was placed.
What is the sequence number in kinesis?
In kinesis data streams, only when consumed, a record has a sequence number that indicates in which part of the shard the record was placed.
How long is data stored in kinesis data streams?
Between 1 to 365 days, depending on configuration of the stream.
You can reprocess data within that window.
What are the properties of all data stored in kinesis?
Data in immutable. Can’t be deleted within the retention period.
Data that shares the same partition goes to the same shard.
What are the 2 data capacity modes for kinesis data streams?
Provisioned mode: Choose number of shards provisioned in a stream. Then you can scale manually.
On Demand mode: You dont manage capacity. It scales automatically based on observed throughput peak during last 30 days.
What do you pay for in kinesis data streams?
In provisioned mode you pay per shard per hour.
In on demand mode you pay per stream per hour and data in and out per GB.
What is the code you type in aws cli to send a record into a kinesis stream?
Producer:
aws kinesis put-record –stream-name test –partition-key user1 –data “user signup” –cli-binary-format raw-in-base64-out
How can kinesis data streams and data firehose work together?
You can use kinesis data stream as a source or producer for firehose, and writes that data in batches to destinations like s3, redshift.
What is the code you type in aws cli to describe a kinesis stream, and consume data?
With describe stream api call you get all the info of the stream:
aws kinesis describe-stream –stream-name test
Consumer:
With get shard iterator you get the code that indicates where you will start reading the shard data records sequentially. “Trim_horizon” tells it to give you the iterator that reads from the beginning.
aws kinesis get-shard-iterator –stream-name test –shard-id shardId-000000000000 –shard-iterator-type TRIM_HORIZON
With this and the iterator from the previous command, you CONSUME the record list with all the data of each record in base64 coding:
aws kinesis get-records –shard-iterator <>
At the end of the consume results you also get the “next shard iterator”, which you will need to use to consume the next records without repeating all which you just consumed.
How does kinesis firehose work?
It takes data from producers similarly to data stream, and writes in batches to destinations like s3.
What is the main difference in how kinesis data streams and firehose write data?
In data streams you have to code how to consume the data.
In firehose you don’t have to code because it’s aws managed, you just choose the destination for the data to be stored in.
What are all the possible destinations for aws kinesis data firehose?
AWS managed: S3, Redshift (through s3 first), opensearch.
3rd party partner destinations.
Or a custom HTTP endpoint.
What is redshift?
A warehouse database
How is lambda involved with firehose?
Firehose lets you transform data optionally with lambda function.
What are backup options for firehose?
You can send all data as a second destination to s3 bucket, or only send all failed data to s3 bucket.
What do you pay for in aws kinesis data firehose?
Only for data going through firehose.
Which kinesis service is real time?
Data streams.
Firehose is near real time.
Which kinesis service is near realtime?
Firehose
Which kinesis service doesn’t store data streamed?
Firehose
What service do the terms ingest, transform and load refer to?
Kinesis Data Firehose
What service do the terms producer, record and consumer refer to?
Kinesis Data Streams
What service could the terms producer and consumer refer to?
SQS, Kinesis Data Streams.
What service do the terms publisher and subscriber refer to?
SNS
What is kinesis firehose buffer?
You can buffer data to not be sent until it reaches a certain size in MiB.
Higher buffer size makes data transfer cheaper.
You also set a buffer interval after which send the data even if you havent reached the buffer size.
What do you need to have more consumers when using SQS FIFO?
You need group ids to have more than 1 consumer when using sqs fifo. For each group id you can have the same amount of consumers.
What is enhanced fan out?
A kinesis data streams mode that gives you more throughput. 2MB per shard per consumer. (Standard is 2MB per shard in total).
What is the difference in ordering between kinesis and SQS?
In kinesis you get ordering at the shard level.
In SQS you get full ordering if you use FIFO. None if you use standard.
What is Amazon MQ
Similar to SQS and SNS, but compatible with traditional on premises protocols.
Aws MQ is a managed service for rabbitMQ and ActiveMQ
How does MQ work?
You configure an MQ broker in 2 AZs within a region in an active-standby setup.
You mount both MQ brokers to an EFS share.
