SQS, SNS, Kinesis, Active MQ Flashcards
What is SQS?
A service that connects 2 or more applications using queues between them.
What is a SQS Queue?
An SQS queue is what receives messages from “Producers” (apps), and sends them to “Consumers” (app that receives the messages).
Producers are apps that send messages to other apps through SQS. These other apps receive the message and process the order it gives them.
What is SQS useful for?
Use cases?
It’s a de-coupler of apps. Meaning you can separate an app in mutliple apps, and have each of those apps perform a function independently, but still be communicated between each other with SQS.
SQS is great for avoiding overloading an app with requests, since these requests go to the queue, and not directly to an app.
Imagine a website that has to upload content to S3. Instead of overloading the website and making it be slow because it has to upload the content to S3, you make the website send the message to upload content to an SQS queue, and another server will receive that message and do the job for you, making the website faster.
Other example if when an app needs to write to a database, if the database is overloaded with requests, it can lose data. But if you use SQS as a buffer between the app and the database, you fix this issue because SQS doesnt lose data. it scales infinitely.
Whats the max size of an SQS message?
256KB
How do producer apps send messages to SQS?
Through the SDK
What is the message retention time for SQS?
up to 14 days. Default is 4 days.
What are some examples of SQS messages?
Order id
Customer id
Address
etc.
How does the consumer interact with SQS?
The consumer polls the SQS for messages.
The SQS Queue responds with up to 10 messages.
Then the consumer has the responsibility to process these messages. And then ask the SQS queue to delete them.
What is the SQS throughput limit?
SQS throughput is unlimited. The limit is how fast the consumer can poll the messages from the sqs queue.
How can you scale consumer throughput in sqs?
You can scale consumer throughput horizontally by adding more servers to poll for messages from the sqs queue in parallel.
You can also scale vertically by adding more computing to your consumer server.
How do parallel consumers work with SQS?
You can set up multiple instances, servers, or lambda functions to poll the SQS queue. Each poll from each consumer should be followed by a delete command to the queue. This way each consumer will poll a different list of messages from the queue.
How can SQS work with Auto Scaling Group?
You can create EC2 instances in an ASG to act as consumer for an SQS queue.
Now, the metric that we will use to scale up or down the ASG will be a “Queue Length” CloudWatch Metric.
This way we will increase or decrease the number of consumers automatically depending of how many messages are in queue.
What is SQS security?
HTTPS encryption for in flight and SSE-SQS or SSE-KMS encryption for at rest.
IAM policies for access control to SQS API.
SQS access policies (Similar to S3 bucket policies)
How can you send and receive messages in sqs?
You can send messages and see them manually, but you can set up apps to send, poll, and delete messages automatically with the AWS SDK.
What is visibility timeout in SQS?
When a message is polled it instantly becomes invisible to other poll requests for 30 seconds. This is the time the app has to delete it before it becomes polled by other consumers too.
In this way we avoid messages being processed twice or more.
In case your app needs more time to process messages, you can change the visibility timeout. You shouldn’t make too long, for example hours, because if the app that polled a message crashed, you would have to wait hours before being able to poll it again. You should set a reasonable visibility timeout for your apps.
What is long polling?
In SQS, consumers can do a long poll, meaning if when the poll begins there are no messages, the longer poll waits for messages to arrive, and when messages arrive within the poll time, they are sent to the consumer.
This decreases the api calls made to sqs while increasing efficiency and latency of our app.
What is SQS FIFO queue?
It’s first in first out method of queueing messages.
Messages are only sent once thanks to the capability of FIFO of removing duplicate messages.
Messages are processed only once by the consumers
Messages are processed in order by the consumers
It has a limited throughput to guarantee the ordering
What is SNS?
Simple Notification Service.
It receives a message from a “Publisher” (The receiver of this message in SNS is an SNS Topic) and sends it to multiple “Subscribers”.
What could be an SNS subscriber?
A lambda function, an SQS queue, https endpoint apps, emails, sms and mobile notifications, kinesis data firehose.
What is SNS good for?
When you have a service app that needs to send a message to multiple destinations, instead of doing it multiple times to all those destinations, it does it only once to the SNS Topic, which will have all the destinations as subscribers, and the SNS will send the message to them. Thus reducing load from the service app.
What could be an SNS publisher?
CloudWatch alarms, ASG notifications, AWS Budgets, lambda, s4 bucket events, dynamodb, cloudformation, etc. (I dont need to remember them).
How do you publish messages to SNS?
Similar to SQS, you use the SDK with your app to publish to the SNS.
How is SNS security?
Same as SQS, HTTPS encryption for in flight and SSE-SQS or SSE-KMS encryption for at rest.
IAM policies for access control to SNS API.
SNS access policies (Similar to S3 bucket policies)
How is SNS needed for S3 Notifications?
You can only have one S3 event rule for an s3 prefix. You need to use fan out with SNS if you want to send this event to multiple SQS queues or other services like lambda.
How could SNS integrate with SQS?
When you need to send messages to multiple SQS queues, you do it with SNS. (Its called Fan-Out)
Set up an SNS topic with the SQS queues as subscribers. And you send the message only once to SNS.
If you were to not use SNS, you would need to send the message multiple times to each SQS queue, and you could experience app failures that could make you send a specific message to only one of the 2 SQS queues. But by using SNS, you send each message only once, making it impossible to send some messages to only one of 2 SQS queues.
This model with SNS + SQS is fully decoupled, and there is no data loss.
How can you integrate SNS with SQS FIFO?
SNS has FIFO capability, which if enabled lets you use SQS FIFO queues as subscribers AND standard queues too. The idea is first in first out, same as SQS FIFO.
FIFO features: Ordering, deduplication,
