SQS

Question 1

Give an overview of Amazon SQS

Answer 1

• Serverless, managed queue, integrated with IAM
• Can handle extreme scale, no provisioning required
• Used to decouple services
• Message size of max 256 KB (use a pointer to S3 for large messages)
• Can be read from EC2 (optional ASG), Lambda
• SQS could be used as a write buffer for DynamoDB (meaning that you write messages to SQS and use another application to write to DynamoDB, this saves you from putting too much load on DynamoDB
  SQS FIFO:
• receive messages in order they were sent
• 300 messages/s without batching, 3000 /s with batching

Question 2

What is Amazon SQS Dead Letter Queue (DLQ)

Answer 2

If a consumer fails to process a message within the
Visibility Timeout…the message goes back to the queue!. We can set a threshold of how many times a message can go back to the queue. After the MaximumReceives threshold is exceeded, the message goes into a Dead Letter Queue (DLQ)

• DLQ of a FIFO queue must also be a FIFO queue
• DLQ of a Standard queue must also be a Standard
  queue

Make sure to process the messages in the DLQ before
they expire:
* Good to set a retention of 14 days in the DLQ

Question 3

What is Amazon MQ and why would we use it

Answer 3

Because SQS and SNS are cloud specific, we may want to continue using what we were using before we migrated to the cloud. Amazon MQ is a managed message broker service for RabbitMQ and ActiveMQ.

• Amazon MQ doesn’t “scale” as much as SQS / SNS
• Amazon MQ runs on servers, can run in Multi-AZ with failover
• Amazon MQ has both queue feature (~SQS) and topic features (~SNS)

Question 4

What is Amazon SNS

Answer 4

• The “event producer” only sends message to one SNS topic
• As many “event receivers” (subscriptions) as we want to listen to the SNS topic notifications
• Each subscriber to the topic will get all the messages (note: new feature to filter messages)
• Up to 12,500,000 subscriptions per topic
• 100,000 topics limit

Question 5

What are possible subscribers to SNS

Answer 5

SQS
Lambda
Kinesis Data Firehouse
HTTP Endpoints
SMS & Mobile Notifications
Emails

Question 6

How to use SNS + SQS for fan out

Answer 6

Have an SNS topic that a lot of SQS Queues subscribe to. Ex: have your S3 event pushed to the SNS topic, which will have many applications subscribed to it

• Push once in SNS, receive in all SQS queues that are subscribers
• Fully decoupled, no data loss
• SQS allows for: data persistence, delayed processing and retries of work
• Ability to add more SQS subscribers over time
• Make sure your SQS queue access policy allows for SNS to write
• Cross-Region Delivery: works with SQS Queues in other regions

Note: You can include a filter policy on SNS so that the some queues won’t receive certain messages, or they will only receive certain messages

Question 7

SNS messages have a delivery policy explain that

Answer 7

Different AWS endpoints receiving SNS messages have delivery policies on what to do when there is an error. However, only HTTP/S endpoints support custom policies

• After exhausting the delivery policy, messages that haven’t been delivered are discarded unless you set a DLQ (Dead Letter Queue)
• DLQ are Amazon SQS queues or Amazon SQS FIFO queues (for SNS FIFO)
• Dead Letter Queues are attached to a subscription (rather than a topic)