SQS, SNS and Kinesis Flashcards
Why would you use a queue for messages?
The messages are asynchronous / event based, and so we need a queue to send them between applications.
Give a high level description of the model of SQS, SNS and Kinesis.
SQS - queue model
SNS - pub/sub model
Kinesis - real-time streaming model
What are the generic names for a objects that send messages into a queue, and objects that poll the queue for the message?
Producer and Consumer
Why is SQS useful for application decoupling?
- Unlimited number of messages in queue, unlimited throughput
- Low latency (<10 ms on publish and receive)
What are some caveats of SQS?
- Default retention of messages is 4 days, max is 14
- Limitation of 256kb per message
- Can have duplicate messages (‘at least once delivery, occasionally’) so need idempotency
- Can have out of order messages (‘best effort ordering’)
What is the API name for producing messages to the queue?
SendMessage API
How long does a message last in the queue (and which API can remove them)?
A message is persisted until a consumer deletes it using the DeleteMessage API, but has a default retention period of 4 days and a max of 14
How many messages are returned on a single Poll?
Up to 10 per Poll
How can you improve throughput of message processing?
Scale consumers horizontally using an ASG on the CloudWatch Queue Length metric (ApproximateNumberOfMessages) - set up an alarm that scales the group when it is hit.
How does SQS allow us to optimize our application in terms of EC2 instance type?
SQS allows us to decouple the front and back end instance types, and so we can configure the instance types separately to be optimal for the type of workload (e.g., if the backend application is video processing, we can use an instance type with a GPU).
What encryption is available for SQS?
- In-flight encryption using HTTPS API
- At-rest encryption using KMS keys
- Client-side encryption if the client wants to perform en/decryption itself
How can you allow cross-account access or other services access to SQS?
SQS Access Policies (similar to S3 bucket policies)
What is required in an SQS Access Policy to allow an EC2 instances (for example) to poll for messages?
The Principal field of the policy is set to “AWS”: [“{account_id}”], where account_id is the ID of the instance.
What is required in an SQS access policy to allow and S3 bucket to publish S3 event notifications to an SQS queue?
The Condition should include:
“ArnLike”:{“aws:SourceArn”: “{bucket_arn}”},
“StringEquals”:{“aws:SourceAccount”: “{bucket_owner_account_id}”}
What is the Message Visibility Timeout?
The period of time after a message has been polled by one consumer that the message remains invisible to other consumers (default 30s). After this timeout, the message becomes visible again, unless the message has been deleted during processing.
What should be done if a Consumer needs more than the allotted time to process a message before it is made visible again? What considerations should be made for this?
Call the ChangeMessageVisibility API to increase the invisibility time.
- If the timeout window is too high and a Consumer crashes during processing, re-processing will take time.
- If the window is too low, we may get duplicate processing.
How can you determine when a message should enter the Dead Letter Queue?
Set a MaximumReceives threshold to determine how many times a message can be reinserted into the queue after processing.
How must an SQS Queue relate to a DLQ?
DLQ of a FIFO queue must also be FIFO; DLQ of a Standard queue must also be a Standard queue
Why should you set a long retention time in the DLQ?
Make sure the messages are processed before they expire.
How do you re-process messages in the DLQ if you have fixed the original processing issue?
Use Redrive to Source to push the message from the DLQ to the SQS queue
How do you prevent consumers from seeing a published message in a queue straight away?
Use a ‘Delay’ Queue - set the delay to a given period of time (default is 0 seconds)
- Can override the default on send using the DelaySeconds param
What is Long Polling?
When a consumer requests messages from a queue, it can ‘wait’ for messages to arrive if there are none in the queue (from 1 to 20 seconds).
Why is Long Polling preferable to Short Polling?
- Decreases the number of API calls made to SQS while increasing the efficiency and latency of your application
How to you enable Long Polling?
Either enable at the queue level or at the API level using ReceiveMessageWaitTimeSeconds
What is the max. standard message size, and how can you send larger than this? What is a use case of this?
Use the SQS Extended Client
- If a Producer wants to send large (say, 1GB) messages, it will send the large message to and S3 bucket, and a small metadata message to the SQS queue containing a pointer to the bucket
- The consumer will poll for the small metadata message, and retrieve the large message from S3
- Can be used for video processing: the video is uploaded to S3 and a message is sent to the SQS queue
List the key API calls for SQS
- CreateQueue (attribute: MessageRetentionPeriod), DeleteQueue (along with all message in the queue)
- PurgeQueue: delete all messages in the queue
- SendMessage (attribute: DelaySeconds), ReceiveMessage, DeleteMessage
- MaxNumberOfMessages: default 1, max 10 (for ReceiveMessage API)
- ReceiveMessageWaitTimeSeconds: Long Polling
- ChangeMessageVisibility: change the message timeout
What batch API calls can you do for SQS?
SendMessage, DeleteMessage, ChangeMessageVisibility
What are the advantages of a FIFO queue?
- Exactly-once send capability (by removing duplicates)
- Messages are processed in order by the consumer
What are the disadvantages of a FIFO queue?
Limited throughput: 300 msg/s without batching, 3000 with
What is the naming constraint on FIFO queues?
The name must end in ‘.fifo’
What is meant by the interval for de-duplication?
The time period after a message is received in which duplicate messages will be removed - 5 minutes
