Section 19: AWS Integration & Messaging: SQS, SNS & Kinesis: SQS

Question 1

Patterns: When we sart deploying multiple applications they will need to communicate with each other. Synchronous and Asynchonous/Event Based are two patterns of communication. What’s the difference?

Answer 1

1. Synchronous is described as application-to-application. Ex: you have an app for buying and an app for shipping and they are in direct contact with each other.
2. Asynchronous /Event based is described as application-to-queue-to-application. Ex: you have an app for buying, and a queue, and an app for shipping. When someone buys something, that item goes into a queue for shipping, and the shipping app picks the item up from the queue.

Question 2

If you have sudden spikes of traffic, might that be more like to become a problem for your synchronous or your asynchonous application architecture pattern?

Answer 2

Other things being equal, it’s more likely to be a problem for your synchronous application. Ex. if you need to encode 1000 videos where usually you only encode 10. This is why people are so into the idea of decoupling applications.

Question 3

what kind of model is it when you decouple an application using:
1. SQS?
2. SNS?
3. Kinesis?
4. can these services scale indepently from our applications

answer options: pub/sub, queue model, real-time streaming

Answer 3

1. queue model
2. pub/sub model
3. real time streaming model
4. yes

Question 4

1. What goes in the middle, SQS, SNS or Kinesis?
2. Is this a queue model, a pub sub model, or a real time streaming model? What going

Answer 4

1. SQS
2. queue model

Question 5

1. Is SQS a managed SQS service?
2. What is SQS used for, coupleing or decoupling applications?

Answer 5

1. yes.
2. Decouple

Question 6

What are the two types of SQS queue types? (just names, more details on both will be asked for soon)

Answer 6

1. Standard
2. FIFO

Question 7

Will messages in an SQS queue of type Standard occassionally be duplicated?

Answer 7

yes for queue type: Standard. Messages will always be sent at least once, but occassionally they’ll be sent multiple times.

Question 8

Will messages in an SQS queue of type FIFO occassionally be duplicated?

Answer 8

FIFO gaurantees “Exactly-Once Processing – A message is delivered once and remains available until a consumer processes and deletes it. Duplicates aren’t introduced into the queue.”.

Question 9

Will messages in an SQS queue of type Standard occassionally come out of order?

Answer 9

Yes. For Standard type SQS queue best effort ordering is used, which means messages will occassionally come out of order..

Question 10

Will messages in an SQS queue of type FIFO occassionally come out of order?

Answer 10

No. FIFO type SQS queues gaurantee first in, first out delivery. The order in which messages are sent and received is strictly preserved.

Question 11

What is the limit on the throughtput /number of messages in an SQS queue of type Standard per day?

Answer 11

For Standard type SQS queue: unlimited throughput, unlimited number of messages in the queue. exact quote is “Unlimited Throughput – Standard queues support a nearly unlimited number of API calls per second, per API action (SendMessage, ReceiveMessage, or DeleteMessage).”

Question 12

What is the limit on the throughtput /number of messages in an SQS queue of type FIFO per day?

Answer 12

For FIFO type SQS queue you can get what is sometimes described as high, and sometimes described as limited, throughput.
* You can get 300 messages per second when you don’t use batch apis (if you use SendMessage, DeleteMessage, ChangeMessageVisibility)
* If you use batch api’s (SendMessageBatch, DeleteMessageBatch, ChangeMessageVisibilityBatch) then those 300 messages would be able to send the data of 3000 messages.

Question 13

So I think both Standard and FIFO type SQS queues are good for decoupled application, but one seems to be best when order is doing steps in sequence is most important, and the other seems to be best when high throughput is most important. Which is most important when?

Answer 13

• Standard: use this one when throughput between applications is most important.
• Use FIFO when order of events is most important.

Question 14

Give a use case example of Standard type SQS queue. Can give up to three.

Answer 14

• Decouple live user requests from intensive background work: let users upload media while resizing or encoding it.
• Allocate tasks to multiple worker nodes: process a high number of credit card validation requests.
• Batch messages for future processing: schedule multiple entries to be added to a database.

Question 15

Give a use case example of FIFO type SQS queue. Can give up to three.

Answer 15

• Make sure that user-entered commands are run in the right order.
• Display the correct product price by sending price modifications in the right order.
• Prevent a student from enrolling in a course before registering for an account.

Question 16

SQS

2. What is the default retention of messages (in days) and the maxium retention of messages (in days)?
3. What is the latency on publish and receive?
4. what is the data size maximum per messages sent?

Answer 16

2. default is 4 days, maximum is 14 days
3. low latency (< 10 ms on publish and receive)
4. limitation of 256 kb per message sent

Question 17

SQS producing messages:

1. How are messages proced to SQS? (what is the name of the API?)
2. How long is a message persisted (how long does a message stay) in SQS?
3. What is an example of something you might use SQS for?

Answer 17

1. SendMessage API (for what it’s worth, the line reads “[Messages are] Produced to SQS using the SDK (SendMessage APi)”)”)
2. The message is persisted in SQS in until a consumer deletes it (though, again, it can stay, presumably awaiting pickup for a default of 4 days and up to 14 days)
3. you might use it to send an order to be processed, along with info like an order id, customer id, and any other relevant attributes.

Question 18

SQS Consuming Messages
1. what aws services do Consumers frequently use?
2. what do SQS consumers do, broadly?
3. how many messages can an SQS consumer receive at a time?

Answer 18

1. ec2 instances, servers, aws lambda
2. poll SQS for messages, process any messages, delete messages
3. 10 messages at a time

Question 19

1. What is an example of what an SQS consumer might do when in it’s processing stage (hint, what’s one place a message might need to go)?
2. What is the API SQS Consumers use to delete messages?

Answer 19

1. They might insert the message into an RDS database
2. DeleteMessage API

Question 20

Say you have multiple SQS EC2 Instance consumers.
1. Do such ec2 instance consumers receive and process messages sequentially, or parallely?
2. If you were experiencing throughput issues, would you scale such consumers vertically or horizontally to improve processing?
3. are messages gauranteed delivery to at least one consumer?
4. are messages gauranteed to be received in the order in which they were submitted?
5. who deletes messages, producers or consumers?

Answer 20

1. parallely
2. horizontally
3. yes
4. no, best effort will be made at ordering though
5. consumers

Question 21

Can you use SQS with an auto scaling group?

Answer 21

yes

Question 22

What are some key aspects of the design for using SQS with ec2 instance consumers and an auto scaling group for the ec2 instance consumers?

Answer 22

• your ec2 instances poll your SQS queue for messages.
• your ec2 instance has an associated auto scaling group.
• your auto scaling group uses a CloudWatch alarm to know when to scale.
• your cloudwatch alarm is based on a cloudwatch metric - queue length (ApproximateNumberOfMessages) in your SQS queue.

Question 23

What is an example of how SQS can be used to decouple a front end from a back end.

Answer 23

requests -> front end web app with an asg -> send messages using SendMessage API to an sqs queue -> back end app (which also has an asg) receives messages via ReceiveMessages API -> data gets inserted into something like an s3 bucket or whatever.

Question 24

SQS and Security: how is encryption handled:
1. in flight
2. at rest
3. client side?

Answer 24

1. in flight encryption using https api
2. at rest encryption using kms keys
3. client side encryption if the client wants to perform encryption / decryption itself

Question 25

sqs security: how is access controlled? (just name the two methods, i’ll ask for info on those methods soon)

Answer 25

1. access controls
2. sqs access policies

Question 26

sqs security: what are access controls?

Answer 26

1. iam policies to regulate access to the sqs api

Question 27

sqs security: what are sqs access policies?

Answer 27

they’re similar to s3 bucket policies. They’re useful for cross-account access to sqs queues, and for allowing other services (like SNS and S3) to write to an SQS queue.

Question 28

SQS queue access policy: how do you get cross account access so an ec2 instance can poll an SQS queue for messages? Give result in SQS queue access policy form. The SQS queue id is 444455556666. the ec2 instance account id is 111122223333 and it’s region is us-east-1.

Answer 28

Question 29

Can you give user access to specific dead-letter queue actions by adding permissions to your policy?

Answer 29

yes

Question 30

Write an SQS queue access policy to publish s3 event notifications to an SQS queue. Say the sqs queue account id is 444455556666, the sqs queue is in region us-east-1. The name of the s3 bucket is bucket1, and the account id of the person who created bucket1 is <bucket1_owner_account_id> but pretend it's just 12345667.</bucket1_owner_account_id>

Answer 30

Question 31

Regarding SQS message visibility timeout,
1. Do queue messages (items) become invisible immediately after they are picked up by a consumer?
2. For how long?
3. why?
4. what happens if the queue message is not done being processed by a consumer and the visibility timeout ends? (this also begs the question, what determines whether a consumer is done processing a queue message/item, so answer that too)

Answer 31

1 .yes
2. the default value is 30 seconds, though you can set it for anywhere from 0 seconds to 12 hours.
3. to prevent another consumer from picking up and processing the same queue message
4. if a queue message is not done being processed by a consumer and the visibility timeout ends, the queue item becomes visible to other consumers, so it can be picked up and processed by another consumer. The answer to the subquestion is that, as mentioned previously, consumers are supposed to delete a queue message/items once it’s finished doing whatever it does when it processes a message/item.

Question 32

Regarding SQS Message Visibility Timeout: is there something a consumer can do to extend the amount of time it has to process a request?

Answer 32

yes, a consumer could call the ChangeMessageVisibilty API.

note that if visibility is too high (hours) and a consumer crashes, reprocessing could take time. If visibility timeout is too low (seconds) you might get duplicates (assume he means duplicate instances of a request being processed.)

Question 33

Regarding SQS Dead Letter Queue: what’s it do? What do I suspect is the alternate term or field name for this?

Answer 33

It sets the threshold of how many times a message can go back into the queue if it keeps failing to get processed by an sqs consumer. Maximum Receives. Or maybe MaximumReceives. In the slides it’s the second, but in aws docs it’s the former. I kind of suspect it’s both, though just at different times.

Question 34

What happens after the MaximumReceives (or Maximum Receives) threshold is exceeded?

Answer 34

the message (the one that’s been failing to be processed) goes into a dead letter queue (DLQ). Something about this is useful for debugging?

Question 35

What are the two types of queues supported / used by SQS? (just names, we’ve covered details in other answers)

Answer 35

FIFO and Standard.

Question 36

1. Can the Dead Letter Queue of a FIFO queue be of queue type Standard or FIFO?
2. Can the Dead Letter Queue of a Standard queue be of queue type Standard or FIFO?

Answer 36

1. the DLQ of a Standard queue must be of queue type Standard.
2. the DLQ of a FIFO queue must be of queue type FIFO.

Question 37

T/F:
1. you should probably process the messages in the DLQ before they expire
2. it’s good to set a retention of 14 days in teh DLQ

Answer 37

1. True
2. True

Question 38

SQS Dead Letter Queue Redrive to Source - what on earth is this talking about/when would you use it?

Answer 38

Say your SQS queue had to move a bunch of messages into a DLQ because your processing code was bad. You can fix your code and then use some of the SQS API to get (redrive) your messages back into the main SQS queue. API actions are words like

StartMessageMoveTask, ListMessageMoveTasks, CancelMessageMoveTask.

Question 39

1. What is an SQS Delay Queue?
2. How long can a message be in a delay queue?
3. What is the default length of time for which a message is in the delay queue?
4. can you set a default length of delay queue time at the queue level (so the delay will apply to every message in the queue)? How?
5. can you override the default delay queue time when sending a message? How?

Answer 39

1. A queue used to delay a message so that consumers don’t use it immediately.
2. up to 15 minutes
3. 0 seconds.
4. yes. I beleive by using the DelaySeconds parameter.
5. Yes, for Standard type queues. No, for FIFO type queus. You override the default delay queue time for a message in a Standard type queue by using a message timer (i beleive, based on my interpretation of some aws docs.)

Question 40

Say you have a consumer doing too many calls into the SQS queue and it’s costing you money, CPU cycles and maybe increasing latency. What should you set up?

Answer 40

Long polling.

Question 41

What is long polling?

Answer 41

• it’s when a consumer sends out a request to the queue and then waits for a response until there’s a message in the queue or the wait time is up. This is done to reduce the number of API calls sent out by the consumer.
• we can think of it as “waiting”, but it’s actually this:

“With long polling, the ReceiveMessage request queries all of the servers for messages. Amazon SQS sends a response after it collects at least one available message, up to the maximum number of messages specified in the request. Amazon SQS sends an empty response only if the polling wait time expires.”

Question 42

T/F

Long polling decreases the number of API calls made to SQS while increasing the efficiency and decreasing the latency of your application.

Answer 42

True

Question 43

1. The wait time of a long poll can be … ?
2. What is the preferred length of time?

Answer 43

1. from 1 second to 20 seconds.
2. preferred length of time is 20 seconds

Question 44

1. Which is preferable, long polling or short polling?
2. which is enabled by default?
3. when is long polling in effect/how does it go into effect?

Answer 44

1. long polling
2. i beleive it’s short polling
3. long polling can be enabled at the queue level through the aws console, queue parameter Receive message wait time, or through API ReceiveMessageWaitTimeSeconds. Any non-zero value means long polling is in effect.

Question 45

Okay, so we’ve heard about how the message size limit for SQS queue is 256 KB. What happens if you need to send large messages? like message of over 1GB?

you

Answer 45

you use the SQS Extended Client (which is a Java Library)

Question 46

when you use the SQS Extended client library for java), what’s the max message size?

Answer 46

2GB

Question 47

when you use the SQS Extended client and are using it (the sqs extended client library for java), where are the messages stored?

Answer 47

in one of two places:
* Either in S3 all the time (even if the message is less than 256 Kb, though i’m not when you’re expected to check that),
* Or in S3 only when the messages are larger than 256 KB. I assume that in that event, the message smaller than 256 kb stay in the queue.

Question 48

If you’ve already set up SQS extended client and are using it (the sqs extended client library for java), how exactly do your consumers get a bigger message?

Answer 48

I think that your producers send a smaller message to the queue, and in that message is a reference to the bigger message object stored in an S3 bucket. Then when your producers pick up the message about the bigger message, the producers can go to the s3 bucket to get that bigger message.

Anyway, that’s my interpretation of what’s being implied by “You can use the Amazon SQS Extended Client Library for Java to do the following: […] Send a message that references a single message object stored in an S3 bucket. Retrieve the message object from an S3 bucket https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html”

Question 49

how can you use the sqs extended client library for java? ()

options: Amazon SQS Extended Client Library for Java with the AWS SDK for Java; the aws cli; the amazon sqs console; the Amazon SQS HTTP API; any of the other AWS SDKs.

Answer 49

You can use the Amazon SQS Extended Client Library for Java to manage Amazon SQS messages using Amazon S3 only with the AWS SDK for Java. You can’t do this with the AWS CLI, the Amazon SQS console, the Amazon SQS HTTP API, or any of the other AWS SDKs.

Question 50

What are the must know SQS APIs?

Answer 50

• CreateQueue
• MessageRetentionPeriod
• DeleteQueue
• PurgeQueue
• SendMessage
• DelaySeconds
• ReceiveMessage
• DeleteMessage
• MaxNumberOfMessages
• ReceiveMessageWaitTimeSeconds
• ChangeMessageVisibility
• SendMessageBatch
• DeleteMessageBatch
• ChangeMessageVisibilityBatch

Question 51

What does SQS API PurgeQueue do?

Answer 51

deletes all the messages in a queue

Question 52

1. what is the possible range of messages for MaxNumberOfMessages?
2. What is the default?

Answer 52

1. 1 to 10.
2. default is 1.

Question 53

What is the api you use to enable long polling?

Answer 53

ReceiveMessageWaitTimeSeconds

Question 54

What is sqs api ChangeMessageVisibility used for again?

Answer 54

Changes the visibility timeout of a specified message in a queue to a new value

Question 55

what

1. What does SendMessageBatch do?
2. How many SendMessage requests can you send with one SendMessageBatch request?

Answer 55

1. You can use SendMessageBatch to send up to 10 messages to the specified queue by assigning either identical or different values to each message (or by not assigning values at all). For a FIFO queue, multiple messages within a single batch are enqueued in the order they are sent.
2. 10

Question 56

1. What does SQS API DeleteMessageBatch do?
2. How many DeleteMessage requests can you send with one DeleteMessageBatch requests?

Answer 56

1. Deletes up to ten messages from the specified queue. This is a batch version of DeleteMessage. The result of the action on each message is reported individually in the response.
2. it doesn’t say how many you can delete with one DeleteMessageBatchRequest

Question 57

1. What does SQS API ChangeMessageVisibilityBatch do?
2. How many ChangeVisibilityMessage requests can you send with one ChangeMessageVisibilityBatch request?

Answer 57

1. Changes the visibility timeout of multiple messages. This is a batch version of ChangeMessageVisibility. The result of the action on each message is reported individually in the response.
2. You can send up to 10 ChangeMessageVisibility requests with each ChangeMessageVisibilityBatch action.”

Question 58

Is it more cost effect to use the batch APIs (SendMessageBatch, DeleteMessageBatch, and ChangeMessageVisibilityBatch) than to use more of their SendMessage, DeleteMessage and ChangeMessageVisibility counterparts?

Answer 58

Yes. It is more cost efficient to use the batch apis.

Question 59

What’s a batch api?

Answer 59

an api that allows you to send multiple apis. Ex: SendMessageBatch allows you to send data it would take 10 SendMessage ‘s to send.

I think. Based on skimming a couple of search results. really fast skilmming I could very well be wrong

Question 60

T/F

SQS FIFO has something called de-duplication. Deduplication means that if you send a message twice in the same five minutes, the second message is refused (steph and aws don’t say who’s doing the sending or refusing. At one point aws says it’s the producer that is creating the dedpulication id, which really just makes things fuzzier for me.)

Answer 60

T

Question 61

Okay, usually aws docs are very well written (noticably so) but the language describing deduplication is honestly a little vague. So, if you don’t entirely know what you’re memorizing here, that’s okay I guess. Or maybe you’ll learn more about this during the hands on?

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html

Question 62

What are the two SQS FIFO deduplication methods?

Answer 62

1. content based duplication: will do a SHA-256 hash of the message body
2. explicity provide a message deduplication id

Question 63

What’s the SQS FIFO deduplication interval?

Answer 63

5 minutes

Question 64

what is an important naming convention you have to adhere to when creating an SQS FIFO queue?

Answer 64

The queue name has to end in “.fifo”. For example, “DemoQueue.fifo”

Question 65

can you enable content-based deduplication with a Standard type SQS queue?

Answer 65

No. You can only enable content-based deduplication with FIFO type SQS queues. (I think).

Question 66

Did it look like you had to set a message deduplication id when you did not enable content-based deduplication when you created a FIFO type SQS queue?

Answer 66

Yes, because content-based deduplication is based on the message of the body (remember, it’s literally like a SHA 256 hash or something)

Question 67

Are there any unusual/unnamed naming convention rules that you need to adhere to when creating a Standard type SQS queue.

Answer 67

No. Just when you’re creating a FIFO type SQS queue. For those you need to make sure your queue names end in .fifo.

Question 68

Does message grouping/do message grouping ids apply to SQS queue type Standard?

Answer 68

No. just for SQS queue type FIFO.

Question 69

Okay, so what is message grouping, as it applies to SQS?

Answer 69

• So message grouping is something that happens in FIFO requests only (at least, the console makes it appear that it’s only for FIFO requests). It’s a way of making sure that groups of messages are associated with each other.
• messages in a message group get processed one by one
• you have to specify a MessageGroupID (you’re forced to). you may optionally specify a message deduplication id, to ensure duplicate messages are not sent.
• if you didn’t mention the FIFO specific nature of this topic, then you got it wrong.

From the picture it looks a lot like a consumer handles each message in a message grouping, but it’s difficult to find documentation to support that and in actuality I don’t believe that’s what’s happening, for reasons I cover in a slide you can find by command finding for “ID xyz”

Question 70

1. Say you want an app that handles submitting (and processing) a sequence of requests per user, and that users have to be served in the order according to whoever submitted thier first request first. (idk, a user has to fill out forms in a step or something, and you have a ton of users in line). What could you use?
2. Would your choice have changed if the users did not need to be served in any particular order?

Answer 70

1. Use SQS queue type FIFO.
2. Choice does not change if the users don’t need to be served in any particular order. If you need all messages in a group to be handled in an order, you have to use FIFO.

Question 71

In an SQS FIFO type queue, do message groups (like a list of message groups) get handled on a first in first out basis, or does FIFO only return to messages in a single group?

Answer 71

Message groups do not get handled in a FIFO manner, only messages in a message group get handled in a FIFO way.

Question 72

ID xyz

If you only had one form that users could submit, but you needed users to be served on a first come first served basis, how would using FIFO work, given that you have to provide a Message Group ID?

Answer 72

You just use the same message group ID for all messages. Then all the messages in that group get handled in order. This does kind of lead me to believe that a single message group can get handled by multiple consumers.