Section 17: Decoupling Applications: SNS, SQS, Kinesis, Active MQ Flashcards
What does “decoupling” mean in “decoupling applications”?
Decoupling refers to app components remaining autonomous and unaware of each other as they do their work
;What are the two patterns of application communication?
1) Synchronous communication (app to app)
2) Asynchronous / Event based communication (application to queue to application)
When can Synchronous communication become problematic and what is the solution?
Synchronous communication can become problematic if there are sudden spikes in traffic. The solution is to decouple your application
What is SQS Queue?
SQS is a distributed message queue that can reliably queue messages, with one component consuming messages that are produced by another
In terms of SQS Queue, what is a producer?
A producer is what sends messages to the queue (like an API)
In terms of SQS Queue, what is a consumer?
A consumer is what receives messages from the queue, reading and deleting them
What types of issues does a queue resolve, and how does it do so?
By acting as a buffer between applications, the queue resolves issues such as:
1) Producer is producing work faster than consumer can handle
2) Issues arising from intermittent network connectivity of either the producer, consumer, or both
What is SQS queueing used for?
SQS queueing is used for decoupling application components so they can run independantly.
Are SQS queues push or pull based?
SQS Queues are pull based, not pushed base. This means someone or something needs to be consuming the messages.
How long do SQS messages persist?
A Consumer message persists in SQS until a consumer deletes it
What are some common SQS consumers?
EC2 instances & Lambda functions are common SQS consumers
How can you save on SQS consumer costs?
Use the DeleteMessage API to delete messages after processing them.
How do consumers receive messages from SQS?
Consumers have to poll the SQS queue, and can only receive up to 10 messages at a time.
What is a very common application architecture using SQS, and why is it beneficial?
An extremely common architecture is to use SQS to decouple a backend and frontend. This improves reliability and performance by allowing us to scale the backend and frontend independently.
What is a very common application architecture using SQS, and why is it beneficial?
An extremely common architecture is to use SQS to decouple a backend and frontend. This improves reliability and performance by allowing us to scale the backend and frontend independently.
What are the two SQS queue types?
1) SQS Standard Queues (default)
2) SQS FIFO (First in First Out) Queues
SQS Standard Queue has what characteristics?
1) Almost unlimited TPS, automatic scaling
2) Guarantees a message is delivered at-least once
3) Can have out of order messages, even though it uses best-effort ordering (app should be able to cope with this)
SQS FIFO Queue has what characteristics?
1) Message Order is strictly preserved
2) Exactly-once processing (no duplicate messages)
3) Max 300 TPS
What are the minimum, maximum, and default SQS message retention periods?
Minimum: 1 minutes
Maximum: 14 days
Default: 4 days
What is the SQS Message Visibility Timeout?
After a message is polled by a consumer, it becomes invisible to other consumers
What happens if a message cannot be processed within the visbility timeout?
The message will be processed twice, or the consumer could call ChangeMessage Visibility API to get more time
What might occur is visibility timeout is low? What if its high?
If the visibility timeout is low (seconds) we may get duplicates. If it is high (hours), a consumer crashing would mean a long re-processing period.
What is the dead letter queue?
the dead letter queue (DLQ) is where messages go after they have gone back into the queue too many times.
What is long polling?
Long polling = a consumer optionally waits for messages to arrive to the queue if there are currently none in the queue
How does Long polling save money?
It saves money on polling costs by reducing API calls, latency, and reduces the number of empty responses.
What is Amazon SNS
SNS = Simple notification service, a service that uses a publisher. subscriber system to push messages to many receivers.
How does SNS work?
An event producer sends messages to an SNS topic, which then has many event receivers (subscribers) listening for the SNS topic notification.
What type of services can be SNS topic subscribers?
Lambda, HTTP/HTTPs endpoints, SQS, Emails, SMS / mobile notifications, etc.
What does “fan out” mean in relation to SNS?
“fanning out” is when an SNS topic can deliver notifications to many downstream subscribers of different types (SQS, Lambda, Email, etc.)
What is an SNS topic?
An SNS topic is an access point for subscribers to all receive notifications. One topic can deliver to multiple endpoint types (IOS vs Android etc.) with each getting the message in the proper format
What is an SNS topic?
An SNS topic is an access point for subscribers to all receive notifications. One topic can deliver to multiple endpoint types (IOS vs Android etc.) with each getting the message in the proper format
What are the main benefits of SNS?
1) Instantaneous and push-based (no need for polling)
2) Flexible delivery over multiple transport protocols
3) inexpensive pay as you go system
Describe the SNS + SQS fan out architectural pattern and its main use case
SNS topic as the producer pushes to multiple subscriber SQS queues (fanning out). This is a fully decoupled architecture with no data loss. The main use case is for sending the same S3 event type (like create object) to many SQS queues
What is Amazon Kinesis?
Kinesis is a platform to load and analyze streaming data, making it easy to collect, process and analyze streaming data in real time.
What are the four (4) Kinesis platforms?
1) Kinesis Data Streams
2) Kinesis Firehose
3) Kinesis Data Analytics
4) Kinesis Video Streams
What is a Data Streaming?
A data stream = data that is generated continuously, from many sources, but sent simultaneously, in small sizes.
What does Kinesis Video Streams do?
Kinesis Video Streams allows you to stream video content from connected devices to AWS for analytics, ML, and other processing.
What is Kinesis Data Streams?
Kinesis Data Streams is scalable, durable, low-latency real time data streaming and ingestion at scale.
How do you scale Kinesis Data Streams?
Kinesis Data Streams scales with sharding. There is one shard per type of data source. More shards gives more throughput, but is also more expensive.
What does Kinesis Firehose do?
Kinesis Firehose allows you to load data into other AWS services and store data into target destinations.
When moving to cloud, instead of re-engineering to use SQS or SNS, what service should be used?
Amazon MQ
What does Kinesis Data Analytics allow you to do?
Kinesis Data Analytics allows you to perform real time analytics on data streams using SQL. Useful for real time dashboards and metrics.
