Section 17: Decoupling Applications: SNS, SQS, Kinesis, Active MQ

Question 1

What does “decoupling” mean in “decoupling applications”?

Answer 1

Decoupling refers to app components remaining autonomous and unaware of each other as they do their work

Question 2

;What are the two patterns of application communication?

Answer 2

1) Synchronous communication (app to app)
2) Asynchronous / Event based communication (application to queue to application)

Question 3

When can Synchronous communication become problematic and what is the solution?

Answer 3

Synchronous communication can become problematic if there are sudden spikes in traffic. The solution is to decouple your application

Question 4

What is SQS Queue?

Answer 4

SQS is a distributed message queue that can reliably queue messages, with one component consuming messages that are produced by another

Question 5

In terms of SQS Queue, what is a producer?

Answer 5

A producer is what sends messages to the queue (like an API)

Question 6

In terms of SQS Queue, what is a consumer?

Answer 6

A consumer is what receives messages from the queue, reading and deleting them

Question 7

What types of issues does a queue resolve, and how does it do so?

Answer 7

By acting as a buffer between applications, the queue resolves issues such as:
1) Producer is producing work faster than consumer can handle
2) Issues arising from intermittent network connectivity of either the producer, consumer, or both

Question 8

What is SQS queueing used for?

Answer 8

SQS queueing is used for decoupling application components so they can run independantly.

Question 9

Are SQS queues push or pull based?

Answer 9

SQS Queues are pull based, not pushed base. This means someone or something needs to be consuming the messages.

Question 10

How long do SQS messages persist?

Answer 10

A Consumer message persists in SQS until a consumer deletes it

Question 11

What are some common SQS consumers?

Answer 11

EC2 instances & Lambda functions are common SQS consumers

Question 12

How can you save on SQS consumer costs?

Answer 12

Use the DeleteMessage API to delete messages after processing them.

Question 13

How do consumers receive messages from SQS?

Answer 13

Consumers have to poll the SQS queue, and can only receive up to 10 messages at a time.

Question 14

What is a very common application architecture using SQS, and why is it beneficial?

Answer 14

An extremely common architecture is to use SQS to decouple a backend and frontend. This improves reliability and performance by allowing us to scale the backend and frontend independently.

Question 14

What is a very common application architecture using SQS, and why is it beneficial?

Answer 14

An extremely common architecture is to use SQS to decouple a backend and frontend. This improves reliability and performance by allowing us to scale the backend and frontend independently.

Question 15

What are the two SQS queue types?

Answer 15

1) SQS Standard Queues (default)
2) SQS FIFO (First in First Out) Queues

Question 16

SQS Standard Queue has what characteristics?

Answer 16

1) Almost unlimited TPS, automatic scaling
2) Guarantees a message is delivered at-least once
3) Can have out of order messages, even though it uses best-effort ordering (app should be able to cope with this)

Question 17

SQS FIFO Queue has what characteristics?

Answer 17

1) Message Order is strictly preserved
2) Exactly-once processing (no duplicate messages)
3) Max 300 TPS

Question 18

What are the minimum, maximum, and default SQS message retention periods?

Answer 18

Minimum: 1 minutes
Maximum: 14 days
Default: 4 days

Question 19

What is the SQS Message Visibility Timeout?

Answer 19

After a message is polled by a consumer, it becomes invisible to other consumers

Question 20

What happens if a message cannot be processed within the visbility timeout?

Answer 20

The message will be processed twice, or the consumer could call ChangeMessage Visibility API to get more time

Question 21

What might occur is visibility timeout is low? What if its high?

Answer 21

If the visibility timeout is low (seconds) we may get duplicates. If it is high (hours), a consumer crashing would mean a long re-processing period.

Question 22

What is the dead letter queue?

Answer 22

the dead letter queue (DLQ) is where messages go after they have gone back into the queue too many times.

Question 23

What is long polling?

Answer 23

Long polling = a consumer optionally waits for messages to arrive to the queue if there are currently none in the queue

Question 24

How does Long polling save money?

Answer 24

It saves money on polling costs by reducing API calls, latency, and reduces the number of empty responses.

Question 25

What is Amazon SNS

Answer 25

SNS = Simple notification service, a service that uses a publisher. subscriber system to push messages to many receivers.

Question 26

How does SNS work?

Answer 26

An event producer sends messages to an SNS topic, which then has many event receivers (subscribers) listening for the SNS topic notification.

Question 27

What type of services can be SNS topic subscribers?

Answer 27

Lambda, HTTP/HTTPs endpoints, SQS, Emails, SMS / mobile notifications, etc.

Question 28

What does “fan out” mean in relation to SNS?

Answer 28

“fanning out” is when an SNS topic can deliver notifications to many downstream subscribers of different types (SQS, Lambda, Email, etc.)

Question 29

What is an SNS topic?

Answer 29

An SNS topic is an access point for subscribers to all receive notifications. One topic can deliver to multiple endpoint types (IOS vs Android etc.) with each getting the message in the proper format

Question 29

What is an SNS topic?

Answer 29

An SNS topic is an access point for subscribers to all receive notifications. One topic can deliver to multiple endpoint types (IOS vs Android etc.) with each getting the message in the proper format

Question 30

What are the main benefits of SNS?

Answer 30

1) Instantaneous and push-based (no need for polling)
2) Flexible delivery over multiple transport protocols
3) inexpensive pay as you go system

Question 31

Describe the SNS + SQS fan out architectural pattern and its main use case

Answer 31

SNS topic as the producer pushes to multiple subscriber SQS queues (fanning out). This is a fully decoupled architecture with no data loss. The main use case is for sending the same S3 event type (like create object) to many SQS queues

Question 32

What is Amazon Kinesis?

Answer 32

Kinesis is a platform to load and analyze streaming data, making it easy to collect, process and analyze streaming data in real time.

Question 33

What are the four (4) Kinesis platforms?

Answer 33

1) Kinesis Data Streams
2) Kinesis Firehose
3) Kinesis Data Analytics
4) Kinesis Video Streams

Question 34

What is a Data Streaming?

Answer 34

A data stream = data that is generated continuously, from many sources, but sent simultaneously, in small sizes.

Question 35

What does Kinesis Video Streams do?

Answer 35

Kinesis Video Streams allows you to stream video content from connected devices to AWS for analytics, ML, and other processing.

Question 36

What is Kinesis Data Streams?

Answer 36

Kinesis Data Streams is scalable, durable, low-latency real time data streaming and ingestion at scale.

Question 37

How do you scale Kinesis Data Streams?

Answer 37

Kinesis Data Streams scales with sharding. There is one shard per type of data source. More shards gives more throughput, but is also more expensive.

Question 38

What does Kinesis Firehose do?

Answer 38

Kinesis Firehose allows you to load data into other AWS services and store data into target destinations.

Question 39

When moving to cloud, instead of re-engineering to use SQS or SNS, what service should be used?

Answer 39

Amazon MQ

Question 40

What does Kinesis Data Analytics allow you to do?

Answer 40

Kinesis Data Analytics allows you to perform real time analytics on data streams using SQL. Useful for real time dashboards and metrics.