Queues and Notifications Flashcards
What types of Application communication are available?
Synchronous and Asynchronous
What synchronous models are available for application communication that are scalable?
- SQS - queue model
- SNS - pub/sub model
- Kinesis - real-time streaming model
What is SQS?
A synchronous application communication model that can have many producers (push) and consumers (poll and can consume up to 10 message at a time) from a single queue.
What are the pros and cons of and SQS Standard Queue?
Pros
- Fully managed by AWS
- scales from 1 message/sec to 10,000s/sec
- no message limit in queue
- low latency
- no consumer limit
- can delay individual messages
Cons
- Can contain duplicate messages
- Can process messages out of order
- Has limitation of 256kb/message
- Data is deleted after it is consumed
- no ordering gaurantee (except in FIFO queues)
What is an SQS delay queue?
This is a delay setting that allows a message to be hidden for up to 15 mins from consumers. It will override the delay delivery parameter that can be set on the individual message.
What is SQS visibility?
This is the time the message is visible by the consumer to pick up. Timeouts range from 0 sec and 12 hours (default 30 sec).
if set too low another consumer could receive the same message.
if set too high and consumer fails to process the message then you have to wait until after timeout to try again.
What is an SQS dead letter queue?
This is when the redrive policy (i.e. number of consumption tries) has been met and the message has still not been processed, the message is sent to the dead letter queue (DLQ). Note: You want to make sure you process the messages in the DLQ before they expire.
What is SQS Long Polling?
This is the process of a consumer waiting for a message to arrive in the queue. This can range from 1 - 20 secs and can be enabled at the queue or API level using the WaitTImeSeconds setting.
What is SQS FIFO Queue?
A newer offering that sets the queue to allow polling of First In-First Out messages.
- name of the queue must end in .fifo
- messages are processed in order by the consumer
- message are sent exactly once
- ability to do content based de-duplication at a 5min interval using Duplication ID.
- ability to group messages using Message Group ID (i.e. just an extra tag on the message)
Can SQS be used with ASG?
Yes, by setting the SQS Queue to poll for messages from a cluster of EC2 instances and setting up a CloudWatch custom metric based on Queue length/Number of instances, you can trigger a CloudWatch Alarm to autoscale.
What is AWS SNS?
This allows message to be sent to multiple consumers using a pub/sub model. The publisher sends the message to one SNS topic and the subscriber polls the SNS topic for notifications. Each subscriber to the topic will receive all the messages unless a filter is used.
What are the characteristics of SNS?
- up to 10,000,000 subscriptions/topic
- limit of 100,000 topics
- can push data to many subscribers
- Subscribers can use (SQS, HTTP/HTTPS, Lambda, Emails, SMS Messages, or Mobile Notifications)
- data is lost if not delivered
- does not require provisioned throughput
- can be used in conjunction with SQS where the SQS Queue is a subscriber to the SNS Topic. (called SQS Fan-out pattern)
What AWS services can SNS integrate with?
- CloudWatch
- Auto Scaling Groups
- S3 (on bucket events)
- Cloud Formation
How do we publish and SNS Model?
- Individual Topic publication
1. Create a topic
2. create one or more subscriptions
3. publish to the topic
or
*Direct publish (works with Google GCM, Apple APNS, Amazon ADM, etc.)
1. Create a platform application
2. Create a platform endpoint
3. Publish to the platform endpoint
What is AWS Kinesis?
A managed alternative to Apache Kafka. Great for real-time big data, logs, metrics, IoT, and clickstreams and is automatically replicated to 3 AZs.
- Consumers “pull data”
- no consumer limit
- Ability to replay data
- Can order data at the shard level
- Data expires after X days
- must provision throughput
