AWS Integration & Messaging

Question 1

There are two patterns of application communication

Answer 1

1) Synchronous communications (application to application)

2) Asynchronous / Event based (application to queue to application)

Question 2

name all three different models for decoupling your application

Answer 2

• using SQS: queue model
• using SNS: pub/sub model
• using Kinesis: real-time streaming model

Question 3

SQS - How many messages can be in the queue?

Answer 3

Unlimited

Question 4

SQS - What’s the size lmitation per message sent?

Answer 4

256kb

Question 5

What’s SQS – Delay Queue

Answer 5

• Delay a message (consumers don’t see it immediately) up to 15 minutes • Default is 0 seconds (message is available right away)

Question 6

What parameter overrides the default SQS – Delay Queue?

Answer 6

DelaySeconds parameter

Question 7

What can you expect to get back from SQS

Answer 7

• Message identifier

* MD5 hash of the body

Question 8

What’s the lifecycle of SQS - Consuming Messages?

Answer 8

• Poll SQS for messages (receive up to 10 messages at a time) • Process the message within the visibility timeout
• Delete the message using the message ID & receipt handle

Question 9

What is SQS - Visibility timeout?

Answer 9

When a consumer polls a message from a queue, the message is “invisible” to other
consumers for a defined period… the Visibility Timeout:

Question 10

What is the limits and default for SQS - Visibility timeout?

Answer 10

• Set between 0 seconds and 12 hours (default 30 seconds)

Question 11

SQS - What’s the API to change the visibility while processing a message?

Answer 11

ChangeMessageVisibility

Question 12

What’s the API to tell SQS the message was successfully processed?

Answer 12

DeleteMessage

Question 13

What is a Dead Letter Queue?

Answer 13

-If a consumer fails to process a message within the Visibility Timeout…
the message goes back to the queue!
• We can set a threshold of how many times a message can go back to the queue – it’s called a “redrive policy”
• After the threshold is exceeded, the message goes into a dead letter queue (DLQ)
• We have to create a DLQ first and then designate it dead letter queue
-Make sure to process the messages in the Failure DLQ before they expire!

Question 14

Define AWS SQS - Long Polling.

Answer 14

-When a consumer requests message from the queue, it can optionally “wait” for messages to arrive if there are none in the queue
• This is called Long Polling
• LongPolling decreases the number of API calls made to SQS while increasing the efficiency and latency of your application.
-The wait time can be between 1 sec to 20 sec (20 sec preferable)

Question 15

SQS Long polling can be enabled at the queue level or at the API level using what param?

Answer 15

WaitTimeSeconds

Question 16

SQS - FIFO has higher or lower throughput?

Answer 16

Lower throughput (up to 3,000 per second with batching, 300/s without)

Question 17

SQS - FIFO Message groups allow for what?

Answer 17

Possibility to group messages for FIFO ordering using “Message GroupID”

Question 18

SNS - Up to ___ subs per topic

Answer 18

Up to 10,000,000 subscriptions per topic

Question 19

SNS - What’s the topics limit?

Answer 19

100,000 topics limit

Question 20

SNS - Who can subscribe?

Answer 20

• SQS
• HTTP / HTTPS (with delivery retries – how many times)
• Lambda
• Emails
• SMS messages
• Mobile Notifications

Question 21

SNS - How do you publish a topic (within your AWS Server – using the SDK)?

Answer 21

• Create a topic
• Create a subscription (or many)
• Publish to the topic

Question 22

Explain AWS Kinesis

Answer 22

• Kinesis is a managed alternative to Apache Kafka
• Great for application logs, metrics, IoT, clickstreams
• Great for “real-time” big data
• Great for streaming processing frameworks (Spark, NiFi, etc…)

Question 23

Kineses Data is automatically replicated to how many AZ’s?

Answer 23

Data is automatically replicated to 3 AZ

Question 24

What are the three types of Kinesis?

Answer 24

• Kinesis Streams: low latency streaming ingest at scale
• Kinesis Analytics: perform real-time analytics on streams using SQL
• Kinesis Firehose: load streams into S3, Redshift, ElasticSearch…

Question 25

Kinesis Streams are ordered into?

Answer 25

Shards / Partitions

Question 26

Kinesis Streams Data retention is what by default and can go up to how many days?

Answer 26

Data retention is 1 day by default, can go up to 7 days

Question 27

Kinesis Streams give you the ability to reprocess / replay data?

Answer 27

True

Question 28

Kineses Streams - Only a single app can consume a single stream. T / F?

Answer 28

False - Multiple applications can consume the same stream

Question 29

Kineses Streams - Once data is inserted in Kinesis, it can’t be deleted (immutability) T/F

Answer 29

True

Question 30

Kineses Streams Shards - What size or how many messages per second?

Answer 30

1MB/s or 1000 messages/s at write PER SHARD

Question 31

Kineses Streams - How many MB per read?

Answer 31

2MB/s at read PER SHARD

Question 32

Kinese Streams Billing?

Answer 32

Billing is per shard provisioned, can have as many shards as you want

Question 33

Kinesis Streams shard records are ___ per shard?

Answer 33

Ordered

Question 34

How to reduce costs and increase throughput of kineses streams?

Answer 34

Use Batching with PutRecords

Question 35

AWS Kinesis API - ProvisionedThroughputExceeded Exceptions. Explain this

Answer 35

Happens when sending more data (exceeding MB/s or TPS for any shard)
• Make sure you don’t have a hot shard (such as your partition key is bad and too much data goes to that partition)

Solution
• Retries with backoff
• Increase shards (scaling)
• Ensure your partition key is a good one

Question 36

AWS Kinesis API – Consumers have two options in terms of use.. Normal consumer (CLI, SDK..) and…

Answer 36

• Can use a normal consumer (CLI, SDK, etc…)
  • Can use Kinesis Client Library (in Java, Node, Python, Ruby, .Net)
  • KCL uses DynamoDB to checkpoint offsets
  • KCL uses DynamoDB to track other workers and share the work amongst shards

Question 37

Kineses Security - Control access / authorization using..

Answer 37

IAM policies

Question 38

Kineses Security - Encryption in flight using…

Answer 38

HTTPS endpoints

Question 39

Kineses Security - Encryption at rest using…

Answer 39

KMS

Question 40

Is the possibility to encrypt / decrypt data client side there?

Answer 40

Yes but it’s harder

Question 41

What support is there for VPC access?

Answer 41

VPC Endpoints available for Kinesis to access within VPC

Question 42

Explain AWS Kinesis Data Analytics

Answer 42

Perform real-time analytics on Kinesis Streams using SQL
• Kinesis Data Analytics: • Auto Scaling
• Managed: no servers to provision • Continuous: real time
• Pay for actual consumption rate
• Can create streams out of the real-time queries

Question 43

Explain AWS Kinesis Firehose

Answer 43

• Fully Managed Service, no administration
• Near Real Time (60 seconds latency)
• Load data into Redshift / Amazon S3 / ElasticSearch / Splunk • Automatic scaling
• Support many data format (pay for conversion)
• Pay for the amount of data going through Firehose

Question 44

What is Amazon MQ

Answer 44

managed Apache ActiveMQ. When migrating to the cloud, instead of re-engineering the application to use SQS and SNS, we can use Amazon MQ

• Amazon MQ doesn’t “scale” as much as SQS / SNS
• Amazon MQ runs on a dedicated machine, can run in HA with failover
• Amazon MQ has both queue feature (~SQS) and topic features (~SNS)