Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

AWS Architect Associate > Decoupling applications: SQS, SNS, Kinesis, Active MQ > Flashcards

Decoupling applications: SQS, SNS, Kinesis, Active MQ Flashcards

1
Q

Two patterns of application communication

A

Synchronous (app to app)

Asynchronous / Event based (app to queue to app)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Synchronous problems

A

Problematic if sudden spikes of traffic appear

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

How can you decouple your apps? (to avoid sudden spike of traffic issues)

A

SQS - Queuing model
SNS - pub/sub model
Kinesis - real time streaming model

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What is the benefit of decoupling?

A

Services (sqs, sns, kinesis) decouple independently from the app

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

SQS architecture

A

Producer sends messages to SQS queue, then the Consumer polls the messages to process them. Once processes they get deleted

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What is the purpose of an SQS queue?

A

Acts as a buffer to decouple between producers (sender) and consumers (processor)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Standard Queue

A

Fully managed

Unlimited throughput and messages in queue

Retention = 4-14 days
Low latency
256KB per message

Can have duplicates
Can have out of order mesasges

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What do producers do

A

Send to SQS using SDK (SendMessage API)
message persists 4-14 days until consumer deletes it

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

How do consumers work

A

Consumers run on EC2 instances, servers or AWS Lambda

Polls / Receive SQS messages (up to 10 at a time)

Process message (e.g. insert into RDS DB)
Delete message using DeleteMessage API

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Multiple EC2 instances consumers

A

Can have many consumers in SQS Queue

receive and process messages in parallel

At least once delivery (because they can work on one message at the same time)

Best-effort message ordering

Message deleted after processed

Can scale consumers horizontally to improve throughput of processing

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

SQS with ASG

A

SQS polls messages to EC2 instances (consumers) which are in an ASG.

CloudWatch Metric - Queue Length (ApproximateNumberOfMessages) will set an alarm and send it to CloudWatch.

If alarm is triggered, CloudWatch notifies ASG

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

How to decouple using SQS?

A

Instead of doing requests & processing on front end web app you can:

Receive the requests in the front-end web app (in an ASG) and send them to SQS.

SQS will send them to the back-end processing application (video processing) which is in an ASG. The processing is done here.

Then get sent to the S3 bucket once done.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

SQS Security

A

Encryption in flight using HTTPS API, at rest encryption using KMS keys.
Client-side encryption if the client wants to handle encryption/decryption

Access Controls - IAM policies to regulate access to SQS API

SQS Access Policies (similar to S3 bucket policies)
-useful for cross-account access to SQS queues
- allowing other services (SNS, S3) to write to an SQS queue

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Message visibility timeout

A

When message gets polled by consumer it becomes invisible to other consumers (30 seconds)

After that it will be visible in the queue and another consumer can work on it

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

ChangeMessageVisibility API

A

When a consumer knows that they need more than 30 seconds (default) to process a message, they can request this API and extend the invisibility window.

This will help with not processing the same message twice

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Long Polling

A

When a consumer requests messages from queue but they want to “wait” for messages to arrive if there are non in the queue.

Can reduce latency
Can reduce API calls

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Long Polling how it works

A

Consumer polls for up to 20 sec, waiting and when a message is received in the SQS queue, then it polls it and processes it.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Downsides of FIFO

A

Limited throughput 300msg/s without batching and 3000 with batching.

Without FIFO there is unlimited throughput

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

Benefits of FIFO

A

exactly once send capability (removing duplicates)

Messages processed in order by consumer

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

How to use SQS as a Buffer to database writes

A

Requests go in your EC2 within an ASG. They get enqueued, meaning they get sent to SQS Queue (infinitely scalable) and then they get sent from the SQS Queue to another ASG which holds Dequeueing EC2 instances. These requests get sent as messages.

From that Dequeueing SQS the messages get insert into the Databases.

This makes sure nothing is lost. Once they are in the DB, then they can be deleted from SQS.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

Amazon SNS

A

Send one message to multiple receivers

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

How to use SNS with Pub / Sub?

A

Buying service -> SNS topic which has multiple subscribers (services) such as email notifications, fraud service, shipping service, SQS queue and they all receive the messages

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

SNS how it works

A

even producer only sends to ONE SNS topic

can have as many event receivers as we want

Up to 12,500,000 subs per topic, 100,000 topics limit

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

Where can SNS public to?

A

SQS, Lambda, Kinesis Data Firehose, Emails, SMS & Mob notifications, HTTP(S) endpoints

25
Q

How does SNS public?

A

Topic Publish (using SDK)

Create topic, create subs, publish to topic

OR

Direct Publish (mobile apps SDK)
create platform app
create platform endpoint
publish platform endpoint
works with google GCM, Apple APNS, Amazon ADM

26
Q

SNS Security

A

Inflight encryption using HTTPS API
at rest encryption using KMS
client-side encryption

Access controls - IAM policies to regulate access to SNS API

SNS Access policies
useful for cross-acc access to SNS topics
useful for allowing other services (S3) to write to an SNS topic

27
Q

SNS & SQS Fan Out

A

Buying service sends message to SNS Topic. From there it gets sent to 2 or more SQS Queues. Each queue sends the request to the services associated with them (fraud, shipping service etc)

28
Q

SNS & SQS Fan Out benefits

A

Push one in SNS receive in all SQS subs
Fully decoupled, no data loss
SQS allows for data persistence, delayed processing and retries of work
ability to add more SQS subs over time
SQS access policy should allow for SNS to write
Cross-region delivery

29
Q

How to send S3 events to multiple queues?

A

Usually you can only have 1 event and prefix (1 S3 event rule)

To send same S3 event to many SQS queues you need to fan out.

S3 object created, send to S3, then to SNS topic, fans out to SQS Queues, lambda function etc

30
Q

SNS to Amazon S3 through Kinesis Data Firehose

A

Buying service -> SNS topic -> Kinesis Data Firehose -> S3 and any Supported KDF Destination

31
Q

SNS Message Filtering

A

JSON policy to filter messages sent to SNS topics subs

If a sub doesnt have a filter policy, it receives every message

32
Q

How does SNS Message Filtering work

A

Buying service sends new transaction with data (order num, product, qty, state) to SNS Topic.

From there it gets sent to SQS Queues. First SQS Queue has placed orders policy filter and only receives placed orders. The next one has cancelled orders, other one declined orders, and another SQS Queue has all.

33
Q

Kinesis Overview

A

Collect, process, analyse streaming data in real-time.

34
Q

What does Kinesis ingest?

A

Real time data such as

App logs
Metrics
Website clickstreams
IoT telemetry data

35
Q

4 types of Kinesis

A

Data Streams - capture, process, store data streams

Data Firehose - load data streams into AWS data stores

Data Analytics - analyse data streams with SQL or Apache Flink

Video Streams - capture, process, store, video streams

36
Q

What do Kinesis Data Streams include?

A

They are composed by Shards (1,2,3, etc) and you need to provision them beforehand.

They define the stream capacity in terms of ingestion and consumption rights.

37
Q

What do producers do for Kinesis Data Streams?

A

They produce records to Kinesis Data Streams.

They could be
EC2
Client
SDK, KPL
Kinesis Agent

38
Q

What do records sent by Producers to Kinesis Data Streams include?

A

2 parts

Partition Key - decides which Shard to go to
Data Blob (up to 1 mb) or value

Can send 1 MB/sec or 1000 msg/sec per shard

39
Q

What are the consumers for Kinesis Data Streams?

A

Apps (KCL, SDK)
Lambda
Kinesis Data Firehose
Kinesis Data Analytics

40
Q

What does the record that goes into Consumers include?

A

Partition Key
Sequence no.
Data Blob

2 MB/sec (shared) per shard all consumers OR 2 MB/sec (enhanced) per shard per consumer)

41
Q

Kinesis Data Streams Architecture

A

Producer (apps, clients etc) -> record (partition key, blob) -> kinesis data stream (shards) -> record (partition key, sequence number, blob) -> consumers (apps, lambda, firehose, kinesis data analytics)

42
Q

Properties of Kinesis Data Streams

A

1-365 days retention period
ability to reprocess (replay) data
cant be deleted if added to kinesis (immutability)
data sharing partition go to same shard (ordering)

producers
-AWS SDK, Kinesis Producer Library (KPL), Kinesis Agent

consumers
- write your own: Kinesis Client Library (KCL), AWS SDK
- managed: AWS Lambda, kinesis Data Firehose, Kinesis Data Analytics

43
Q

Capacity Modes for Kinesis Data Streams

A

Provisioned Mode
- choose n of shards provisioned, scale manually or using API
- each shard gets 1MB/s in (or 1000 records per sec)
- Each shard gets 2MB/s out (classic or enhanced fan-out consumer)
- pay per shard provisioned per hour

On Demand Mode
- no need to provision or manage capacity
- default capacity provisioned (4MB/s in or 4000 records per sec)
- automatic scaling based on observed throughput peak during last 30 days
- pay per stream per hour & data in/out per GB

44
Q

Kinesis Data Streams Security

A

Deployed within a Region

  • Control access / auth using IAM policies
  • HTTPS endpoints encryption in flight
  • at rest using KMS
  • implement encryption/decryption of data on client side (harder)
  • VPC endpoints available for Kinesis to access within VPC
  • Monitor API calls using CloudTrail
45
Q

Kinesis Data Firehose

A

Takes data from Producers (Apps, Clients, Kinesis Data Streams, CloudWatch, AWS IoT)

And write data into destinations (batch writes)

Transformation of data can happen via Lambda functions

46
Q

Kinesis Data Firehose destinations

A

AWS Destinations (s3, RedShift, Opensearch)

3rd-party Partner Destinations (splunk, mongoDB)

Custom Destinations (HTTP endpoints)

47
Q

Kinesis Data Firehose characteristics

A

Fully managed
redshift, S3, opensearch
3rd party partners
custom - http

pay for data going through firehose
Near real time
- 60 sec latency
- min 1 MB of data at time

supports many data formats, conversations, transformations, compression

supports custom data transformations using AWS lambda

can send failed or all data to a backup S3 bucket

48
Q

Kinesis Data Streams vs Firehose

A

Kinesis Data Streams
- streaming service for ingest at scale
- write custom code (producer / consumer)
- real time (200ms)
- manage scaling (shard splitting / merging)
- data storage 1 to 365 days
- supports replay capability

Kinesis Data Firehose
- load streaming data to S3, Redshift, Opensearch, 3rd Party, Custom HTTP
- fully managed
- near real-time (buffer time min. 60 sec)
- automatic scaling
- no data storage
- doesnt support replay capability

49
Q

Data ordering in Kinesis

A

If you have 3 shards and 3 objects. If one objects partition key says “object_id” and gets sent to shard one.

Then the second object id is for shard 2. Every time they send data, the data will be sent to the shard they were assigned to

50
Q

SQS Data Ordering

A

Since there is no data ordering in SQS unless you use FIFO.

You can use Group ID (like partition key in Kinesis) and then you can have different Groups.

This allows to have more consumers Otherwise you can only have 1 consumer

51
Q

SQS vs SNS vs Kinesis

A

SQS
- consumer pulls data
- data deleted after consumed
- can have many consumers
- no need to provision
- order guarantee only with FIFO
- individual message delay capability

SNS
- push data to subscribers
- up to 12,500,000 subs
- data not persistent (lost if not delivered)
- pub / sub
- up to 100,000 topics
- no need to provision
- integrates with SQS for fan-out
- FIFO capability for SQS FIFO

Kinesis
- Pull data (2 MB per shard)
- Push data enhanced fan-out (2 MB per shard per consumer)
- replay data possibility
- for real-time big data, analytics
- ordering at shard level
- data expiration after x days
- provisioned more or on-demand capacity mode

52
Q

Amazon MQ

A

Managed Message Broker service for

RabbitMQ
ActiveMQ

53
Q

Amazon MQ downsides

A

does not scale as much as SQS/SNS

54
Q

How to have HA with Amazon MQ?

A

runs on servers so you can have Multi-AZ with failover

55
Q

HA architecture for Amazon MW

A

1 Region, 1 Active AZ, 1 Standby AZ, 1 Amazon EFS

If client is talking to Amazon MQ broker, and it crashes, then the Client will start talking to the one in the Standby AZ, and nothing will be lost because both AZ were connected to EFS.

56
Q

Does SQS scale automatically?

A

Yes

57
Q

What should you do in Kinesis when you have 6 shards but traffic spikes and you get a ProvisionedThroughputExceeded warning?

A

Add more shards

58
Q

Is Amazon Kinesis Data Streams a supported subscriber for AWS SNS?

A

No

AWS Architect Associate (15 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions