Data Engineering

Question 1

Amazon Kinesis Data Streams

Answer 1

Collect and store streaming data in real-time
1. Retention up to 365 days (can’t be deleted until it expires)
2. Data ordering guarantee

Question 2

Kinesis Data Streams - Capacity Modes

Answer 2

Provisioned mode

• Choose number of shards
• Each shard gets 1000 records per second
• Scale manually
• Pay per shard provisioned per hour

On demand mode:

• Default capacity provisiones (4000 records per second)
• Scale based on observed throughput peaks
• Pay per stream per hour & data in/out

Question 3

Amazon Data Firehose - AWS Destinations

Answer 3

• Amazon S3: Supports compression
• Amazon Redshift (copy through S3)
• Amazon OpenSearch

Question 4

Amazon Data Firehose

Answer 4

1. Receive records (up to 1MB) from producers
2. Can make data transformation with Lambda functions
3. Batch writes to destinations based on buffer time and size, Near Real Time
4. All or failed data can be backup in S3

Question 5

Firehose Buffer Sizing

Answer 5

Firehose accumulates records in a buffer and is flushed based on time and size rules

Question 6

Amazon Managed Service for Apache Flink

Answer 6

Framework for processing data streams. Can’t read from Amazon Data Firehose.

Question 7

Amazon Managed Streaming For Apache Kafka

Answer 7

Amazon MSK creates & manages Kafka brokers nodes.
* Deployed in your VPC, multi-AZ
* Data stores on EBS volumes

Question 8

Kinesis Data Streams vs Amazon MSK

Answer 8

Kinesis Data Streams:
* 1 MB message size limit
* 12 months maximum retention
* Shard splitting and merging

Amazon MSK:
* Configure for bigger messages
* No retention limit
* Can only add partitions to a topic

Question 9

AWS Batch

Answer 9

Run batch jobs as Docker images

Question 10

AWS Batch - Multi node mode

Answer 10

Leverages multiple EC2 / ECS instances. One main node and multiple childs. Doesn’t work with Spot Instances.

Question 11

Amazon Elastic MapReduce

Answer 11

EMR creates Hadoop clusters in a single AZ to analyze and process vast amount of data

Question 12

EMR File System

Answer 12

EMRFS stores persistent data in Amazon S3 while providing data encryption

Question 13

EMR node types

Answer 13

• Master node: Manage the cluster
• Core node: Run tasks and store data
• Task node: Justo to run tasks, usually uses spot

Question 14

EMR instance configuration

Answer 14

• Uniform instance groups: select a single instance type and purchasing option for each node. Has auto scaling.
• Instance fleet: Select target capoacity, mix instance types and purchasing options. No auto scaling.

Question 15

AWS Glue

Answer 15

Managed extract, transform and load service

Question 16

AWS Glue Data Catalog

Answer 16

Data crawler writes metadata of databases and supportes storages to the data catalog