1 AWS ML Eng Assoc Data Eng and Storage

Question 1

AWS S3

Answer 1

Amazon S3 is an object storage service that offers industry-leading scalability; durability; availability; security; and performance. Customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases; such as data lakes; websites; mobile applications; backup and restore; archive; enterprise applications; IoT devices; and big data analytics.

Question 2

S3 Buckets

Answer 2

S3 stores data as objects within buckets. You can store any type of object in a bucket; such as text files; photo files; video files; etc. Buckets are created in a specific AWS Region and can be accessed from anywhere.

Question 3

S3 Object Key

Answer 3

The name of the object is known as the key. The combination of bucket name and key uniquely identifies the object.

Question 4

EBS Volumes

Answer 4

EBS (Elastic Block Store) volumes are network drives that can be attached to and detached from EC2 instances. They allow persistent data storage; even after the instance is terminated.

Question 5

EBS Volumes Use Case

Answer 5

EBS volumes allow you to persist data; even after the instance is terminated. This is helpful when you need to recreate an instance and mount the same EBS volume from before to get your data back.

Question 6

EFS (Elastic File System)

Answer 6

EFS is a managed NFS (Network File System) that can be mounted on many EC2 instances across multiple Availability Zones (AZs). It’s highly available; scalable; and expensive (about 3x the cost of GP2 EBS volumes). Use cases include content management; web serving; data sharing; and WordPress.

Question 7

Data Ingestion and Storage

Answer 7

All machine learning starts with potentially large amounts of data that need to be stored in a central repository in a scalable; secure manner. This section covers types of data; properties of data; storage strategies like data warehouses; data lakes; and data lakehouses; as well as pipelines for extracting; transforming; and loading data.

Question 8

3 Types of Data

Answer 8

Structured (organized in a defined schema; found in relational databases); Unstructured (data without a predefined schema; like raw text files; videos; audios); Semi-structured (has some structure like tags or hierarchies; but needs work to extract; like XML; JSON; log files).

Question 9

3 V’s of Data Properties

Answer 9

Volume (amount/size of data); Velocity (speed at which data is generated/processed); Variety (different types/sources of data).

Question 10

Data Warehouse

Answer 10

A centralized repository optimized for analysis; where data from different sources is stored in a structured format. Designed for complex queries and analytics. Data is cleaned; transformed and loaded using an ETL process. Typically uses a star or snowflake schema.

Question 11

Data Lake

Answer 11

Stores a vast amount of raw data in its native format (structured; semi-structured; unstructured). Data is loaded as-is without predefined schema. Supports batch; real-time and stream processing for data transformation and exploration.

Question 12

Data Lakehouse

Answer 12

A hybrid architecture combining features of data lakes and data warehouses. Supports structured and unstructured data; schema-on-write and schema-on-read; detailed analytics and machine learning. Built on cloud/distributed architectures.

Question 13

Data Mesh

Answer 13

An organizational paradigm for decentralized data management. Individual teams own their data and offer it as data products to others in the organization. Promotes domain ownership; federated governance; and self-service data infrastructure.

Question 14

ETL (Extract; Transform; Load)

Answer 14

A process in data integration that extracts data from sources; transforms it into a desired format; and loads it into a target data repository. Used for data warehouses; Contrast with ELT for data lakes.

Question 15

ELT (Extract; Load; Transform)

Answer 15

A process that extracts data from sources; loads it into a data repository in its raw format; and transforms it later as needed. Used for data lakes.

Question 16

Structured Data Use Case

Answer 16

Database tables; CSV files (if consistent columns); Excel spreadsheets (organized rows and columns).

Question 17

Unstructured Data Use Case

Answer 17

Raw text files (books; websites; social media); videos; audio files; emails; word docs (need to extract meaning first).

Question 18

Semi-structured Data Use Case

Answer 18

XML and JSON files (can have varying schemas); structured email headers; log files (structure not always consistent).

Question 19

Data Lake Use Case

Answer 19

Advanced analytics; machine learning; data discovery systems.

Question 20

Data Warehouse Use Case

Answer 20

Business intelligence; analytics when you know the schema upfront and require fast; complex queries from structured data sources.

Question 21

S3 Performance Mnemonic

Answer 21

Use MultiPart Upload for big files greater than 100MB; Get Byte Range for downloads; Use S3 Transfer Acceleration

Question 22

S3 Object Encryption

Answer 22

Server-side encryption: SSE-S3 (AWS-managed keys); SSE-KMS (user-managed KMS keys); SSE-C (customer-provided keys). Client-side encryption where you manage everything.

Question 23

Kinesis Data Streams

Answer 23

Used to collect and process large streams of data records in real time. Producers send data to Kinesis Data Streams which is then consumed by consumers. Data is stored for 1-365 days and can be replayed.

Question 24

Kinesis Data Streams Architecture Metaphor

Answer 24

Imagine a river with many streams flowing into it (the producers) and then many people trying to collect water from the river (the consumers). The river stores water for 1-365 days.

Question 25

Kinesis Data Streams Shards

Answer 25

A Kinesis Data Stream is made up of shards - each shard gets 1MB/s write and 2MB/s read capacity. You provision the number of shards required for your stream.

Question 26

Kinesis Data Streams Producers

Answer 26

SDK; Kinesis Producer Library (KPL) with record aggregation/batching; Kinesis Agent for sending logs; 3rd party libs like Spark

Question 27

Kinesis Data Streams Consumers

Answer 27

SDK; Kinesis Client Library (KCL) with shard discovery; Kinesis Connector Library; Kinesis Data Firehose; AWS Lambda; Spark

Question 28

Kinesis Enhanced Fan-Out Consumers

Answer 28

A feature that allows multiple consumers to get 2MB/s per shard each instead of sharing the 2MB/s capacity. Provides better scaling and reduced 70ms latency.

Question 29

Kinesis Scaling

Answer 29

You can reshard (split or merge shards) to increase/decrease stream capacity. This takes planning as not instantaneous.

Question 30

Kinesis Security

Answer 30

IAM for producer/consumer permissions; HTTPS for encryption in-flight; KMS for encryption at rest; Manual client-side encryption; VPC endpoints

Question 31

Kinesis Resharding Gotcha

Answer 31

If consumer hasn’t read all parent shard data before resharding; it may read child shard data first; leading to out-of-order records for a partition key. Ensure reading parent till end first.

Question 32

Kinesis Data Firehose

Answer 32

Near real-time service to capture; transform and load streaming data into destinations like S3; Redshift; OpenSearch and Splunk. Fully managed; auto-scaling; serverless data transformation with Lambda. No data storage; batches data before delivery.

Question 33

Kinesis Analytics

Answer 33

For running SQL queries to analyze; filter and transform streaming data. Can integrate with S3; Lambda and other services.

Question 34

MSK (Amazon Managed Streaming for Apache Kafka)

Answer 34

A managed Apache Kafka service on AWS. Provides more configuration options than Kinesis like larger message sizes up to 10MB (vs 1MB on Kinesis). Can manually provision brokers/topics as needed. More complex but also more flexible than Kinesis.

Question 35

MSK Use Case vs Kinesis

Answer 35

Choose MSK over Kinesis if you need larger than 1MB message sizes; need more custom configurations; or need tighter control over scaling brokers and topics.