Data Engineering Flashcards
Amazon Kinesis Data Streams
Collect and store streaming data in real-time
1. Retention up to 365 days (can’t be deleted until it expires)
2. Data ordering guarantee
Kinesis Data Streams - Capacity Modes
Provisioned mode
- Choose number of shards
- Each shard gets 1000 records per second
- Scale manually
- Pay per shard provisioned per hour
On demand mode:
- Default capacity provisiones (4000 records per second)
- Scale based on observed throughput peaks
- Pay per stream per hour & data in/out
Amazon Data Firehose - AWS Destinations
- Amazon S3: Supports compression
- Amazon Redshift (copy through S3)
- Amazon OpenSearch
Amazon Data Firehose
- Receive records (up to 1MB) from producers
- Can make data transformation with Lambda functions
- Batch writes to destinations based on buffer time and size, Near Real Time
- All or failed data can be backup in S3
Firehose Buffer Sizing
Firehose accumulates records in a buffer and is flushed based on time and size rules
Amazon Managed Service for Apache Flink
Framework for processing data streams. Can’t read from Amazon Data Firehose.
Amazon Managed Streaming For Apache Kafka
Amazon MSK creates & manages Kafka brokers nodes.
* Deployed in your VPC, multi-AZ
* Data stores on EBS volumes
Kinesis Data Streams vs Amazon MSK
Kinesis Data Streams:
* 1 MB message size limit
* 12 months maximum retention
* Shard splitting and merging
Amazon MSK:
* Configure for bigger messages
* No retention limit
* Can only add partitions to a topic
AWS Batch
Run batch jobs as Docker images
AWS Batch - Multi node mode
Leverages multiple EC2 / ECS instances. One main node and multiple childs. Doesn’t work with Spot Instances.
Amazon Elastic MapReduce
EMR creates Hadoop clusters in a single AZ to analyze and process vast amount of data.
* Can access data in DynamoDB and S3
EMR File System
EMRFS stores persistent data in Amazon S3 while providing data encryption
EMR node types
- Master node: Manage the cluster
- Core node: Run tasks and store data
- Task node: Justo to run tasks, usually uses spot
EMR instance configuration
- Uniform instance groups: select a single instance type and purchasing option for each node. Has auto scaling.
- Instance fleet: Select target capoacity, mix instance types and purchasing options. No auto scaling.
AWS Glue
Managed extract, transform and load service
AWS Glue Data Catalog
Data crawler writes metadata of databases and supportes storages to the data catalog
Redshift
Online analytical processing
* Columnar storage of data: easier to aggregate
* Massively Parallel Query Execution
* Not all clusters support multi AZ
Redshift nodes
Leader node: query planning and results aggregation
Compute node: performing queries and send results to leader
Redshift Enhanced VPC Routing
Copy and unload goes through VPC
Redshift snapshots
Point in time backups stored in S3
* Incremental
* Restore into a new cluster
* Automated or manual
* Copy automatically to another region
Redshift spectrum
Query data that is in S3 without loading it
* Must have a Redshift cluster to start the query
Redshift resource link
Data catalog object which is linked to a database, allows integrated services to run queries on the database. These services will not be able to access directly across cross accounts.
Redshift Workload Management
Flexibly manage queries priorities within workloads
* Multiple query queues
* Route queries to the appropriate queue
Redshift EVEN distribution
The leader node distributes the rows across the slices in a round-robin fashion, regardless of the values. Is appropriate when a table doesn’t participate in joins
Redshift KEY distribution
The leader node places matching values in one column on the same node slice. Matching values from the common columns are physically stored together.
Redshift ALL distribution
A copy of the entire table is distributed to every node. Ensures that every row is collocated for every join. Appropriate only for relatively slow moving tables
DocumentDB
Fully managed MongoDB used to store, query and index JSON data
* Replication across 3 AZ
* Automatically grows in increments of 10GB
DocumentDB Pricing
- On-demand instances per second
- Database I/O
- Database storage per GB
- Backup storage per GB
Amazon Timestream
Fully managed time series database
* Faster and cost effective comparing with relational database
* SQL compatibility
* Recent stored in memory, historical cost optimized storage
* Built-in time series analytics functions
* You ccan configure schedule queries
Amazon Athena
Serverless query service to analyze data stored in S3
* Commonly used with Quicksight
Athena Performance Improvement
- Columnar data for cost savings. Parquet or ORC is recommended
- Compress data for small retrievals
- Partition datasets in S3 (Partioning pruning)
- Use larger files to minimize overhead
- Predicate pushdown to filter data at source
Amazon QuickSight
Serverless business intelligence service to create interactive dashboards
Amazon Forecast
Fully managed service that uses statistical and machine learning algorithms to deliver highly accurate time-series forecasts. To import your data you must store it in an S3 bucket.
