Data Ingestion and Storage Flashcards
1
Q
What are the three types of data?
A
Structured
Unstructured
Semi-Structured
2
Q
What is semi-structured data?
A
XML, JSON, Log Files with varied formats, etc..
3
Q
What is unstructured data?
A
Video files, emails, text files with no fixed format.
4
Q
What are the three properties of data?
A
Volume
Velocity
Variety
5
Q
What is meant by the term data variety?
A
Different types of data formats and sources.
6
Q
Is a data warehouse good for storing structured information?
A
Yes
7
Q
Is a data warehouse good for storing files and images?
A
No
8
Q
Is a data warehouse good for OLAP?
A
Yes
9
Q
If you have structured, semi-structured, and unstructured data, what is the best way to store it?
A
A data lake
10
Q
Do you use ETL or ELT for a Data Warehouse?
A
ETL
11
Q
Why is ELT used for a Data Lake?
A
You need to read the file to know the format.
12
Q
What is more expensive, a data lake or data warehouse?
A
Usually, a data warehouse
13
Q
What is an example of an AWS Data Lakehouse?
A
AWS S3 with Redshift Spectrum
14
Q
What is a Data Mesh?
A
The governance and organization of data.
15
Q
What is Avro?
A
A binary format for that that stores the data and its schema.
16
Q
What is Parquet?
A
A columnar storage format optimized for analytics.
17
Q
What is the S3 Key?
A
The full path of the file. Everything after the bucket name all the way until the file name.
18
Q
What are the two parts to an S3 Key?
A
The Prefix and Object name
19
Q
What is a Prefix?
A
The path of the file, but not the bucket or file name.
20
Q
Is versioning required for S3 replication?
A
Yes
21
Q
Will S3 replication work on existing objects?
A
No. Unless you decide to do this from a batch operation.
22
Q
What is Glacier Instant Retrieval?
A
low cost storage with millisecond instant retrieval.
23
Q
What is the minimum object duration for for Glacier Instant Retrieval?
A
90 days
24
Q
What is Glacier Flexible Retrieval?
A
Used to be Glacier. Now has three retrieval tiers.
Expedited (1 - 5 min)
Standard (3 - 5 hours)
Bulk (5 - 12 hours)
25
What are the Glacier Deep Archive tiers?
Standard (12 hours)
Bulk (48 hours)
26
When S3 executes SQS, SNS, or Lambda, what kind of access control policy is needed and where is it configured?
A resource based policy on the target service. e.g., SQS.
27
What file sizes are recommended for S3 multi-part uploads?
100MB or greater. It is a requirement for 5GB or larger files.
28
How does S3 Transfer Acceleration work?
It sends data to the nearest edge location.
29
What is an S3 Byte Range Fetch
Allows you to fetch parts of a file. Good for only downloading partial data like headers.
30
What is the difference between AWS SSE-KMS and SSE-C?
One uses a key in KMS and the other is a customer provided key. Could be from your own HSM.
31
Is SSE-S3 enabled by default on S3 buckets?
Yes
32
Are there rate based limitations that can cause throttling in KMS?
Yes
33
How can you force encryption in transit?
Resource (bucket) policy
34
What are S3 Access Points?
They point to specific prefixes in your bucket.
35
How are S3 Access Point permissions managed?
Through Access Point Policies. These are resource based policies.
36
Can S3 access points be private?
Yes, using VPC origins.
37
Do VPC endpoints have resource based policies?
Yes.
38
What is an S3 Object Lambda?
It is used to change the object before it is retrieved by the called application.
39
What protocol does AWS EFS use?
NFS
40
Is EFS more expensive than EBS?
Yes, around 3x.
41
Does EFS scale automatically?
Yes
42
What are the performance modes for EFS?
General Purpose
Max I/O for big data and media processing
43
What are the throughput modes for EFS?
Bursting
Provisioned
Elastic
44
Does EFS support lifecycle policies?
Yes.
45
What is Amazon FSx
It allows you to launch 3rd party high-performance file systems on AWS.
46
What are the four file systems that FSx supports?
Lustre, Netapp ONTAP, Windows File Server, OpenZFS
47
Does FSx for Windows File Server support SMB?
Yes
48
Does FSx for Windows support Active Directory?
Yes
49
Can FSx for Windows be launched on Linux?
Yes
50
Can FSX be accessed by On-Prem?
Yes, using direct connect or a VPN
51
What is a key use case for FSx for Lustre?
High-Performance computing.
52
What is an FSx scratch file system?
Temporary storage. Data is not replicated.
53
What FSx file system type supports point-in-time instant cloning?
FSx for Netapp ONTAP and OpenZFS
54
What are the two parts that make up a record in Kinesis Data Streams when you send the record to the stream?
A partition key and data blob (1MB)
55
How large is a Data Blob in Kinesis Data Streams?
1MB
56
What is the throughput for Kinesis Data Streams from the producer to the stream?
1MB per second or 1000 messages per second.
57
What are the parts that make up a record in Kinesis Data Streams when you send a record to the consumer?
Partition Key
Sequence Number
Data Blob
58
What is the throughput for Kinesis Data Streams from the Stream to the Consumer?
2MB per second, per shard SHARED
OR
2MB per second per shard per consumer using enhanced fan out.
59
What is the max retention in Kinesis?
365 Days
60
What are the two capacity modes in Kinesis?
Provisioned
On Demand - based on peak over the last 30 days.
61
What are three flavors of the Kinesis SDK?
Kinesis SDK, Kinesis Producer Library, and Kinesis Agent
62
Does The PutRecords API call have a batch option?
Yes. This allows you to increase throughput.
63
What error will you get when you exceed the allowed throughput for Putrecords?
ProvisionedThroughPutExceeded
64
What is a good use case for the Kinesis Producer SDK?
Low throughput , higher latency
65
How can you address a hot shard that causes a ProvisionedThroughPutExceeded error?
Exponential backoff
Increase Shards
66
What languages does the KPL support?
C++ and Java
67
Does the KPL support asynchronous operations?
Yes
68
Can the KPL compress records?
No
69
If a producer uses the KPL, what are the requirements for the consumer?
The records MUST be de-coded with the KCL by the consumer
70
What does batching in the KPL do?
It can aggregate messages into one record that is less than 1MB.
71
How can you control batch sending delays in Kinesis?
The recordMaxBufferedTime configuration option.
72
When is batching in the KPL not a good idea?
When the application cannot tolerate a latency.
73
Can Apache Spark, Log4J appenders, Flume, or Kafka be used as a consumer for Kinesis Data Streams?
Yes
74
What is the throughput for the GetRecords API call?
10MB of Data or up to 10K records
75
What is the GetRecords API call limit?
Maximum of 5 GetRecords API calls per shard per second. 200ms latency
76
Does the KCL support checkpointing?
Yes. It leverages dynamoDB for this.
77
What does the ExpiredIteratorException indicate in relation to Kinesis Client Library?
That the WCU needs to be increased to support checkpointing.
78
Can Lambda read from a Kinesis Data Stream?
Yes
79
What is Kinesis Fan Out for Consumers?
Each consumer gets 2mbps per shard per consumer.
80
What is the latency with Kinesis Enhanced Fanout for Consumers?
70ms
81
What is a good use case for standard consumers?
An application that can tolerate 200ms latency.
82
Can you split shards in kinesis?
yes using the split operation
83
When will a split shard be removed?
When the data is expired.
84
What is a common root cause of reading data out of order in Kinesis Data Streams?
Data was not fully read after a resharding event.
85
Can resharding be done in parallel?
No. It takes a few seconds for resharding operations.
86
What is a common root cause for duplicates coming from a Kinesis Producer?
Network Timeouts. This can be prevented by embedding a unique record ID in the data to deduplicate on the consumer side.
87
What is a common root cause for duplicates coming from a Kinesis Consumer?
When a worker terminates
Worker instances are added or removed
Shards are merged or Split
The application is deployed
88
How do you handle data transformation with Amazon Firehose?
Using a lambda
89
What are some common destinations for AWS Data Firehose?
S3, Redshift, Amazon Opensearch
90
Can data be replicated to an S3 bucket in Amazon Firehose?
Yes. Both failed and all records can be stored in a backup bucket.
91
Is Amazon Data Firehose real-time?
No. It is near-real-time
92
What data formats are supported by Amazon Data Firehose?
CSV, JSON, Parquet, Avro, Raw Text, Binary data.
93
Can Amazon Data Firehose perform data compression or conversions?
Yes...
Parquet to ORC
Compression with gzip or snappy
94
What is the Flink framework used for?
Processing data streams
95
Is Managed Service for Apache Flink serverless?
Yes
96
What are some common use cases for the Managed Service for Apache Flink
Streaming ETL
97
What is RANDOM_CUT_FOREST used for in Kinesis Analytics?
Anomaly detection.
98
What is Amazon Managed Streaming for Apache Kafka?
It is a fully managed Apache Kafka cluster. It is a replacement for Kinesis.
99
What is the maximum size for a Amazon Managed Streaming for Kafka message?
10MB
100
Kafka producers send code to what?
Brokers. They act similar to shards in Kinesis.
101
Can you encrypt data in-transit with AWS Kafka?
Yes. This is enabled by default
