Data Engineering Fundamentals - Data Ware house vs Lakes

Question 1

Centralised repo, where optimised for analytics and where data is from different sources, stored in a structured format.

a) Data warehouse
b) Data Lake

Answer 1

Data warehouse

Question 2

Designed for complex queries,
Data is cleaned, transformed and loaded.

Optimised for read heavy operations.

a) Data warehouse
b) Data Lake

Answer 2

a) Data Warehouse

Question 3

What database offering would be commonly used for a Data Warehouse in AWS?

a) Redshift
b) NoSQL

Answer 3

a) Amazon Redshift

Question 4

a storage repo that holds vast amounts of raw data in its native format, including structured, semi structured and unsutructured data.

a) Data warehouse
b) Data Lake

Answer 4

b) Data Lake

Question 5

can store large amounts of volumes of raw data without predfined schemas.

a) Data warehouse
b) Data Lake

Answer 5

b) data lake

Question 6

Often stores data as is, no predfeined format, can be queried for data tranformation exploration purposes.

a) Data warehouse
b) Data Lake

Answer 6

b) data lake

Question 7

What AWS service would you typcially use for Data Lake storage?

a) S3
b) RDS
c) Redshift

Answer 7

a) Typically use AWS S3 when using as a Data lake.

You would then use Glue -> to extract, Athena to use that catalog to figure out the data and how to query it.

Question 8

Schema on Write (predefined schema before writing data)

ETL

a) Data warehouse
b) Data Lake

Answer 8

a) Data Warehouse

Question 9

schema on read (schema defined at the time of reading data)

ELT

a) Data warehouse
b) Data Lake

Answer 9

b) Data Lake

Question 10

You would use __________ when data integration from different sources is important, BI and analytics are the primary use cases. You have structured and data sources and require fast and complex queries.

a) Data warehouse
b) Data Lake

Answer 10

a) Data warehouse

Question 11

You would use ______________ when you have a mixed of structured, semi and unstrucuted data.

When you need a scalable and cost-effective solution to store massive amounts of data.

Future needs for data are uncertain and you want flexibility in storage and processing.

a) Data warehouse
b) Data Lake

Answer 11

b) data lake

Question 12

_____________ is hybrid data architecture that combines the best of data lakes and data warehouse.

Supports both structure and unstructured data, allow for schema on write and schema on read, provides capabilities for both detailed analytics and ml.
Typically built on top of cloud or distributed architecture.

a) Data Lake
b) Dataware house lake
c) Data lakehouse

Answer 12

c) data lakehouse

Question 13

____________ would the recommended aws service for the Data Lakehouse set-up.

COnsist of S3 and Redshift Spectrum

a) AWS Lakehouse house
b) AWS Lake Formation

Answer 13

Lake Formation

Hybird between Data warehouse and Data lake.

Question 14

__________ where indv teams own “data products” within a given domain.

These data products serve various “use cases” around the organisation.

a) Data Lake
b) Data Mesh
c) Data Domain

Answer 14

b) Data Mesh

You would hit the domains instead of the raw data.
Each domain would then have federated govt with central standards.

Question 15

________ pipelines, stands for extract, transform and load.

Process used to move data fro source systems into a data warehouse.

a) ELT
b) ETL

Answer 15

ETL

Question 16

E stand for ______________ in ETL.

retreive raw data from source systems. ENsure data integrity during the ______ phase.
Can be done in real-time or in batches, depending on requirements.

a) enormous
b) extract
c) enhance

Answer 16

b) extract

Question 17

T in ETL stands for __________

Where you convert data into a format suitable for the target data warehouse.

Can involve various operations such as:
- Data cleansing, Data enrichment, Format changes, Aggregations, Encoding or decoding data, handling missing values.

Answer 17

Transform

Question 18

L in ETL stands for __________

Move the transformed data into the target data warehouse or another data repository.

Can be done in batches or in a streaming manner.

Ensure that data maintains its integrity during the loading phase.

Answer 18

Load

Question 19

What service would you use to automate ETL pipelines.

a) AWS RDS
b) AWS Glue

Answer 19

AWS Glue

You would then have some sort of orchestration service:

Event bridge, Amazon managed workflow for Apache Airflow, Lambda, GLue workflows.