chp1 Fundamentals of data engineering

Question 1

What questions should I be asking if I want to display information to my end user

Answer 1

Who will consume the data? , What data sources should I use? , where should I store the data? , When should the data arrive? , Why does the data need to be stored in this place? , How should the data be processed?

Question 2

What is the first thing to learn as a data engineer

Answer 2

The data lifecycle

Question 3

What is it called when data is stored in different places

Answer 3

data silos

Question 4

what is a data silo

Answer 4

When data is stored in different and independent places

Question 5

What are the four logical building blocks in a data warehouse?

Answer 5

sql interface, schema, compute, and storage

Question 6

what is a data lake

Answer 6

a centeralized repository that allows you to store all your stuctured and unstructured data

Question 7

Is a schema madatory for a data lake

Answer 7

no. Why?

Question 8

Where is a schema mandiotry ? Data lake or datawarehouse

Answer 8

data warehouse

Question 9

What are two benefits of a data lake?

Answer 9

Scalability and cheap storage

Question 10

Data is a data warehouse is modeled for a ….?

Answer 10

business purpose

Question 11

What is one of the key requirements for building a data warehouse?

Answer 11

know the business requirements

Question 12

The frontend applications in most cases acts as what?

Answer 12

the first data upstream

Question 13

What is a data mart?

Answer 13

A data mart is an area for storing data that serves specific user groups

Question 14

Each data mart is usually under control of ….?

Answer 14

each deparment within an organization

Question 15

what are the three most common usages for the last stage in the data lifecycle?

Answer 15

Reporting and Dashboard, Ad hoc query, Machine Learning

Question 16

data should only be stored where? based on business needs ?

Answer 16

data warehouse

Question 17

Where is a schema manditory?

Answer 17

data warehouse

Question 18

A data engineer is someone that…?

Answer 18

designs and builds data pipelines

Question 19

What is a Job Orchestrator?

Answer 19

Design and build jobs dependancy and scheduler that runs data movement from upstream to downstream. Why?

Question 20

What is ETL ?

Answer 20

Extract Transform Load. Why?

Question 21

What is the difference between ETL and ELT?

Answer 21

ETL is where you make your transformations before the data is loaded in the downstream source and ELT is where you load the data into the downstream source than make the necessary transformations. Why ?

Question 22

What is extract?

Answer 22

This is the step to get the data from the upstream system. Why?

Question 23

What is transform?

Answer 23

This is the step to apply any transformations to the extracted data. Why>?

Question 24

What is load?

Answer 24

This is the step to put the transformed data to a downstream system. Why?

Question 25

All things in the data lifecycle are…?

Answer 25

ETL

Question 26

any part of the data lifecycle that happens from upstream to downstrean is …?

Answer 26

ETL … Why?

Question 27

Big data technology needs to ….?

Answer 27

be able to distribute the data in mulitple servers. Why ?

Question 28

What is the key foundation of data engineering?

Answer 28

ETL - Why ?

Question 29

Not all systems can ____ large dataset

Answer 29

Transform. Why?

Question 30

What is the common terminology for mutliple servers that are working together?

Answer 30

Cluster. Why?

Question 31

What is a cluster?

Answer 31

multiple servers working together. Why?

Question 32

In a distributed filesystem, a large file will be ….?

Answer 32

split into multiplle small parts. Why?

Question 33

What are the four steps to mapreduce?

Answer 33

Map , Shuffle, Reduce, Result. Why?

Question 34

What is the use case of MapReduce?

Answer 34

To process distributed filesystems. Why?

Question 35

MapReduce happens in ….?

Answer 35

Parallel. Why?

Question 36

MapReduce always maintains …?

Answer 36

three parallel boxes when performing map, shuffle , and reduce. Why?

Question 37

What is the definition of a data lake?

Answer 37

A centralized repository that allows you to store all your Structured and unstructured data 

Question 38

Outline or draw detail, life cycle diagram

Answer 38

Question 39

The last stage of data will come back to humans as what?

Answer 39

Information

Question 40

The application is the interface from blank to blank?

Answer 40

The application is the interface from human to the machine. Why?

Question 41

Did engineers someone who designs and builds…?

Answer 41

#DATA pipelines

Question 42

Explain what a job orchestrator is:

Answer 42

Program or software that designs and builds a job dependency and schedulwe that runs data movement from upstream downstream 

Question 43

What are the focus points of a data engineer?

Answer 43

Question 44

Why is designing and building a data mart not as simple as it seems?

Answer 44

Question 45

What does ETL stand for?

Answer 45

Extract transform load

Question 46

What is big data?

Answer 46

Any data that’s large enough that it needs to be distributed across several servers in order for you to run calculations

Question 47

Draw the diagram for ETL

Answer 47

Question 48

What is extract?

Answer 48

This is the step to get the data from the upstream system

Question 49

What is transform?

Answer 49

This is a step to apply any transformation to the extracted data 

Question 50

Why is every step in the data lifecycle ETL

Answer 50

Question 51

Draw the diagram for ELT

Answer 51

Question 52

You have two options of where to transform your data. What are they?

Answer 52

In a intermediary system or in the target system.