Data Engineering Fundamentals

Question 1

What is Avro?

Answer 1

A Binary storage format that keeps information about the schema.

Question 2

What is Parquet?

Answer 2

Columnar storage optimized for analytics.

Question 3

What does random sampling do?

Answer 3

It gives everything an equal chance at being selected.

Question 4

What is stratified sampling?

Answer 4

It splits the population, but ensures representation of each subgroup.

Question 5

What is systemic sampling?

Answer 5

When you are going to select every N item.

Question 6

What is data skew?

Answer 6

Unequal distribution between partitions.

Question 7

What can be done to address data skew?

Answer 7

Adaptive partitionig

Salting

Repartitioning

Question 8

What does the YEAR() function in SQL do?

Answer 8

It selects only the year from a date field.

Question 9

What does a pivot table do?

Answer 9

It makes row level data into columnar data.

Question 10

What is the default SQL join?

Answer 10

An inner join?

Question 11

How does inner join work?

Answer 11

It select all the rows from table A that have a matching identifier in table B.

Question 12

How does a left outer join work?

Answer 12

It selects everything in Table A regardless of whether there is a match in Table B. Only records with a match in Table B are returned.

Question 13

How does a right outer join work?

Answer 13

It selects everything in Table B regardless of whether there is a match in Table A. Only records with a match in Table A are returned. Opposite of Left Join.

Question 14

How does a full outer join work?

Answer 14

Data from Table A and Table B is returned, but only matching records will have values.

Question 15

What does Regex do?

Answer 15

It pattern matches.

Question 16

What is the RegEx operator for case sensitivity?

Answer 16

~*

Question 17

What is the RegEx expression operator?

Answer 17

~

Question 18

What is the RegEx expression to not match?

Answer 18

!~*

Question 19

In GIT, how do I get files from the repository to my local workspace?

Answer 19

GIT Pull

Question 20

How would you initialize a new Git repository?

Answer 20

GIT Init

Question 21

What does GIT Config do?

Answer 21

Sets configuration values for user info and aliases.

Question 22

How do you clone or download a repository from an existing URL?

Answer 22

git clone

Question 23

What does git status do?

Answer 23

It checks the status of your changes in your working directory. This is local.

Question 24

How do you view commit logs in git?

Answer 24

Git log

Question 25

What does git branch do?

Answer 25

It shows all branches

Question 26

How would you create a new branch

Answer 26

git branch newBranchName

Question 27

How do you switch branches?

Answer 27

git checkout branchname

Question 28

How do you create a new branch and switch to it?

Answer 28

git checkout -b

Question 29

How do you delete a branch?

Answer 29

git branch -d

Question 30

How do you push your changes to the remote repository?

Answer 30

git push

Question 31

What does git pull do?

Answer 31

Pulls changes from a remote repository branch into the current local branch

Question 32

What is a transition action in s3?

Answer 32

It is used to move objects from one storage glass to another.

Question 33

What are expiration actions in S3?

Answer 33

They are used to configure object expiration / delete after N period of time.

Question 34

Can lifecycle rules be created based on tags or prefixes?

Answer 34

Yes, on Both

Question 35

What is the level hierarchy for S3?

Answer 35

Standard

Standard IA

Intelligent Tiering

One Zone IA

Glacier Instant Retrieval

Glacier Flexible Retrieval

Glacier Deep Archive

Question 36

What does S3 analytics do?

Answer 36

Helps you decide when to transition objects to the right storage class.

Question 37

What are the targets for S3 event notifications?

Answer 37

Lambda, SNS, and SQS