FinalExamHadoop

Question 1

What are the three areas of expertise that you need to make a unicorn…. datascientist

Answer 1

Hacking skills(programming, get/manipulate/explore data)
Math and Stats (foundational statistics, internals of algorithms, practical knowledge)
Domain Knowledge( understand the business problem, Understand the data)

Question 2

What is the truth about learning?

Answer 2

When you stop learning your salary stops growing.

Question 3

what are the last two v’s of big data?

Answer 3

Value (statistical, hypothetical)

Veracity (trustworthiness)

Question 4

What is a winning interview question for you to ask the employer

Answer 4

Do you have a centralized or decentralized data model?

Question 5

what is the basis of machine learning?

Answer 5

Extracting knowledge from data.

at the intersection of stats, AI and Computer Science

Question 6

what is supervised learning?

Answer 6

automate decision making processes by generalizing from known examples.

Question 7

what is unsupervised learning?

Answer 7

harder to understand and evaluate than the supervised learning example you have merely data. You have zero answers going in.

Question 8

what is overfitting?

Answer 8

when you fit a model too closely to particularities of the training set and works well for it but not for new data

Question 9

what is underfitting?

Answer 9

making your model too simple

Question 10

what are HIVE and Impala

Answer 10

SQL like programs. Impala is super fast where HIVE is a little more robust

Question 11

what is a select statement in hadoop

Answer 11

map

Question 12

what is a group by statement in hadoop

Answer 12

reduce by

Question 13

what is a where clause in hadoop

Answer 13

filter

Question 14

what are the benefits of spark?

Answer 14

ii. Benefits: optimization, lazy loading/evaluation, speed, efficiency
iii. Blazing fast because it keeps stuff in memory until an action is needed
III. fault tolerance you can go back and rerun a portion.

Question 15

what is an RDD

Answer 15

steps on a map

Question 16

what is a DAG

Answer 16

it is a process of how all your rdd’s work together to make a complete map of the transformtions you are going to make to the data.

Question 17

what are key value pairs?

Answer 17

an array with keys (which can change), know how this is different from a unique identifier (primary key in SQL); why do they matter, why do they get their own command set

Question 18

why does a simple reduce not work in hadoop?

Answer 18

k. Why does the simple reduce not work with aggregates across partitions? (reduce hits the rows and combines two at a time, then moves on and combines the next one together; reduces everything by nodes, not overall, then between nodes combines, which can lead to inaccurate results)

Question 19

what is the benefit of reduce by key?

Answer 19

helps with partition stability