Week 1&2

Question 1

What are 2 things that can be done at large scale that cannot be done at small scale

Answer 1

extract new insights

create new forms of values

Question 2

four V’s in big data

Answer 2

Volume

Velocity ( how fast is data coming in and how fast do you have to analyze and use it)

Variety (# of sources)

Veracity(can you trust data/source/process .. data cleaning .. user entry errors)

Question 3

Why was mapreduce made

Answer 3

to provide an abstraction that allows engineers to preform simple computations while hiding the details of parralellizations, data distribution, load balancing and fault tolerance

Question 4

what does the mapper do

Answer 4

maps input key/val pairs to another set of intermediary key/vals (may map to zero or many output pairs)

Question 5

what does the reducer do

Answer 5

reduces a set of intermediary vals which share a key to a smaller set of values

Question 6

what are the 3 phases of the reducer

Answer 6

shuffle, sort, reduce

Question 7

what are somne challenges that mapreduce solves/still has

Answer 7

dividing the work into equal size pieces, limited by slowest node, combing results when done

Question 8

what does the programmer need to specify in map and reduce

Answer 8

the type of input and output key vals

map function

reduce function

Question 9

how will this be reduced to?

Answer 9