Map Reduce

Question 1

Which technology is Map Reduce a part of?

Answer 1

Hadoop. Hadoop consists of HDFS and Map Reduce

Question 2

What is the data format of the Input in Map Reduce?

Answer 2

(Key, Value) pairs, of arbitrary serializable types, that should fit in memory

Question 3

What strategy should be employed when cluster components fail during computation in Map Reduce?

Answer 3

To address cluster component failures, it is advisable to parallelize computation into small tasks. In the event that a task fails to deliver results, the recommended approach is to restart that specific task.

Question 4

Where does the data come from in Map Reduce?

Answer 4

The (H)DFS

Question 5

What are the 4 steps in the Map task?

Answer 5

• Read (key, value) pairs in input, from the DFS
• One Map task per pair (Will be scheduled on/near the machine where the input is).
• Computes a number of (key, value) pairs, decided by you
• Outputs to the (local!) disk in a buffer region

Question 6

What is the main operation of the Shuffle (Master controller) task?

Answer 6

Keeps track of the (key, value) pairs in the output of all Map tasks. It then does a distributed group by key operation, which outputs the key(s) and its list of values

Question 7

What 3 qualities defines the Reduce task?

Answer 7

• One reduce works on one key at a time
• Computes a combined value per key, decided by you
• The output is saved to (H)DFS files (one reduce per task)

Question 8

What are the 3 switch levels of data acquisition from HDFS to the MapReduce task, in order of fastest to slowest?

Answer 8

• Data Local (On the same machine in the rack)
• Rack Local (On the same rack, but different machines)
• Off Rack (Between racks)

Question 9

In general terms, what does the MapReduce task do?

Answer 9

It compresses several data entries of the same value, to a single self-specified new key, value (often count). Example: given the input “w1, w2, w3, w2, w3, w3, w3”, the output could be “(w1, 1), (w2, 2), (w3, 4)”.