Week 6 - Data Management in MapReduce Systems

Question 1

What is Map Reduce (3 items)

Answer 1

1) distributed computing
2) Map
3) Reduce

(key/value pairs).

Question 2

Map Reduce - Map

Answer 2

Map takes a set of data and converts it into another set of data

Question 3

Map Reduce - Reduce

Answer 3

Reduce takes the output from a map as an input and combines those data tuples into a smaller set of tuples

Question 4

What is Hadoop

Answer 4

1) open-source framework
2) process big data
3) On clusters of computer
4) simple programming models.

Question 5

What are the two main phases of Map Reduce

Answer 5

1) Map Phase

2) Reduce Phase

Question 6

What does the master note/ master process do?

Answer 6

1) Keeps tract settle the cluster
2) Address local machines
3) Decides which machines run what for each phase

Question 7

Where does each worker store it’s results?

Answer 7

On it’s local disk

Question 8

What is the hidden phase between the Map phase and Reduce phase?

Answer 8

Data shuffle phase or data transfer phase (lot of data transfer)

Question 9

Can the Reduce phase happen while the Map phase is still going on?

Answer 9

No, the Mapper has to finish first

Question 10

Hadoop is what

Answer 10

An open source implication of the map reduce paradigm

Question 11

Hadoop file system is called

Answer 11

HDFS

Hadoop Distributed File System

Question 12

What does HDFS provide

Answer 12

1) Single name space for entire cluster

2) Replicated data 3x fault tolerance

Question 13

Two items for MapReduce Framework

Answer 13

1) Executes user jobs as “map” and “reduce” functions

2) Manages work and distributes & fault-tolerance

Question 14

HDFS is make up of these two elemets

Answer 14

1) NameNode

2) DataNode

Question 15

HDSF what size of blocks

Answer 15

128MB

Question 16

HDSF are block replicated

Answer 16

Yes, over several DataNodes

Question 17

HDSF Optimized for large or small files

Answer 17

Large sequential reads

Question 18

HDSF are file read and write?

Answer 18

No append only

Question 19

HDSF DataNodes are what

Answer 19

Each Node is a machine

Question 20

HDSF NameNode

Answer 20

Stores the meta data about machines and locations

Question 21

Centralized nameNode contains

Answer 21

1) Filename
2) number of Replicas
3) Block-ids
4) More

Question 22

What do you need to write a program in MapReduce

Answer 22

1) Data type
2) Map Function
3) Reduce Function

Question 23

MapReduce - Data type

Answer 23

Key-value Records

Question 24

MapReduce - Map Function

Answer 24

(Key,1 VALUE1) ->

list(Key2, VALUE2)

Question 25

MapReduce - Reduce Function

Answer 25

(Key2, list(VALUE2) ->

list(Key3, VALUE3)

Question 26

MapReduce - Can Reduce run in parallel and independantly?

Answer 26

Yes

But no sharing data, Map Functions work the same way

Question 27

Should Mappers be placed on the same node or Rack as their input block?

Answer 27

Yes, it minimized network use

Question 28

Where do the mappers save their output

Answer 28

Local disk (mainly for fault tolerance and recovery)

Question 29

Advantage of storing Mapper output on local disks.

Answer 29

1) allows having more reducers than nodes

2) Allows recovery if a reducer crashes

Question 30

In Hadoop if a task crashes (map)

Answer 30

1) Retry on another node
2) OK for a map because it had no dependencies
3) OK for a reduce because map outputs are on disk

Question 31

What if a machine(node) crashes

Answer 31

1) Re-launch its current tasks on other nodes (machines)

2) Re-launch any maps the node ran previously (necessary because their output files were lost at the same time)

Question 32

Do you always need the reduce phase?

Answer 32

No, sometimes the task is simple, and it takes advantage of parallelism (reading files in parallel)

Question 33

Can you use projection in Hadoop?

Answer 33

Yes, you can only select a few columns if that is what you want (reduce phase might now be necessary)

Question 34

two of the best operators in MapReduce

Answer 34

1) Sorting

2) Group by

Question 35

How to calculate statistics in Hadoop

Answer 35

1) reduce phase
2) Group by
3) count

Question 36

What is an equi-join

Answer 36

An equijoin returns only the rows that have equivalent values for the specified columns.