Batch - Yarn and MapReduce

Question 1

YARN (Yet Another Resource Negotiator)

Answer 1

Resource management system designed to handle distributed computing

Question 2

YARN APIs

Answer 2

Request and work with cluster resources (not made by user code! by framework!)

Question 3

Two key components of YARN

Answer 3

1. Scheduler (allocate resources)
2. Applications Manager (accept job submissions, containers for AMP)

Question 4

What makes Yarn scheduler a “pure scheduler”?

Answer 4

no monitoring status for application
no restarting mechanism

Question 5

Applications Manager job

Answer 5

accept job submissions, negotiate container for executing AMP, provide service for restarting if AMP fails

Question 6

YARN provides choice of schedulers, here are 3 types:

Answer 6

FIFO (no configuration necessary, bad for clusters)
Capacity Scheduler (fixed amount of capacity to each job)
Fair Scheduler (balances available resources between running jobs)

Question 7

YARN Resource Manager (RM)

Answer 7

Global daemon (one per cluster) manages resource allocation in the cluster

Question 8

YARN Node Manager (NM)

Answer 8

A daemon running on each node in cluster monitoring resource usage and reporting back to Resource Manager.

Question 9

Upon request from client, RM finds a NM to launch ______ in a ___________.

Answer 9

Application Master Process; container

Question 10

Job of AMP

Answer 10

Execute computation!

Question 11

Data Locality (YARN)

Answer 11

YARN optimizes job execution by ensuring tasks are run as close to the data as possible. (move computation, not data)

Question 12

4 Levels of Data Locality

Answer 12

1. Node-level
2. Rack-level
3. Data Center-level
4. Inter-data center

Question 13

MapReduce

Answer 13

A programming model that allows developers to write programs that can process large amounts of data in parallel across a cluster

Question 14

Map Phase

Answer 14

Large dataset is partitioned into smaller chunks (input splits) and processed in parallel by map tasks that turn data into key-value pairs

Question 15

Sort & Shuffle

Answer 15

Data is sorted by key and shuffled (moved) in groups to reducers.

Question 16

Reduce phase

Answer 16

Intermediate data is aggregated by reduce function

Question 17

Combiner

Answer 17

In some cases, a mini-reduce function used as an optimizer between Map and Reduce phases so less data is transferred.

Question 18

Map or Reduce task failure

Answer 18

rescheduled on another node

Question 19

Map or Reduce node failure

Answer 19

all tasks on node rescheduled on another node
worst case: restart entire job