batch_applications_flashcards

Question 1

What is Apache YARN?

Answer 1

Apache YARN (Yet Another Resource Negotiator) is Hadoop’s cluster resource management system, introduced to improve MapReduce by managing and allocating cluster resources for distributed computing.

Question 2

What is the role of the Resource Manager (RM) in YARN?

Answer 2

The RM manages resource allocation across all applications in the cluster, ensures fault tolerance, and uses scheduling policies to manage concurrency.

Question 3

What is the Node Manager (NM) in YARN?

Answer 3

The NM is a per-node agent responsible for monitoring resource usage of containers and reporting to the RM. It launches and manages containers on each node.

Question 4

What is a container in YARN?

Answer 4

A container is an abstraction used to run application-specific processes with allocated CPU and memory resources.

Question 5

What is the Application Master (AMP) in YARN?

Answer 5

The AMP controls the execution of a specific application, manages its lifecycle, and communicates with the RM to request resources and launch tasks.

Question 6

What are the benefits of using YARN for big data applications?

Answer 6

YARN provides scalability, flexibility for running multiple jobs, improved resource utilization, and fault tolerance.

Question 7

What is data locality in YARN?

Answer 7

Data locality means moving code to the data instead of moving data to the code, optimizing processing speed by reducing data transfer across nodes.

Question 8

What are the three main types of schedulers in YARN?

Answer 8

The FIFO Scheduler, Capacity Scheduler, and Fair Scheduler.

Question 9

How does the FIFO Scheduler in YARN operate?

Answer 9

It processes jobs in the order they arrive, which is simple but not ideal for shared clusters due to lack of resource balancing.

Question 10

What is the Capacity Scheduler in YARN?

Answer 10

A scheduler that allocates a reserved amount of resources to each job to ensure predictable job performance.

Question 11

What is the Fair Scheduler in YARN?

Answer 11

A scheduler that dynamically balances resources across all running jobs to ensure fairness.

Question 12

How does YARN handle job submission and execution?

Answer 12

A client submits the job, the RM allocates a container for the AMP, which then requests additional containers for task execution.

Question 13

What is the fault tolerance mechanism in YARN?

Answer 13

The AMP sets failed tasks to idle and reschedules them. If the AMP fails, the RM can use job history to recover the state or restart the job if necessary.

Question 14

What is the main drawback of Hadoop’s MapReduce?

Answer 14

It is slow due to disk I/O, writes output to HDFS after each job, and is not suitable for interactive processing or iterative algorithms.

Question 15

What are the advantages of Apache Spark over Hadoop MapReduce?

Answer 15

Spark offers better RAM usage, faster data processing, support for iterative algorithms, and a unified framework for batch and real-time processing.

Question 16

What is an RDD in Apache Spark?

Answer 16

A Resilient Distributed Dataset (RDD) is an immutable distributed collection of objects partitioned across cluster nodes, supporting operations like transformations and actions.

Question 17

What are transformations and actions in Spark?

Answer 17

Transformations create a new RDD from an existing one, while actions trigger computation and return a result or save output.

Question 18

What is lazy evaluation in Spark?

Answer 18

Spark delays the computation of transformations until an action is called, optimizing execution by applying transformations only when necessary.

Question 19

What is the Directed Acyclic Graph (DAG) in Spark?

Answer 19

A DAG represents the sequence of computations performed on RDDs, with nodes as RDDs and edges as operations. It is transformed into a physical execution plan for task scheduling.

Question 20

How does Spark’s architecture differ from MapReduce?

Answer 20

Spark uses a driver program and executors that run multiple tasks, leveraging memory for intermediate data storage, whereas MapReduce writes intermediate results to disk.

Question 21

What is YARN’s role in running Spark jobs?

Answer 21

YARN acts as the cluster manager that allocates resources for Spark jobs and monitors their execution, with Spark’s driver handling task assignments to executors.

Question 22

What is data caching in Spark?

Answer 22

Caching stores RDDs in memory, improving performance by avoiding recomputation for repeated access.