Hadoop and Pig

Question 1

What specific concept is HDFS designed to support?

Answer 1

HDFS is designed to support high streaming read performance and follows the concept of “write once and read many times.” It doesn’t encourage frequent updates on files.

Question 2

How does HDFS ensure fault tolerance in its data storage?

Answer 2

by storing data in fixed blocks on distributed nodes called DataNodes, replicating them to handle potential failures.

Question 3

How does Hadoop achieve its purpose?

Answer 3

It scales from one server to thousands, making it easy to process large datasets. It’s designed to stay available even if some computers fail, ensuring reliability on less-than-perfect systems.

Question 4

What is the significance of heartbeats in HDFS, and how does the NameNode use them to manage DataNodes?

Answer 4

Heartbeats in HDFS are signals sent by DataNodes to the NameNode. The NameNode uses heartbeats to track individual DataNodes and manage the overall health of the system.

Question 5

What happens if the NameNode doesn’t receive a response from a specific DataNode in HDFS?

Answer 5

it considers that DataNode as failed or faulty, and appropriate actions are taken to maintain system integrity and fault tolerance.

Question 6

What is the role of the NameNode in the File System Namespace?

Answer 6

The NameNode manages and maintains the FileSystem Namespace, keeping track of all Namespace properties and information

Question 7

What is the default size of data blocks in HDFS, and why is it designed to be much larger than the standard file block size?

Answer 7

The default size of data blocks in HDFS is 128 MB, significantly larger than the standard file block size of 512 bytes. This larger size is chosen for fault tolerance and availability through replication.

Question 8

What potential issue arises from using small blocks in HDFS, and what does this issue result in?

Answer 8

The use of small blocks in HDFS leads to a large number of files, causing considerable interaction between the NameNode and DataNodes. This interaction creates overhead for the entire process.

Question 9

What are the two main components of an HDFS cluster, and what are their respective roles?

Answer 9

An HDFS cluster includes a single NameNode (managing metadata) and multiple DataNodes (managing storage)

Question 10

How does a file get stored in an HDFS cluster, and what role does the NameNode play in this process?

Answer 10

Files in HDFS are split into blocks, stored in DataNodes; the NameNode maps blocks to DataNodes.

Question 11

How does HDFS handle data block storage, and what triggers communication between DataNodes and the NameNode?

Answer 11

HDFS stores data blocks locally initially and communicates with the NameNode for the next DataNode when a block is full.

Question 12

What is the role of the replication factor in HDFS, and when is it determined?

Answer 12

The replication factor, set during file creation, ensures fault tolerance by creating copies on different DataNodes. If not specified, the default is three.

Question 13

How does Hadoop’s rack awareness enhance fault tolerance, and what’s a drawback associated with it?

Answer 13

by distributing replicas across racks. However, it increases I/O costs due to block transfer across racks.

Question 14

What happens when a DataNode fails health checks in HDFS, and how does the system handle block replication?

Answer 14

the NameNode removes the block from the pipeline and re-replicates it to a different DataNode.

Question 15

What model does HDFS adopt and what is its single point of failure?

Answer 15

Master/slave model where the slave is a single point of failure

Question 16

Give a solution to recovering data when a NameNode fails in HDFS

Answer 16

Use two NameNodes, one active and one on standby. The standby node takes over as the active node in the case of failure

Question 17

What is the performance trade-off in MapReduce

Answer 17

Increasing the number of computing nodes enhances computation speedup in distributed systems. However, this improvement comes with a trade-off of higher data volume for intermediate reports that need to be exchanged among the nodes.

Question 18

When not to use MapReduce

Answer 18

small dataset

need to process data in real time

processing data streams

Question 19

Give a brief description of each of the Pig Components

Answer 19

1. The Parser processes Pig Scripts, ensuring correct syntax, performing type checks, and generating a directed acyclic graph (DAG) to represent Pig Latin statements and their logical flow.
2. The Optimizer refines the logical plan (DAG) by executing optimizations like projection and pushdown, enhancing the efficiency of subsequent processing.
3. The Compiler translates the optimized logical plan into a series of MapReduce jobs, facilitating the execution of data processing tasks.
4. The Execution Engine submits and manages the execution of MapReduce jobs on Hadoop, orchestrating the processing of data to produce the desired results.

Question 20

What are some of the advantages of using Pig over MapReduce

Answer 20

High-level language

Ease of use

Significant reduction in code length

No need for compilation

Question 21

What does Pig’s infrastructure layer consist of, and how does it leverage existing large-scale parallel implementations?

Answer 21

Pig’s infrastructure layer includes a compiler producing sequences of Map-Reduce programs, leveraging existing large-scale parallel implementations like Hadoop

Question 22

What is MapReduce

Answer 22

A programming paradigm for processing big datasets in a distributed fashion

Question 23

What are steps in MapReduce

Answer 23

1. Prepare the input to Map by selecting a key
2. Execute Map in each node and generate output based on another key
3. Shuffle the output from Map to the Reduce nodes
4. Execute Reduce
5. Produce the final output

Question 24

What is the salient (key) property of Pig

Answer 24

the structure of Pig programs is conducive to leveraging parallel processing, making them well-suited for handling very large datasets in a distributed computing environment.

Question 25

What is the application workflow in YARN

Answer 25

1. Client submits an application
2. Resource Manager allocates a container to start the application manager
3. The application manager registers itself with the resource manager
4. The Application Manager negotiates containers from the Resource Manager
5. The Application Manager notifies the Node Manager to launch containers
6. Application code is executed in the container
7. Client contacts Resource Manager/Application Manager to monitor application’s status
8. Once the processing is complete, the Application Manager un-registers with the Resource Manager