Revature Hadoop 2

Question 1

Explain the significance of Mapper[LongWritable, Text, Text, IntWritable] and Reducer[Text, IntWritable, Text, IntWritable]

Answer 1

The Mapper reads input key-value pairs (LongWritable, Text) and outputs intermediate key-value pairs (Text, IntWritable). The Reducer takes these intermediate key-value pairs and outputs final results (Text, IntWritable).

Question 2

What needs to be true about the types contained in the above generics?

Answer 2

The input and output key-value pairs must match the data types specified by the Mapper and Reducer, and the input types of Reducer must match the output types of Mapper.

Question 3

What are the 3 Vs of big data?

Answer 3

Volume, Velocity, Variety.

Question 4

What are some examples of structured data? Unstructured data?

Answer 4

Structured: Relational databases, CSV files. Unstructured: Images, videos, emails.

Question 5

What is a daemon?

Answer 5

A background process that runs continuously to provide services or perform tasks (e.g., NameNode, DataNode in Hadoop).

Question 6

What is data locality and why is it important?

Answer 6

Data locality refers to the concept of storing and processing data on the same computer or network node where it is needed, minimizing the need to transfer large amounts of data across the network, thus significantly improving performance and efficiency

Question 7

How many blocks will a 200MB file be stored in in HDFS, if we assume default HDFS block size for Hadoop v2+?

Answer 7

2 blocks (default block size is 128MB).

Question 8

What is the default number of replications for each block?

Answer 8

3

Question 9

How are these replications typically distributed across the cluster? What is rack awareness?

Answer 9

Replications are stored across different nodes and racks for fault tolerance. Rack awareness ensures copies of data are distributed across racks to prevent data loss from rack failure.

Question 10

What is the job of the NameNode? What about the DataNode?

Answer 10

NameNode manages metadata and file system structure. DataNode stores actual data blocks and performs read/write operations.

Question 11

How many NameNodes exist on a cluster?

Answer 11

Typically one active NameNode, but high-availability setups include one standby NameNode.

Question 12

How are DataNodes fault tolerant?

Answer 12

Data blocks are replicated across multiple DataNodes.

Question 13

How does a Standby NameNode make the NameNode fault tolerant?

Answer 13

It keeps a synchronized copy of metadata from the active NameNode and can take over if the active NameNode fails.

Question 14

What purpose does a Secondary NameNode serve?

Answer 14

It periodically merges NameNode’s metadata and edits logs, reducing recovery time.

Question 15

How might we scale a HDFS cluster past a few thousand machines?

Answer 15

HDFS Federations with multiple NameNodes can be used for large clusters with tens of thousands of machines.

Question 16

In a typical Hadoop cluster, what’s the relationship between HDFS data nodes and YARN node managers?

Answer 16

HDFS DataNodes and YARN NodeManagers typically coexist on the same nodes, enabling data locality for optimized processing.

Question 17

When does the combine phase run, and where does each combine task run?

Answer 17

The combine phase runs after the Mapper phase, locally on the Mapper’s output.

Question 18

Know the input and output of the shuffle + sort phase.

Answer 18

Input: Intermediate key-value pairs from Mapper output. Output: Sorted key-value pairs grouped by key for the Reducer.

Question 19

What does the NodeManager do?

Answer 19

Manages resources on a node and launches containers for task execution.

Question 20

What about the ResourceManager?

Answer 20

Manages cluster resources and schedules jobs.

Question 21

Which responsibilities does the Scheduler have?

Answer 21

Allocates resources to applications based on policies like FIFO, Capacity, or Fair scheduling.

Question 22

What about the ApplicationsManager?

Answer 22

Manages the lifecycle of applications, including application submission and monitoring.

Question 23

What is an ApplicationMaster? How many of them are there per job?

Answer 23

The ApplicationMaster manages the lifecycle of a specific job, including resource requests and task execution. There is one ApplicationMaster per job.