Hadoop - Sheet1 Flashcards
What’s Hadoop
Apache Hadoop is a free, open-source, Java-based software framework used to store, maintain, and process large-scale sets of data across numerous clusters of commodity hardware
Why is Hadoop useful
Hadoop is fault tolerant, meaning the system will simply redirect to another location and resume work when a node is lost. Hadoop is also schema-less and can absorb data of all types, sources, and structures, allowing for deeper analysis
What are the four modules that make up the Apache Hadoop framework?
- Hadoop Common, which contains the common utilities and libraries necessary for Hadoop’s other modules.
- Hadoop YARN, the framework’s platform for resource-management
- Hadoop Distributed File System, or HDFS, which stores information on commodity machines
- Hadoop MapReduce, a programming model used to process large-scale sets of data
What are “slaves” and “masters” in Hadoop?
In Hadoop, slaves are a list of hosts for task tracker servers and datanodes. Masters list hosts for secondary namenode servers.
What is a Namenode?
Namenode exists at the center of the Hadoop distributed file system cluster. It manages metadata for the file system, and datanodes, but does not store data itself.
How many Namenodes can run on a single Hadoop cluster?
Only one Namenode process can run on a single Hadoop cluster. The file system will go offline if this Namenode goes down.
What is a datanode?
Unlike Namenode, a datanode actually stores data within the Hadoop distributed file system. Datanodes run on their own Java virtual machine process.
How many datanodes can run on a single Hadoop cluster?
Hadoop slave nodes contain only one datanode process each.
What is job tracker in Hadoop?
Job tracker is used to submit and track jobs in MapReduce.
How many job tracker processes can run on a single Hadoop cluster?
Like datanodes, there can only be one job tracker process running on a single Hadoop cluster. Job tracker processes run on their own Java virtual machine process. If job tracker goes down, all currently active jobs stop.
What sorts of actions does the job tracker process perform?
- Client applications send the job tracker jobs.
- Job tracker determines the location of data by communicating with Namenode.
- Job tracker finds nodes in task tracker that has open slots for the data.
- Job tracker submits the job to task tracker nodes.
- Job tracker monitors the task tracker nodes for signs of activity. If there is not enough activity, job tracker transfers the job to a different task tracker node.
- Job tracker receives a notification from task tracker if the job has failed. From there, job tracker might submit the job elsewhere, as described above. If it doesn’t do this, it might blacklist either the job or the task tracker.
How does job tracker schedule a job for the task tracker?
When a client application submits a job to the job tracker, job tracker searches for an empty node to schedule the task on the server that contains the assigned datanode.
What does the mapred.job.tracker command do?
The mapred.job.tracker command will provide a list of nodes that are currently acting as a job tracker process.
What is “jps”?
jps is a command used to check if your task tracker, job tracker, datanode, and Namenode are working.
What is a “map” in Hadoop?
In Hadoop, a map is a phase in HDFS query solving. A map reads data from an input location, and outputs a key value pair according to the input type
