Hadoop and HDFS

Question 1

Hadoop

Answer 1

Apache open source software framework for reliable, scalable,
distributed computing over massive amount of data

Question 2

Consists of 4 sub projects:

Answer 2

MapReduce

Hadoop Distributed File System (HDFS)

Yarn

Hadoop Common

Question 3

Hadoop: principles

Answer 3

Handles massive amount of data:

Structred, unstructred or semi-structred

Uses Google’s MapReduce and Google File System (GFS)
technologies as its foundation

Uses commodity hardware: relatively inexpensive
computers

Great perfomance for massive parallel processing
Very large file (big) rather than many small files

Question 4

Hadoop : scalability

Answer 4

Vertical scalability : Disk latency (speed of reads and
write)

Horizontal scalbility: spread the data out accross a huge
cluster of machines

Question 5

Hadoop is good for:

Answer 5

processing massive amounts of data through parallelism

handling a variety of data (structured, unstructured,
semi-structured)

using inexpensive commodity hardware

Question 6

Hadoop is not good for:

Answer 6

processing transactions (random access)

when work cannot be parallelized

low latency data access

processing lots of small files

intensive calculations with small amounts of data

Question 7

NameNode reads fsimage in memory
DataNode sends blocks reports

Answer 7

NameNode Startup
Read FsImage
Read EditLog Transactions
Apply Transactions to FsImage
Save FsImage
Clear EditLog
Start Accepting Requests

Question 8

In case of failure on NameNode what to do ?

Answer 8

recover from FsImage
and EditLog

Question 9

Adding a file to HDFS:

Answer 9

1. Client ask NameNode to dertermie which DataNodes will store the
   replication of each block
2. NameNode create a file in the file system
3. File is added to NameNode memory by persisting info in edits log
4. Data is written in blocks to DataNodes
   o DataNode starts chained copy to two other DataNodes
   o if at least one write for each block succeeds, the write is
   successful

Question 10

HDFS

Answer 10

Distributed, scalable, fault tolerant, high throughput
NOT RANDOM ACCESS

Question 11

Master: NameNode

Answer 11

manages the file system namespace and metadata
 FsImage
 Edits Log
 regulates client access to files

Question 12

Slave: DataNode

Answer 12

manages storage attached to the nodes
*periodically reports status to NameNode