class 2

Question 1

file system types

Answer 1

fs (File system)

DFS (distributed file system)

HDFS (Hadoop distributed file system)

Question 2

drawbacks of ‘FS’

Answer 2

1. Storing the large amount of data
2. Processing large amount of data
3. dataloss:
   - power failure
   - network failure
   - Hardware or software failure
4. Auto meta data concepts

DFS:

multi node concept

remaining all are same defects of FS

Question 3

cluster

Answer 3

group of machines in a network

Question 4

distribution of hadoop

Answer 4

cloudera

hortonworks

IBM big insights

pivotal hd

mapr

Cloud computing:

Amazon EMR

google cloud

windows azure

Question 5

Hadoop framework

Answer 5

Hadoop 1.x –> 2010

Hadoop 2.x –> Yarn -> 2013

Question 6

Hadoop 1.x

Answer 6

1. Name node -> to store metadata
2. data node -> to store actual data
3. Secondary Name node -> To maintain backup of name node
4. Job tracker -> to split the job into tasks and assign tasks to task tracker
5. Task tracker -> to execute the task

Question 7

client request process sequence

Answer 7

1. client sends request to name node (which stores metadata)
2. if it is a namenode, the metadata is updated
3. if it is an existing request -> the location of actual data is provided (data node)
4. secondary namenode backups the name node
5. Job tracker: split the tasks and assign to task trackers
6. task trackers are initiated where the data resides

Question 8

Name node contains

Answer 8

1. metadata
2. location of the data
3. ip address of the data nodes

secondary namenode only backups the data, but doesn’t interact with job trackers

hadoop 1.x is also called master slace architecture

Question 9

master slave architecture

Answer 9

job tracker and task tracker maintaing rpc communication

rpc - remote procedural communication

bydefault - 3 seconds

every 3 second task tracker sends heartbeat to job tracker, to notify that it is working and not down.

case 1: if data node is down

since we have replication, we can execute the task

case 2: if task tracker is down

we can assign the task to another task tracker

case 3: if secondary name node is down

we will not stop job execution, just the backup is stopped

case 4: if the job tracker has stopped

we stop the execution - SPOF (single point of failure)

case 5 : namenode is down

since the secondary namenode not for processing, so SPOF

we have two drawbacks in hadoop, namenode or job tracker is down, then processing is stopped