Big Data Lecture 04 Distributed File Systems

Question 1

What are two different types of data by source?

Answer 1

<ul><li>Raw data: sensors, events, logs...</li><li>derived data: aggregated data, intermediate data...</li></ul>

Question 2

What are two types of big data? And which is handle by HDFS and which by Cloud Storage?

Answer 2

<ul><li>A huge amount (billions) of large (&lt;TB) files (handled by S3 or Azure),</li><li>or large amount (millions) of huge (&lt;PB) files (handled by HDFS).</li></ul>

Question 3

How are files organized in HDFS?

Answer 3

<ul><li>File structure,</li><li>underneath which is block storage.</li></ul>

Question 4

Why do we require fault tolerance in big file systems?

Answer 4

If there is a large number of nodes, the system is guaranteed to fail from time to time.

Question 5

What is the file reading and updating model in DFS?

Answer 5

<ul><li>Reading: we need to scan the file, we do not need random access,</li><li>update: we need to append to the file, possibly from many different atomic clients.</li></ul>

Question 6

What are performance requirements of DFS?

Answer 6

We want the bottleneck to be throughput, not latency.<br></br><br></br>That is, we do not want to spend time fetching stuff, and then waiting for it to go onto the network. We want to be limited by the network speed, that is why we prefer bigger blocks, and we fetch those.

Question 7

How do we solve discrepancy of capacity and throughput, and throughput and latency?

Answer 7

<ul><li><span>Paralelization,</span></li><li><span>batch processing.</span></li></ul>

Question 8

What is the size of the block in HDFS?

Answer 8

64/128 MB

Question 9

What is the network architecture of HDFS?

Answer 9

One coordinator node, the rest are worker nodes, however, they can still talk to each other.

Question 10

How are files treated in HDFS?

Answer 10

They are split into blocks, which are replicated over worker nodes.

Question 11

What does coordinator node hold in HDFS?

Answer 11

1. File namespace (structure in the file system),<br></br>2. file to block mapping,<br></br>3. block locations.

Question 12

What if a file is not exactly a multiple of 128 MB sized?

Answer 12

The final block will not take up more space, only the space that is needed.

Question 13

How does NameNode talk to client?

Answer 13

Client sends requests, and namenode sends DataNode locations and IDs of blocks. Client has Java API to hold stuff together to actually download this data from the DataNode.

Question 14

How does NameNode talk to the DataNode?

Answer 14

It sends a heartbeat every 3s, saying it is alive, and if it has received any new blocks, in response to that NameNode sends commands back.<br></br><br></br>Every 6 hrs it sends a block report of the whole storage.

Question 15

How does download of a file look over HDFS?

Answer 15

The clients streams from different nodes at the same time and then recollects and assembles the blocks together.

Question 16

How does user add a file to the HDFS?

Answer 16

They first talk to the NameNode, which sends instructions about where to upload to (per block), then it sends a packet of 64 kB for each file, for each of those it receives acknowledgement that it has finished uploading and replicating to all the nodes.<br></br><br></br>Finally, acknowledgment and release of lock comes from the NameNode.

Question 17

How are replicas placed in HDFS?

Answer 17

1. Same node as the client (easy replication),<br></br>2. node in a different rack B,<br></br>3. node in the same rack B.<br></br>Further: random rack, but if possible at most one replica per node and at most two replicas per rack.

Question 18

What is the weak point of HDFS?

Answer 18

The NameNode, there is only single one, and if it fails, we have a problem.

Question 19

What data is backed-up from the NameNode and how?

Answer 19

The directory tree and mapping of files from blocks is saved.<br></br><br></br>We do not save which DataNodes have which blocks, cause those can be easily recovered from BlockReport that is given every 6 hours.<br></br><br></br>We store (onto a separate drive, or on cloud) into a Namespace file, on top of which we store Edit log, which is merged to the Namespace file from time to time.

Question 20

How long does it take to restart the NameNode and why?

Answer 20

We need to apply the Edit log onto the Namespace, so we need around 30 minutes for that.

Question 21

How can we achieve High Availability (HA)?

Answer 21

We need to make the NameNode redundant, this can be done in multiple ways:<br></br><ul><li>Standby NameNodes,</li><li>Observer NameNodes</li><li>Federated DFS.</li></ul>

Question 22

What are Standby NameNodes? What is Quorum Journal Manager?

Answer 22

Running duplicate of the NameNode, which is inspecting the tail of the edit log and copying it. It also receives reports from DataNodes.<br></br><br></br>The NameNodes can be parralelized and as long as the majority agrees, then the system can be kept alive and running.

Question 23

What is Observer NameNode?

Answer 23

StandbyNode++, additionally can serve requests for reading data from DataNodes.

Question 24

What is Federated DFS?

Answer 24

Different parts of directories get a different NameNode, additionally, there exists duplication between the nodes, so one could be a standby node for the other.

Question 25

How to use HDFS locally?

Answer 25

From the command line, one can use<br></br><i>hadoop fs <args></i><br></br>which will work with basically any system using correct domain (local - with FILE as domain, or S3 and HDFS).<br></br><br></br>In the flags, we can specify commands like<br></br><i>-ls, -cat, -rm, -mkdir, -copyFromLocal, -copyToLocal</i>

Question 26

What is Apache Flume?

Answer 26

Tool that collects, aggregates and moves the log data into HDFS.

Question 27

What is Apache Sqoop?

Answer 27

Tool that imports data from relational database into HDFS.

Question 28

How to create a directory in HDFS?

Answer 28

First talk to the Name node, obtain DataNode ID, then talk to that one and upload.

Question 29

Is DynamoDB CAP?

Answer 29

No, it is not time constistent, it uses vector clocks! It is AP though.