Big Data Lecture 03 Cloud Storage

Question 1

What is the issue with big datasets?

Answer 1

Do not fit on a single machine, e.g. the Sloan Digital Sky Survey dataset has 273TB of data in 680 000 directories and 176 000 000 files.

Question 2

What things are broken in the world of NoSQL?

Answer 2

<ul><li>Relational integrity,</li><li>domain integrity,</li><li>atomic integrity (1st normal form),</li><li>2nd/3rd/Boyce-Codd normal form.</li></ul>

Question 3

What new data properties are added in NoSQL?

Answer 3

<ul><li>Heterogenous data (no schema),</li><li>nested data (break atomic integrity),</li><li>denormalized data (no normal forms).</li></ul>

Question 4

Describe the tech stack of big data systems.

Answer 4

<img></img>

Question 5

What is ETL?

Answer 5

When loading data into traditional database, we need to Extract, Transform  and Load (ETL it).

Question 6

What is a data lake? What file operations is it meant for?

Answer 6

As opposed to traditional database, it reads data directly from the file system.<br></br><br></br>Meant for reading, querying not for editing.

Question 7

How are files stored in a file system?

Answer 7

File content is stored in blocks, usually of 4kB, if a file is not exactly of this side, a whole block is taken up anyway.

Question 8

What networks does local storage use?

Answer 8

<ul><li>Local machine,</li><li>LAN (local area network), NAS (drive on network),</li><li>not WAN (wide area network).</li></ul>

Question 9

Principle of scaling from simple data storage into data lake?

Answer 9

<div>Simplify!</div>

<ul><li>Throw away folder structure, use flat objects,</li><li>give data unique ID (key-value model).</li></ul>

Question 10

How to scale a system expensively and cheaply?

Answer 10

<ul><li>Expensively: scale up - buy a larger, stronger machine,</li><li>cheaply: scale out - buy many cheap machines,</li><li>be smart! Optimize your code!</li></ul>

Question 11

What are the constraints on the data centers? What are the number of machines and cores?

Answer 11

Due to electricity grid and cooling:<br></br><ul><li>1000 - 100 000 machines in a data center,</li><li>1-200 CPU cores per machine.</li></ul>

Question 12

How big is a local storage, memory and bandwidth per server?

Answer 12

<ul><li>1-30 TB of storage,</li><li>16GB-24TB of RAM,</li><li>1-200 Gbit/s.</li></ul>

Question 13

How are servers stored in a data center?

Answer 13

They are in server racks, one rack has 42 rack units.<br></br><br></br>This ensures modularity, as we can stack servers storage and routers into the same rack.<br></br><br></br>Each unit has 1-4 rack units.

Question 14

Describe S3 data storage model.

Answer 14

<ul><li>Data is stored in buckets, each has a (worldwide) unique ID,</li><li>files (max. 5 TB) are stored as objects in the buckets, denoted by (in-the-bucket) unique ID.</li></ul>

Question 15

What guarantees does S3 offer in SLA (service level agreement)?

Answer 15

<ul><li>Durability: 99.999999999% (lose 1 in 10¹¹&nbsp;objects),</li><li>availability: 99.99% (down 1h/year),</li><li>response time: &lt; 10 ms in 99.9% of cases (not mean or average).</li></ul>

Question 16

Explain CAP theorem.

Answer 16

Impossibility triangle - storage system cannot be:<br></br><ul><li><i>C</i>onsistent (all data agree in all backups and versions),</li><li><i>A</i>vailable (reachable with low latency),</li><li><i>P</i>artition tolerant (breaking up network),</li></ul><div>all at the same time.</div>

Question 17

What are REST APIs?

Answer 17

<b>Representational state transfer</b>: peer-2-peer HTML-style protocol for file transfer. 

Question 18

How are resources reffered to? What are parts of it?

Answer 18

Using URI (uniform resource identifier), which has<br></br><ul><li>scheme: https</li><li>domain: www.example.com</li><li>path: api/collection/foo/object/bar</li><li>query: ?id=foobar</li><li>fragment: #head</li></ul>

Question 19

What HTTP methods are there and what do they do? Are they idempotent?

Answer 19

<ul><li>GET: obtains resource,</li><li>PUT: stores resource,</li><li>DELETE: deletes resource,</li><li>POST: anything else.</li></ul>

<div>Only POST&nbsp;is not idempotent.</div>

Question 20

How does the HTTP protocol work?

Answer 20

<ul><li>Request is send with header and body,</li><li>Response is issues with status code, header and body.</li></ul>

Question 21

Do URIs on Cloud Storage use file structure with slashes?

Answer 21

No, but you can use slashes to create logical structure for yourself.

Question 22

Do data centers get filled up to full?

Answer 22

No, they are filled up to 70-80% then resources are rellocated / new center has to be built.

Question 23

What is intra-stamp replication?

Answer 23

Synchronous method of duplication of data, done on client upload.

Question 24

What is inter-stamp replication?

Answer 24

After user has finished uploading, the resources are duplicated to different places in the data center asynchronously.

Question 25

Why are data centers spread around different regions?

Answer 25

1. To optimize user latency,<br></br>2. increase resilience to natural catastrophes.

Question 26

Is object storage a database?

Answer 26

No, the retrieval takes too long (>100 ms, cf. <10 ms for typical databases), so we cannot do our lovely operations.

Question 27

What are key-value stores?

Answer 27

1. Similar data model to object storage,<br></br>2. but with smaller objects,<br></br>3. and no metadata.

Question 28

Why and how do we simplify to key-value store?

Answer 28

We require:<br></br><ul><li>simplicity,</li><li>only eventual consistency,</li></ul><div>to obtain</div><ul><li>increased performance,</li><li>scalability.</li></ul>

Question 29

How do we query key-value store?

Answer 29

Associative array, aka map, since hash-map is not scalable to multiple machines.

Question 30

What is the design principle of incremental stability?

Answer 30

Possibility to add and remove machines from the network.

Question 31

What is the design principle of symmetry?

Answer 31

All machines run the same system on board.

Question 32

What is the design principle of decentralization?

Answer 32

There is no central machine in the system.

Question 33

What is the design principle of heterogeneity?

Answer 33

The machines can be different, have different resources, and it is okay.

Question 34

How are nodes connected in a data center?

Answer 34

As a peer-2-peer network.

Question 35

How are files assigned to devices in a cluster? How are new nodes added and removed?

Answer 35

They are hashed (uniformly), and then on a logical circle machines take care of certain data range. <br></br><br></br>The range is cut up / merged and the data is transferred.<br></br><br></br>The data is redundant, the node ranges overlap (over N ranges).

Question 36

How does the protocol know where to find the data?

Answer 36

The nodes hold a list of pointers:<br></br><ul><li>Chord: finger tables (powers of two of what is where).</li><li>Dynamo: preferrence lists (every node knows about the ranges of all other nodes).</li></ul>

Question 37

What are R, W and N and how do they relate?

Answer 37

<ul><li>N - number of data duplicates,</li><li>R - number of nodes each node reads from,</li><li>W - number of nodes each node write to (synchronously).</li></ul>

<div>It must be that at each moment: R + W &gt; N, then we know that if we are reading from + writing to more nodes then there are, then if there is conflict we will know.</div>

Question 38

How does the data lake system handle a request for data?

Answer 38

<div><br></br></div>

<ul><li>First load balancer assigns a random node to be asked,</li><li>the random node figures out who is the coordinator (e.g first node with data) and asks them,</li><li>coordinator redirects the request to N-1 nodes hosting replicas.</li></ul>

Question 39

What are pros and cons of distributed hash tables?

Answer 39

+: highly scalable, robust against failure, self-organizing,<br></br>-: no lookup or search, data integrity, security issues.

Question 40

How to increase the number of nodes and the elasticity of a distributed storage system?

Answer 40

Do not depend on machines for duplication, but introduce tokens (multiple for each machines) as a form of virtualization. Those can now have the same role as nodes have had previously.

Question 41

How are tokens spread over machines?

Answer 41

They are distributed over different machines to increase robustness.

Question 42

What is the concept of a vector clock? How are conflicts resolved?

Answer 42

Each node in the system keeps a counter of its modifications, which is incremented whenever the node writes something. This is passed onto other nodes/client on request.<br></br><br></br>There can different nodes writing, and they increase their own counter at each time.<br></br><br></br>The vectors form a directed acyclic graph (DAG), so on comparison of two versions, if we cannot compare (all are maximal elements, but not suprema) we need to merge (done by user).

Question 43

What is the difference of Amazon and Azure mindsets?

Answer 43

Amazon has many different services, that all do one thing, and you get rerouted to them.<br></br><br></br>Azure has all hardware being able to do everything, and doing a bit of everything.

Question 44

Describe Azure data storage.

Answer 44

<ul><li>Objects, are denoted by account, container and blob,</li><li>there are 3 types of blobs, Block Blob (file, at most 190.7 TB), Append Blobs (at most 195 BG)
and Page Blobs (for storing and accessing the memory of virtual
machines).<br></br></li></ul>