4.11 - Big Data

Question 1

What is Big Data?

Answer 1

Catch-all term for data that doesn’t fit usual containers

Question 2

What are the three defining features of big data?

Answer 2

Volume, velocity and variety

Question 3

What is volume?

Answer 3

Too much data to fit on conventional hard drive or server, so data must be stored over multiple servers

Question 4

What is velocity?

Answer 4

Data is created and modified rapidly, so servers must respond to frequently changing data within a matter of milliseconds.

Question 5

What is variety?

Answer 5

Data held on servers consists of many different types, from binary files to multimedia like photos and videos.

Question 6

What is the biggest problem with big data and why?

Answer 6

Its unstructured nature makes it difficult to analyse the data. Conventional databases are not suited to it because they require data to conform to a row and column structure. Furthermore, conventional databases do not scale well over multiple servers.

Question 7

How is useful information extracted from big data?

Answer 7

Machine learning techniques are used to discern patterns in the data.

Question 8

What is the solution to processing data over multiple machines and why?

Answer 8

Functional programming, since functional programs are stateless made make use of immutable data structures. Furthermore, it supports higher-order functions.

These attributes make it easier to write correct, efficient, distributed code.

Question 9

What is the fact-based model for representing data?

Answer 9

Each individual piece of info is stored as a fact. Facts are immutable and can’t be overwritten. Each fact has time stamp stored with it, indicating date and time when info was recorded. Facts never deleted or overwritten, so multiple values can be held for same attribute.

Reduces risk of accidentally losing data due to human error, and does away with index for data and instead simply appends new data to dataset.