Week 1: Big Data Analytics

Question 1

Human-generated Data

Answer 1

Examples include social media content, email messages, documents, etc.

Question 2

Machine-generated Data

Answer 2

Examples include DBMS log, sensor readings, network traces, etc.

Question 3

Structured Data

Answer 3

Sticks to a data model or schema. Can be managed by DBMS. Usually in dataset/table format. Examples include banking transactions, electronic health records.

Question 4

Unstructured Data

Answer 4

Doesn’t stick to data model or schema. Data types include textual, binary data. Can be stored as BLOBS (Binary Large Objects) in a DBMS or in NoSQL databases. Examples include tweets, video files.

Question 5

Semi-structured Data

Answer 5

Non-relational data with a certain level of structure or consistency. Can be hierarchical or graph-based. Examples include spreadsheets, XML data, sensor data, JSON data

Question 6

JSON

Answer 6

An open standard format that uses human-readable text to send data objects made up of attribute-value pairs. Used in MongoDB.

Question 7

Metadata

Answer 7

It provides information about a dataset’s characteristics and structure. Examples include XML tags for the author and the creation date of the document. In Linux, common metadata includes, size, permissions, creation date, access date, inode number, file type, and etc. Can be accessed by “ls -la”, “stat” commands in Linux.

Question 8

Big Data Characteristic: Volume

Answer 8

Indicates data quantity, which is large and evergrowing. Specialised technologies are needed to store and process large columns of data.

Question 9

Big Data Characteristic: Velocity

Answer 9

indicates data speed, which might be high. High-velocity data can change quickly. Velocity can impact elasticity (sensitivity to changes in other variables), and available time for data processing.

Question 10

Big Data Characteristic: Variety

Answer 10

Indicates how big data can have multiple formats and types. Some big data might have special requirements for integration, especially how to join and combine the data. It can also affect integration, transformation, processing, and storage requirements.

Question 11

Big Data Characteristic: Veracity

Answer 11

Indicates level of bias, noise, abnormalities in big data. As such, removing the noise and invalid values is essential. This process can vary based on different requirements.

Question 12

Big Data Characteristic: Value

Answer 12

Indiciates utility and usefulness of the data. For example, if it takes 3 days to predict the price of a stock, there’s no room for day trading.

Question 13

Analytics Goal: Descriptive

Answer 13

Focuses on what happened, based on past data presented in a summarised form.

Question 14

Analytics Goal: Diagnostic

Answer 14

Focuses on why something happened based on past data.

Question 15

Analytics Goal: Predictive

Answer 15

Focuses on what is likely to happen based on existing data.

Question 16

Analytics Goal: Prescriptive

Answer 16

Focuses on what can be done to make something happen based on existing data.

Question 17

Major Computational Task: Basic Statistics

Answer 17

Statistically summarising data. Popular measures include mean, median, variance, count, top-N, distinct values, etc. The goal is descriptive.

Question 18

Major Computational Task: Linear Algebraic Computation

Answer 18

The result is a model describing the data or a smaller dataset built from the data. The goal is descriptive, diagnostic, and predictive.

Question 19

Major Computational Task: Generalised N-body Problem

Answer 19

This kind of problem involves finding similarities between data points in the dataset. Examples include clustering and classification. Challenges include high dimensionality. The goal is diagnostic, predictive, and prescriptive.

Question 20

Major Computational Task: Graph-theoretic Computations

Answer 20

These computations involve data in graph form. Example tasks include searching for nodes and finding the shortest paths. Challenges include high interconnectivity. The goal is diagnostic, predictive, and prescriptive.

Question 21

Major Computational Task: Optimisation

Answer 21

This involves finding the set of parameters in which the selected objective function is solved. Can be used to find optimal models and validate findings. The goal is prescriptive.

Question 22

Major Computational Task: Integration

Answer 22

This involves finding the high dimensional integrals of functions. The goal is predictive and prescriptive.

Question 23

Major Computational Task: Alignment Problems

Answer 23

This involves determining whether two entities are the same. Examples include finding synonyms intext, and seeing if the same entity is present in multiple images. The goal is predictive and prescriptive.

Question 24

Data Configuration Type: Default

Answer 24

In this type, the dataset is stored in RAM.

Question 25

Data Configuration Type: Streaming

Answer 25

The data arrives in a stream, with a part/window being stored.

Question 26

Data Configuration Type: Distributed

Answer 26

The data is distributed over multiple machines, in RAM and/or disk.

Question 27

Data Configuration Type: Multi-threaded

Answer 27

The data is stored in one machine, and multiple processors share the RAM of the machine.