College 1: Introduction

Question 1

Types of data

Answer 1

• Structured: data warehouse repository, relational data
• Unstructured: wikipedia, news articles (no predefined format or organization, often human-generated)
• Graph data: social networks, road networks, knowledge bases
• Semi-structured: log files, product reviews, json, cml (some level of organization by tags, labels, marker)

Question 2

Large scale data:

Answer 2

Very large collection (terrabytes to exabytes): scientific, business, or user generated.

Question 3

Streaming data

Answer 3

Sensors, feeds, continous analytics. Response times in seconds to nanoseconds.

Question 4

Heterogeneous data

Answer 4

a collection of data consisting of diverse types, formats, or structures. Multiple data sources, data elements varying in terms of representation, organization, or underlying technology. (Computer or human generated)

Question 5

Private data

Answer 5

Correlating data from multiple sources. This poses risks, so privacy and accountability is important. Data such as credit card history, mobile phone usage, GPS tracking.

Question 6

Uncertain data

Answer 6

Inprecision, inconsistencies, incompleteness, ambiguities, latency, deception, approximations, privacy preserving transformations. Process, data and model are uncertain.

Question 7

Data processing pipeline

Answer 7

1. Acquisition & Storage
2. Cleaning, Annotation, Integration, Aggregation
3. Exploration & Querying
4. Modeling & Data Analysis

Question 9

Knowledge Discovery Process (KDD)

Answer 9

**Data Sources ** - Data Cleaning - **Data Warehouse ** - Selection - Task-relevant data - Data mining - Pattern Evaluation -> KNOWLEDGE

Question 10

3V of Big Data Challenges

Answer 10

1. Volume: size of data
2. Velocity: speed at which data is generated, collected, and processed
3. Variety: Refers to the diverse types and formats of data that are encountered in big data (heterogenity)

Question 11

TF-IDF

Answer 11

TF(t,d) = number of times t appears in d/total number of terms in d
IDF(t) = log(N/1+df)
TF-IDF(t,d) = TF(t,d) * IDF(t)

N = total N documents, df = n docs which contain t

Question 12

Clustered vs. Unclustered index

Answer 12

A clustered index is used to define the order or to sort the table or arrange the data by alphabetical order just like a dictionary. A non-clustered index collects the data at one place and records at another place.

Question 13

Hash function

Answer 13

last 2 bits in its binary representation

Question 14

Opportunities data exploitation

Answer 14

Energy saving, financial systemic risk analysis, security, computer security, democracy (citizens understanding how government operates), education, healthcare, urban planning, intelligent transportation, environmental modeling