Week 1: Big Data Introduction

Question 1

Where does machine generated data come from? (3 types)

Answer 1

1. Internet of things
2. smart devices
3. sensing

Question 2

Where does people generated data come from ? (4 examples) (BOSE)

Answer 2

• Blogging
• Online photo/video sharing
• Social networks
• Emails

Question 3

Describe big data generated by people (4 formats)

Answer 3

• Typically unstructured and text heavy
• Multiple data formats:

1. Web pages
2. images
3. PDF
4. XML

Question 4

Describe big data generated by organisations (3 points)

Answer 4

• Typically structured
• stored in RDBMS
• Often siloed by department
• Limited infrastructure to share and integrate this data

Question 5

Define: big data

Answer 5

• Big data is high volume, high velocity and/or high variety information assets
• require new forms of processing
• to enable enhanced decision making
• insight discovery
• process optimisation.

Question 6

Characteristics of big data volume

Answer 6

• Refers to the vast amount of data that is generated every second, minutes, hours and days
• the dimension of big data and its exponential growth

Question 7

Characteristics of big data velocity

Answer 7

• Refers to the speed at which data is being generated
• increasing speed at which the data needs to be stored and analysed
• Processing of data in real-time to match its production rate

Question 8

Characteristics of big data variety

Answer 8

• Refers to the ever increasing different forms that big data can come in such as
• text
• images
• voice
• spatial data

Question 9

Characteristics of big data veracity

Answer 9

Limited value if data is not accurate

Refers to the quality of the data, free from

• biases
• noise
• abnormalities

Data must be:

• accuracate
• trustworthy
• reliable
• and have context within analysis

Question 10

5 steps in the data science process

Answer 10

• Acquire
• Prepare
• Analyse
• Report
• Act

Question 11

What 3 ways can we acquiring data

Answer 11

1. Finding the right data sources
2. Conventional data may exist in RDBMS
3. NoSQL storage system can be used to manage variety of data types in big data

Question 12

Prepare data

Answer 12

• Preliminary investigations
• Correlations,
• general trends,
• outliers
• Summary statistics
• Mean
• Median
• Range
• std dev
• Visualisation
• Histogram
• scatter plots

Question 13

Why do we pre-process data

Answer 13

• to clean the data to address data quality issues

Question 14

Why is domain knowledge important

Answer 14

To make informed decisions on how to handle incomplete or incorrect data

Question 15

Methods of pre processing data

Answer 15

• Scaling
• Transformation
• Feature selection
• Dimensional reduction
• Data manipulation
• One hot encoding (categorical data)
• Normalise

Question 16

Analysing data techniques

Answer 16

• Classification
• Regression
• Clustering
• Association analysis
• Graph analysis

Question 17

Report

Answer 17

To report the insights we gained from our analysis

Question 18

Reporting visualtion tools

Answer 18

• Python
• R
• D3
• Leafleet
• Tableau

Question 19

Act

Answer 19

• To determine what action or actions should be taken based on the gained insights
• Does additional analysis need to be performed?
• What data should be revisted

Question 20

Varieties of big data

Answer 20

• Structural variety - format and models
• Media variety - medium of delivery
• Semantic variety - how to interpret and operate on
• Availability variations - real-time vs intermittent

Question 21

Where does machine generated data come from? (3 examples)

Answer 21

1. network logs,
2. equipment logs
3. call detail logs