Section B.1: Data Exploration/Data Cleansing

Question 1

Data Exploration

Answer 1

The process of examining and understanding data in order to identify patterns, relationships, and trends that can be used to generate insights and support decision-making.

Question 2

Data exploration involves

Answer 2

Data exploration involves the collection, cleaning/pre-processing, visualisation, analysis, of data.

Question 3

Data Exploration processes (VMOVVCUBM)

Answer 3

Variable identification
Missing values treatment
Outlier treatment
Variable transformation
Variable creation
Check data stucture
Univariate analysis
Bivariate analysis
Multivariate analysis

Question 4

Variable identification

Answer 4

Identifying: Variable type (dependent or idependent), Data type (numeric or character), and Variable category (categorical or continuous)

Question 5

Data Cleansing/Pre-processing

Answer 5

Data cleaning is the process of detecting and correcting anomalies within the data to ensure it is accurate, complete, and useful for analysis.

Question 6

Missing values deletion

Answer 6

Missing values are deleted either:
List-wise: Removing the whole row for simplicity
Pair-wise: Analysing only the values that are present for keeping as many cases as possible

Question 7

Missing values

Answer 7

Missing values are data that is missed because of data extraction problems, that can distort estimates, they can be classified in three types: MAR (Missing at Random), MNAR (Missing Not At Random), or MCAR (Missing Completely At Random).

Question 8

Methods for treating missing values (3)

Answer 8

Mean/mode/median imputation
Prediction modelling
KNN imputation

Question 9

What is Mean/median/mode imputation for missing value treatment?

Answer 9

Mean/mode/median imputation - estimating and replacing missing values based on the mean, median, or mode of all known values of that variable

Question 10

What is prediction modelling for missing values treatment?

Answer 10

Predicting the missing values based on regression modelling

Question 11

What is KNN imputation?

Answer 11

kNN imputation uses an algorithm to impute the missing value based on the values nearest or most similar to the missing values.

Question 12

What is an outlier?

Answer 12

An outlier is an extreme value within a dataset that diverges from the overall sample pattern, which can distort statistics, measures, models and graphs.
They are defined as values beyond ±1.5 x IQR, or 3 x σ from the mean (three sigma limit)

Question 13

How would you detect outliers in a dataset?

Answer 13

Detect outliers by graphing the features or data points using a scatterplot or boxplot.

Question 14

What is Feature/Variable creation?

Answer 14

Feature or Variable Creation is the process of generating a new variable or feature based on existing variables.

E.g. data$windchill <- windspeed/temp

Question 15

What are the Data Structures (in R)?

Answer 15

All data in R is stored in an object, there are five different types of these objects, which are: Vector, Matrix, List, Data Frame, Factor)

Question 16

Vector

Answer 16

A vector is what is called an array in all other programming languages except R - a collection of cells with a fixed size where all cells are the same type (numerical or characters)
created using an equation (below) or the c() function

> x = 1:7
x[1] 1 2 3 4 5 6 7

Question 17

Matrix

Answer 17

A matrix is a two-dimensional vector with a fixed size and all cell types the same. Created using the matrix() function

Question 18

List

Answer 18

A list is a data structure that can hold different types of data and is not a fixed size, and is created using the list() function.

Question 19

Factors

Answer 19

A factor is a data structure that can use levels to store predefined categorical data, and is created using the factor() function.

Question 20

Data Frame

Answer 20

A data frame is a data structure that is a table with rows and columns, where each column stores data of the same type, but different columns can store different data types. It is created using the data.frame()

Question 21

Data cleansing methods? (5)

Answer 21

Duplicate removal, missing values treatment, outlier treatment, consistency checking, format correction.

Question 22

Duplicate removal

Answer 22

Duplicate removal is the process of removing duplicate entries from a dataset. They should be removed as they may lead to bias, hurting the accuracy of estimates made about the data.

Question 23

What are the causes of missing values within a dataset? (2)

Answer 23

Data can have missing values for many reasons, which are categorised in two ways as data collection problems, and data extraction problems.

Question 24

What are the causes of outliers within a dataset?

Answer 24

1. Bad data entry - unintentional or intentional error
2. Measurement error
3. Experimental/sampling error
4. Processing error
5. Natural outliers

Question 25

Methods for treating outliers? (4 DOST)

Answer 25

1. Delete observations
2. Split the model into two ( 1 w/, 1w/o)
3. Overwrite (mean, median, mode imputation)
4. Check for Typos