Lecture 3

Question 1

How is data analytics perfomed in practice ?

Answer 1

Data analytics goes through 4 phases in practice:
1. Preparation
2. Preprocessing
3. Analysis
4. Postprocessing

Question 2

What are the tasks performed during the preparation phase ?

Answer 2

The tasks perfomed during preparation are:
* Planning
* Data collection
* Feature generation
* Data selection

Question 3

What are the tasks performed during the preprocessing phase ?

Answer 3

The tasks perfomed during preprocessing are:
* cleaning
* filtering
* completion
* correction
* standardization
* transformation

Question 4

What are the tasks performed during the analysis phase ?

Answer 4

The tasks perfomed during the analysis are:
* visualisation
* correlation
* regression
* forecasting
* classification
* clustering

Question 5

What are the tasks performed during the postprocessing phase ?

Answer 5

The tasks perfomed during postprocessing are:
* interpretation
* documentation
* evaluation

Question 6

This is for the data preparation!

Why is merging often necessary in data analytics ?

Answer 6

It is often the case that one dataset is insufficient to perform the whole analysis, hence dataset have to be merged. This depends on the number of datasets and type of variables.

Question 7

What are the types of merging possible ?

Answer 7

There are 4 types of merging:
* Appending
* Horizontal stacking
* Join family
* Variable selection

Question 8

When is Appending used ?

Answer 8

Appending is used with 2 datasets with the same variables. That is when the data gets vertically stacked

Question 9

When is Horizontal stacking used ?

Answer 9

Horizontal stacking is used for 2 datasets with the same variables. They get horizontally sequenced.

Question 10

When is Join familiy used ?

Answer 10

It is used for two datasets with different variables. There are 6 types of Join families:
* inner_join: everything that is in one AND the other
* full_join: everything that is in one OR the other
* left_join: everything in the left join AND in the intersection of both
* right_join: everything in the right join AND in the intersection of both
* semi_join: only what is in both, with only variables from the first
* anti_join: what is in neither keeping only the first

Question 11

What is common variable selection?

Answer 11

With more than two datasets, it is about selecting all the common features and joining them

Question 12

What is creation of multiple subdataset?

Answer 12

It is a second technique employed to merge more than 2 datasets. That is to merge subsets of datasets.

Question 13

When does duplicate data occur ?

Answer 13

It occurs from data entry:
* lack of unique identifiers
* lack of integrity or validation checks
* data errors

Or from data merging:
* structural heterogeneity
* lexical heterogeneity

Question 14

What can you do to limit duplicates ?

Answer 14

Duplicates can be prevented by design with:
* use of standards
* integrity rules
* validation checks

Question 15

What is strucutral heterogeneity?

Answer 15

Fields of different databases represent the same information in a structurally different manner.

EX:
DB1: Contact Name
DB2: Salutation, First Name, Last Name

Question 16

What is Lexical heterogeneity?

Answer 16

Fields of different databases are structurally the same, but they represent the same information in a different manner.

DB1 - Address: 32 E St. 4.
DB2 - Address: 32 East,4 th Street

Question 17

What is a unique identifiers?

Answer 17

A unique identifier is a data field that is always unique for an entity

Ex: Social Security Number for customer data, Manufacturer Part Number..

Question 18

What are validation checks ?

Answer 18

Validation check: Do the unique identifiers conform to valid
patterns (for example, AAA-GG-SSSS for SSN)?

Question 19

What are
integrity constraints?

Answer 19

Integrity constraint: Do the identifier comply with the standard length? (for example, 11 char limit for SSN)

Question 20

What can you do when faced with duplicate data ?

Answer 20

You can use deduplication, which is the process of removing duplicates from a dataset. Deduplication techniques for a single dataset are:
* Identify duplicate rows
* Find duplicates in one column
* Find duplicates in one column
* Drop duplicates in one column
* Drop duplicates in multiple columns

Question 21

What is preprocessing?

Answer 21

Preprocessing is often used as an umbrella term to define all the operations that are performed prior to start the
analysis.
There is :
* General purpose preprocessing
* Preprocessing for diagnostic analytics

Question 22

We are now moving onto the data preprocessing section

Why is preprocessing needed ?

Answer 22

The data quality strongly influences the analysis quality (Garbage In => Garbage Out)
Preprocessing brings solutions to improve data quality in our dataset.

Question 23

How is data quality measured?

Answer 23

There are 4 measures of data quality:
* Validity
* Accuracy
* Consistency
* Uniformity
* Completeness

Question 24

What is validity ?

Answer 24

Validity is the degree to which the data conform to defined
business rules or constraints.

Question 25

What is accuracy ?

Answer 25

Accuracy: The degree to which the data is close to the true values.

Note! This is different from precision.

Precision: How consistent results are when measurements are repeated.

Data might be valid (i.e. respecting validity constraints) but not accurate.

Question 26

What is consistency?

Answer 26

Consistency: The degree to which the data is consistent,
within the same data set or across multiple data sets.

Question 27

What is uniformity?

Answer 27

Uniformity: The degree to which the data is specified using the same unit of measure.
In terms of probability distributions, uniformity can also refer to the data having a uniform distribution.
We consider uniformity in terms of measurement units (1st
example)

Question 28

What is completeness?

Answer 28

Completeness: The degree to which all required measures are known.
Several reasons could exist for incomplete data:
* Human-intensive workflow
* Multiple data sources
* Optional data

Question 29

What are some contraints of data validity ?

Answer 29

Some constraints are:
* Data-Type Constraints: Values in a particular column must be of a particular datatype, e.g., boolean, numeric, date, etc.
* Range Constraints: Numbers or dates should fall within a certain range.
* Mandatory Constraints: Certain columns cannot be empty.
* Unique Constraints: A field, or a combination of fields, must be unique across a dataset.
* Set-Membership constraints: Values of a column come from a set of discrete values, e.g. enum values
* Foreign-key constraints: As in relational databases, a foreign key column can’t have a value that does not exist in the referenced primary key
* Cross-field validation: Certain conditions that span across multiple fields must hold. For example, a patient’s date of discharge from the hospital cannot be earlier than the date of admission.
* Regular expression patterns: text fields that have to be in a
certain pattern. For example, phone numbers may be required to have the pattern (999) 999–9999.

Question 30

What is missing data ?

Answer 30

• Structurally missing data
• Missing completely at random (MCAR)
• Missing at random (MAR)
• Missing not at random

Question 31

What is structurally missing data?

And how can you deal with it ?

Answer 31

Structurally missing data is data that is missing for a logical reason.
Solutions:
* Excluding from any analysis the variables with the structurally missing values.

• If we can logically deduce that the correct value, this value should be used in place of the missing values in our analysis (imputation)

imputation is the assignment of a value to something by inference from the value of the products or processes to which it contributes

Question 32

What is missing data completely at random ?

MCAR = missing completely at random

Answer 32

It is relatively easy to check the assumption that data is missing completely at random. If you can predict which units have missing data (e.g., using common sense, regression, or some other method), then the data is not MCAR.
A more formal way of testing is to use Little’s MCAR test.
To solve these analyses using only observations that have complete data (provided we have enough of such observations) should be used.

N.B. The MCAR assumption is rarely a good assumption. It is only likely to be true in situations where the data is missing due to some truly random phenomena (e.g., if people were
randomly asked 10 of 15 questions in a questionnaire).

Question 33

What is Missing (data) at random (MAR) ?

Answer 33

In the case of missing completely at random, the assumption was that there was no pattern. An alternative assumption, known somewhat confusingly as missing at random (MAR), instead assumes that we can predict the value that is missing based on the other data.
Solution: When data is missing at random, it means that we
need to either use an advanced imputation method, such as
multiple imputation, or an analysis method specifically designed for missing at random data.

Question 34

What is Missing (data) not at random?

Answer 34

Solution: When data is missing not at random, it means that we cannot use any of the standard methods for dealing
with missing data (e.g., imputation, or algorithms specifically designed for missing values). If the missing data is missing not at random, any standard calculations give the wrong answer.

Question 35

What is diagnostic analysis?

Answer 35

Diagnostic analytics aims to look at a phenomenon and understand the reasons behind it and the relationships between the observed variables.
In this course, it is focused on analysing the relationships among variables:
* Correlation analysis (~ Is there a relationship between a pair of variables)
* **Regression analysis **(~ Is there a relationship between one (dependent) variable and one or more (independent) variables)

Question 36

We are now talking about diagnostic analytics

What is feature engineering and why is it applied?

Diagnostic analytics aims to look at a phenomenon and understand “why did it happen”?

Answer 36

feature engineering is *transforming and combining *existing data to provide new information and extra more information from the dataset.
This is used because the raw data is sometimes not sufficient to provide all the necessary information.

Question 37

How can you create more data?

Answer 37

You can create more data through:
* Feature extraction
* Feature combination

Question 38

How can you transform existing data?

Answer 38

Depending on whether the data is missing or whther it has to be scaled, we need to use different techniques.
For Missing data :
* Numerical imputation
* Categorical imputation
For Scaling:
* Log-transform
* Normalization
* Standardization
Otherwise, we can work with outlier removal.

Question 39

What is numerical imputation?

Answer 39

numerical imputation is about “Filling the gaps” = Transforming a variable to remove missing values
Imputation techniques involve:
* Replacement with statistically ininfluent values (mean,median)
* Replacement with null values
* Replacement with model-generated values
* Exclusion of missing values

Question 40

What is categorical imputation?

Answer 40

It is also about “Filling the gaps” = Transforming a variable to
remove missing values
A technique is
Hot-deck imputation: Replace missing value with most similar value from existing dataset

Question 41

What is log-transformation ?

Answer 41

It is taking the logarithm of a given feature.
It allows to turn a skewed-distribution into a more or less
normal one
Useful feature transformation technique for tools that makes
an assumption of normality

Question 42

What is standardisation ?

Answer 42

Standardising is rescaling a dataset so that its mean is 0 and its standard deviation is 1
No specific upper or lower bound for the maximum and
minimum values.

Question 43

What is normalisation?

Answer 43

It is rescaling a variable so that the range of the variable is [0,1]

Question 44

What are outliers ?

Answer 44

Outliers are values that are situated in the tails of the distribution, outside the “normal” range for the values.
Can be caused by different reasons:
* Measurement errors
* Statistically relevant anomalies (rare events)
* Missing values
Can be tackled by
* Removal
* Imputation
* Capping

Question 45

How does outlier removal work ?

Answer 45

The outlier removal process goes through a few steps:
1. Visual inspection/EDA
2. Human-in-the-loop procedure to assess validity of outlier
3. Outlier/anomaly removal:
A. Via training of a dedicated algorithm
B. Via human inspection

Question 46

What is feature extraction ?

Answer 46

Feature extration is the Process of creating (extracting) new variables from the existing one.
Includes:
* Creating derived variables (e.g. aggregations, time since)
* Discretizing existing variables
* Encoding variables in different formats (e.g. one-hot)
* Applying transformations on existing variables (logarithm, exponentiation, standardization, normalization)

Question 47

What is feature combination?

Answer 47

Process of creating (extracting) new variables by combining existing ones
Includes:
* Applying operations among variables (sum, counts, ratios) on numerical variables
* Polynomial/spline/trigonomet rical combination
* Combining categorical variables (e.g. outer product)

Question 48

What is causation ?

Answer 48

when one thing (a cause) causes another thing to happen (an effect)

Question 49

What is correlation ?

Answer 49

when two or more things appear to be related.

Correlation doesn’t always mean causation!

Question 50

What is the difference between a correlation and an experimental research ?

Answer 50

Correlational (C) and experimental (E) research both use quantitative methods to investigate relationships between variables. But there are important differences in data collection methods and the types of conclusions you can draw. They differ in
* Purpose:
* C - used to test strength of association between variables
* E - used to test cause-and-effect relationships between variables
* Variables:
* C - Variables are only observed with no manipulation or intervention by researchers
* E - independent variable is manipulated and a dependent variable is observed
* Control:
* C - limited control is used, so other variables may play
a role in the relationship
* E - extraneous variables are controlled so that they
can’t impact your variables of interest
* Validity:
* C - high external validity: you can confidently generalize
your conclusions to other populations or settings
* E - high internal validity: you can confidently draw
conclusions about causation

Question 51

What is correlation analysis ?

Answer 51

Using a correlation analysis, you can summarize the relationship between variables into a correlation coefficient: a single number that describes the strength and direction of the relationship between variables.
With this number, you’ll quantify the degree of the relationship between variables.
The Pearson product-moment correlation coefficient, also known as Pearson’s r, is commonly used for assessing a linear relationship between two quantitative variables.
Correlation coefficients are usually found for two variables at
a time, but you can use a multiple correlation coefficient for three or more variables.

Question 52

What is regression analysis?

Answer 52

Simple linear regression is a parametric test, meaning that it makes certain assumptions about the data. These assumptions are:
Homogeneity of variance (homoscedasticity): the size of the error in our prediction doesn’t change significantly across the values of the independent variable.
Independence of observations: the observations in the dataset were collected using statistically valid sampling methods, and there are no hidden relationships among observations.
Normality: The data follows a normal distribution.
Simple linear regression is a parametric test (with all the limitations of statistical testing):
* Statistical power/sample size
* Dependency on assumptions (if assumptions are violated, conclusions are unreliable)
When all the assumptions are verified, conclusions can be drawn only on the observed data sample, not extrapolated to the entire population.

The formula for a simple linear regression is:
𝑦 = 𝛽0 + 𝛽”𝑋 + 𝜖

Question 53

How can you characterise distributions ?

Answer 53

Several statistics can be computed about a probability
distribution:
* Mean
* Mode
* Median
* Standard deviation/Variance
* Skewness
* Kurtosis

Question 54

What are the steps for statistical testing ?

Answer 54

1. Experimental design and Hypothesis definition (Null hypothesis and alternative hypothesis)
2. Data collection (Probability sampling, Selective sampling, Sample size determination)
3. Data summarization (Central tendency measure, Variability measure, Test statistic)
4. Hypothesis testing
5. Result interpretation (One-sample test, Paired/Unpaired samples test (Two-sample), One-tailed/Two-tailed test (Directional vs Undirectional), Correlation-testing, Regression-testing)