Module 2: Chapter 1 - Tools & Techniques

Question 1

What are the four advantages of using machine learning techniques over traditional econometric methods?

Answer 1

1) Handling Big Data
2) Handling non-linearity
3) Reducing dimensionality
4) Handling missing data

Question 2

What is the most crucial difference between classic econometric models and machine learning models when it comes about its approach understanding data?

Answer 2

Standard econometric models assume that the data-generating process can be approximated through a model and parameters are tested for statistical significance.
ML techniques make no assumptions on the data generating process

Question 3

What is regularization in the context of ML techniques?

Answer 3

With regularization, the number of parameters is kept low by introducing a penalty when parameters are included that do not significantly improve the prediction power of an ML algorithm

Question 4

What is the crucial philosophy behind ML techniques?

Answer 4

Model selection and fine-tuning of parameters to produce reliable predictions out-of-sample

Question 5

What are ML tests focusing on?

Answer 5

1) Out-of-sample prediction accuracy
2) Bias-variance trade-off

Question 6

What is the bias-variance trade-off?

Answer 6

The bias–variance tradeoff is a central problem in supervised learning. Ideally, one wants to choose a model that both accurately captures the regularities in its training data, but also generalizes well to unseen data. Unfortunately, it is typically impossible to do both simultaneously. High-variance learning methods may be able to represent their training set well but are at risk of overfitting to noisy or unrepresentative training data. In contrast, algorithms with high bias typically produce simpler models that may fail to capture important regularities (i.e. underfit) in the data.

Question 7

Translate these statistical terms into ML parlance
- Data point
- Dependent variable
- Independent variable
- Estimation
- Estimator

Answer 7

ML parlance:
- Example, instance
- Output, outcome, label, target, response var., ground truth
- Feature, signal, input, attribute
- Training, learning
- Learner (classifier), algorithm

Question 8

What are the four machine-learning methodologies?

Answer 8

1) Unsupervised learning
2) Supervised learning
3) Semi-supervised learning
4) Reinforcement learning

Question 9

Provide an overview behind the logic of supervised learning and used data structures

Answer 9

Supervised learning is about prediction problems (e.g. oil price next week) or classification problems (e.g. identify an animal on a picture)

For the estimation of the model we use a vector of attributes and associated output/labels.

Can be used with time-series or cross-sectional data.

Example: credit risk

Question 10

Provide an overview behind the logic of semi-supervised learning and used data structures

Answer 10

The goal of semi-supervised learning is the same as for supervised learning (prediction/classification).

The difference to supervised learning is that only a part of the training data is labeled. The unlabeled part is used to:
- find patterns in explanatory variables
- fit unlabeled data with pseudo-labels based on the algorithm trained on the labeled data

Question 11

Provide an overview behind the logic of unsupervised learning and used data structures

Answer 11

The goal of unsupervised learning is to recognize patterns in data. We have a vector of attributes but no output/classification to predict. Hence, prediction is not a goal of unsupervised learning.

Problems to solve:
- Clustering of data
- Find small number of factors that explain the data

Goals:
- Characterize datasets
- Learn data structures

Use cases:
- Anomaly detection
- Grouping of data (e.g. stocks based on some features)
- Dimensionality reduction

Question 12

Provide an overview behind the logic of reinforcement learning and used data structures

Answer 12

The goal is to make a series of decisions to achieve an objective. The environment can be static or dynamic (changing rules, adapting opponents, etc.). Final result is a recommended action (not a prediction or classification or cluster)

Feedback is provided through “rewards” (or sanctions if negative) that indicate that a certain type of behavior is desired but no specific instructions are provided (trial-and-error approach).

Question 13

Provide an overview behind the logic of parametric vs. non-parametric methods

Answer 13

Parametric methods: Modeler makes an assumption about the functional relationship between features and outputs/labels (mapping)

Non-parametric methods: no assumption on functional relationship between features and outputs/labels. Problem: large data sets are needed

Question 14

What is the purpose of EDA?

Answer 14

EDA = Exploratory Data Analysis
Collecting, cleaning, visualizing, and analyzing data

Collecting = data gathering
Cleaning = correct errors, remove duplicates, etc.
Visualization = Explore relationships between attributes and labels/outcomes

Question 15

Explain the differences between:
1) Structured, semi-structured, and unstructured data
2) Numerical and Categorical data
3) Longitudinal and cross-sectional data
4) Textual and other data

Answer 15

Structured data = tabular form
Unstructured data = data that is not structured according to a preset data model
Semi-structured data = Mixture of structured/unstructured data (e.g. emails, csv files, html files)

Numerical data = continuous/discrete data
Categorical data = nominal/ordinal data

Longitudinal data = time-series data (repeated measurements) of the same entity over time. Can also be based on geography or network relations
Cross-sectional data = snapshot (single point-in-time) of independent entities

Other data = audio, video, photographs, maps, drawings, etc.

Question 16

What is the difference between interval and ratio data?

Answer 16

Interval data = clear ordering with consistent and meaningful intervals but no true zero point

Ratio data = interval data with true zero point

Zero point = absence of a value

Question 17

Which five factors necessitate data cleaning?

Answer 17

1) Inconsistent recording
(recordings must be done in the same way)
2) Unwanted observations
(observations collected that should not be in our sample)
3) Duplicate observations
4) Outlier (treatment)
(for some analysis purposes, we ant to keep outliers, e.g. fraud detection)
5) Missing data
(can be structurally caused, i.e. impossible to get certain data or recording errors. Important to understand the reason why data is missing. Informative missingness exists when missingness is not random and can lead to bias. Imputation or estimation can be used to fill in missing data)

Question 18

When is a distribution of data not symmetric?

Answer 18

When the probability of data occurring is not equal left and right to the mean, then the data sample is skewed with a longer tail on one side.
Negative skewed = left tail is longer

Question 19

When are Quantile-versus-Quantile (QQ) Plots used?

Answer 19

QQ plots are used to verify that data is distributed according to an assumed data distribution. Plotting real quantile distribution (y-axis) vs. the theoretical distribution (x-axis). Typically used to check if data is normally distributed

Question 20

What is a fat tail distribution?

Answer 20

Tails of a normal distribution where abnormal events are rare, but not completely improbable

Question 21

Which visualization diagram is typically used to detect outliers?

Answer 21

Box-and-Whiskers Plot

Question 22

What visualization diagram is typically used to identify relationships between attributes?

Answer 22

Scatterplots

Question 23

What is Feature Extraction / Feature Engineering?

Answer 23

Translate features collected in the dataset to features used for analysis.

Qualitative data has to be transformed to numerical information (encoding). E.g. categorical variables used for qualitative information (e.g. geographic region).

Categorical features are represented with dummy variables (binary) because there is no meaningful interpretation between the value differences (e.g. 1- single, 2-married) as there is no natural ordering.

Question 24

How is the process of creating dummy variables from qualitative data called?

Answer 24

One-hot encoding / binarization

Question 25

What is the Dummy Variable Trap?

Answer 25

Multicollinearity induced in regression models fail as dummy variables become perfectly correlated with each other.

Consequence: inaccurate calculations of regression coefficients and standard errors

Question 26

What can be done against the Dummy Variable Trap?

Answer 26

1) Omitting one dummy variable
2) Setting the constant term (bias) of the equation to zero

Question 27

If there is natural ordering in a categorical variable, how is categorized?

Answer 27

It is an ordinal variable

Question 28

Why is data scaling needed?

Answer 28

Many machine-learning approaches require all the variables to be measured on the same scale; otherwise, the technique will not be able to determine the parameters appropriately and the results will be dominated by the feature with the largest magnitude

Question 29

What two data scaling techniques are used?

Answer 29

1) Standardization
Effect: mean of 0 and standard deviation of 1
2) Normalization (min-max transformation)
Effect: variable is bounded between 0 and 1

Standardization > Normalization
–> presence of outliers because normalization would remove much of the distance between the data points

Question 30

What are the three reasons for data scaling?

Answer 30

(1) Numerical stability of the learning algorithm. The difference in the scale of the variables could cause overflow/underflow easily

(2) Ease of interpretation of the model parameter estimates. If the scales of the variables differ significantly, so do those of the parameter estimates and it would make evaluation of the importance of each estimate difficult

(3) Determining whether the out-of-sample prediction is within the range of the training data. If the out-of-sample data corresponds to a normalized value greater than 1 or smaller than 0, that means the prediction is an extrapolation that may correspond to a large prediction error.

Underflow: Underflow is a type of rounding error that can be extremely damaging. When integers near zero are rounded to zero, underflow occurs. When the argument is zero instead of a small positive number, many functions act qualitatively differently.

Overflow: Overflow is another highly harmful type of numerical error. When numbers of enormous magnitude are approximated as -\infty or \infty , overflow occurs. These infinite numbers are frequently converted to NaN values with more mathematics.

Question 31

What is the rule of thumb to determine if a dataset is highly skewed?

Answer 31

If the ratio of the highest value to the lowest value is larger than 10 (in absolute value)

Question 32

What is a common practice to to transform highly skewed data? What are alternatives?

Answer 32

Transforming via the natural logarithm

Alternatives:
- Square root
- Inverse transformations

Question 33

What is the most commonly used dimensionality reduction technique?

Answer 33

Principal Component Analysis

Question 34

What is the logic behind PCA?

Answer 34

Find linear combinations of the original predictors that summarize most of the variability of the data.

Linear combinations = Principal Components

PC1 = linear combination of the original predictors that captures the most variability in the data among all possible linear combinations

PC2 = linear combination of the predictors that captures the most variability that is not already explained by the first PC

This second PC is constructed to be orthogonal to the first PC, that is, the two PCs are uncorrelated

Question 35

When is PCA most useful?

Answer 35

With highly correlated data and data experiencing multicollinearity.

Improves numerical stability of models that require low correlation among the predictors

Question 36

Why is it important to scale data before using PCA?

Answer 36

PCA seeks to find linear combinations to explain the variability in the predictors without any understanding of the predictors’ measurement scale or their distribution (e.g., if they are skewed)

PCA will first summarize the predictors with more variation, that is those with the largest magnitude

Question 37

How can we find an appropriate number of Principle Components?

Answer 37

Plot the variance explained per PC and check where we see a drop (elbow)

Question 38

Why should we train the model / algorithm only on a part of the available data?

Answer 38

The analyst wants the model to be able to generalize well to data that have not been used to estimate it

This leads to a distinction between in-sample (model estimation) and out-of-sample (sometimes known as a hold-out sample) parts of the data.

Question 39

Why is it especially in the case of ML important to have a hold-out sample?

Answer 39

There is typically little economic or financial intuition behind the modeling assumptions and the risk of choosing a complex model that accurately fits the dataset at hand but does not generalize well to unseen data is high

Question 40

What are the two common problems when a model is trained on a dataset?

Answer 40

Overfitting
Estimation of a model that is too complex and captures the noise in the dataset at hand rather than the true nature of relationships between the features and the output(s)

Underfitting
Significant patterns in data are not captured by the model

Question 41

Once a dataset is ready to be used for model estimation, in which parts does the data have to be divided?

Answer 41

(1) Training set
–> estimate model parameters

(2) Validation set
–> select between competing models

(3) Test set
–> determine the final chosen model’s effectiveness

Question 42

What is the rule-of-thumb for splitting portions of data?

Answer 42

2/3 for training
1/6 for validation
1/6 for testing

Question 43

Why should the training sample not be too small?

Answer 43

Introduction of bias in the parameter estimation

Question 44

Why should the validation sample not be too small?

Answer 44

Model evaluation can become too inaccurate so that it is hard to identify the best specification

Question 45

Which method to use to split cross-sectional data?

Answer 45

Choose randomly data based on the assigned proportions

Question 46

Which method to use to split time-series data?

Answer 46

Training on older data, testing and validation on more recent data

Question 47

What is cross-validation? Which technique is used?

Answer 47

Cross-validation involves combining the training and validation data into a single sample, with only the test data held back. Then the combined data are split into equally sized sub-samples, with the estimation being performed repeatedly and one of the sub-samples left out each time

k-fold cross validation technique

The technique splits the combined training and validation data available, n, into k samples, with the test data excluded from the combined sample

Question 48

What are the most used Python packages for ML estimation?

Answer 48

NumPy, SciPy, Pandas, Scikit-Learn, Tensorflow and Keras

Question 49

What is a popular package for ML estimation in R?

Answer 49

A popular package for building and evaluating machine learning models in R is caret (which is short for Classification And REgression Training)

Question 50

For each of the following terms used in classical statistics, provide the equivalent term in machine learning parlance:

Intercept
Slope
Explanatory variable
Dependent variable
In-sample period
Out-of-sample period

Answer 50

Intercept – bias
Slope – weight
Explanatory variable – feature
Dependent variable – output or label
In-sample period – training data
Out-of-sample period – test data

Question 51

For what kinds of problems would machine learning likely be more suitable than conventional econometric modeling?

Answer 51

Machine-learning techniques have advantages when applied to problems where there is little theory regarding the nature of a relationship or which features are relevant. It is used when the number of data points and the number of features are large (big data or wide data, as opposed to tall data where the number of predictors is strictly smaller than the number of observations). Machine learning might also be preferable when the relationships between features (and targets) are nonlinear.