3 (QM) - Machine Learning

Question 1

What is machine learning (ML)?

Answer 1

The use of algorithms to make decisions by generalizing (or finding patterns) in a given data set. The goal is to use data to automate decision-making.

Question 2

Target Variable

Answer 2

The dependent variable (i.e. the “Y” variable). Can be continuous, categorical, or ordinal.

Question 3

Features

Answer 3

These are the independent variables (i.e. the “X” variables).

Question 4

Training data set

Answer 4

The sample data set used to fit the ML-model.

Question 5

Hyperparameter

Answer 5

The ML-model input specified by the researcher.

Question 6

Supervised Learning

Answer 6

A ML-algorithm uses labeled training data (inputs and outputs are identified) to model relationships in the data and achieve superior forecasting accuracy.

Question 7

Unsupervised Learning

Answer 7

An ML algorithm is not given labeled training data. Instead, the inputs (i.e. features) are provided without any conclusions around those inputs and the algorithm aims to determine the structure of the data.

Question 8

Deep Learning

Answer 8

An ML-algorithm that is used for complex tasks such as image recognition, natural language processing, and so on.

Question 9

What are the three major types of deep learning algorithms?

Answer 9

1. Reinforced learning algorithms (RL) - An ML model that learns from their own prediction errors. Learn from their errors to maximize a defined reward.
2. Neural Networks - Comprise an input layer, hidden layers (which process the input), and an output layer. The nodes in the hidden layer are called neurons, which comprise a summation operator (that calculates a weighted average) and an activation function (a nonlinear function).
3. Deep learning networks (DLN) - Neural networks with many hidden layers (20+) that are useful for pattern, speech, and image recognition.

Question 10

Overfitting

Do overfit models generalize well to new data?

Answer 10

An issue with supervised ML that results when a large number of features (i.e. independent variables) are included in the data sample. This results in an overly complex model that may have generalized random noise that improves in-sample forecasting, however, it decreases the accuracy of the model to forecast other out-of-sample data.

No, they do not generalize well to new data. This results in a low out-of-sample R-squared.

Question 11

Bias error

Answer 11

The in-sample error that results from models with a poor fit.

Question 12

Variance error

Answer 12

The out-of-sample error that results from overfitted models that do not generalize well.

Question 13

Base error

Answer 13

The residual errors due to random noise.

Question 14

Training sample
v.
Validation sample
v.
Test sample

What are these three data sets used for?

What are the types of errors associated with each data set?

Answer 14

These data sets are used to measure how well a model generalizes. All three datasets are nonoverlapping.

Training - Data set used to develop the model. In-sample prediction errors.

Validation - Data set used for tuning the model. Out-of-sample prediction errors.

Test - Data set used to evaluate the model using new data. Out-of-sample prediction errors.

Question 15

Learning curve

Answer 15

Curve that plots the accuracy rate (i.e. 1 - error rate) in the validation or text sample versus the size of the training sample.

Question 16

Accuracy Rate =

Answer 16

Accuracy Rate = 1 - Error Rate

Question 17

What two methods to data scientists use to mitigate the problem of overfitting?

Answer 17

1. Complexity Reduction - A penalty is imposed to exclude features that aren’t meaningfully contributing to out-of-sample prediction accuracy. The penalty value increases with the number of independent variables used by the model.
2. Cross Validation - A sampling technique that estimates out-of-sample error rates directly from the validation sample. This ensures the validation sample is both large and representative of the population, just like the training sample.

K-fold cross validation is a method of randomly dividing a dat aset into any number of parts

Question 18

What method is used to collect average in-sample and out-of-sample error rates?

Answer 18

K-fold cross validation. Randomly dividing a data set into any number of parts - “k”.

The training sample comprises k-1 parts with one part left for validation. Error rates are the measures for the model in each of the parts. This process is then repeated “K” times.

Question 19

Penalized regression

Answer 19

Supervised learning

An algorithm that reduces overfitting by imposing a penalty on or reducing the nonperforming features.

Question 20

Support vector machine (SVM)

Answer 20

Supervised learning

A linear classification algorithm that separates the data into one of two possible classifiers based on the model-defined hyperplane.

Question 21

K-nearest neighbor (KNN)

Answer 21

Supervised learning

An algorithm used to classify an observation based on nearness to the observations in the training sample.

Question 22

Classification and regression tree (CART)

Answer 22

Supervised learning

An algorithm is used for classifying categorical target variables when there are significant nonlinear relationships among variables.

Question 23

Ensemble learning

Answer 23

Supervised learning

An algorithm that combined predictions from multiple models, resulting in a lower average error rate.

Question 24

Random forest

Answer 24

Supervised learning

A variant of the classification and regression tree (CART) whereby a large number of classification tress are trained using data bagged from the same data set.

Question 25

Principal components analysis (PCA)

Answer 25

Unsupervised learning

An algortihm that summarized the information in a large number of correlated factors into a much smaller set of uncorrelated factors, called eigenvectors.

Question 26

K-means clustering

Answer 26

Unsupervised learning

An algorithm that partitions observations into “K” nonoverlapping clusters; a centroid is associated with each cluster.

Question 27

Hierarchical clustering

Answer 27

Unsupervised learning

An algorithm that builds a hierarchy of clusters without any predefined number of clusters.

Question 28

Hierarchical clustering

Agglomerative vs. divisive

Answer 28

method of building a hierarchy of clusters without any predefined number of clusters.

Agglomerative - Bottom-up. starts with one observation and its own cluster and adds similar observations to that group

Divisive - Top-down. Starts with one giant cluster and then partitions that cluster into smaller and smaller clusters.