05_supervised learning methods

Question 1

What is a linear model?

Answer 1

they assume linearity in the underlying data

rather simple but convey many of the concepts utilized in other, more complex models

eg linear regression, linear classification

Question 2

What does a linear regression do)

Answer 2

find weights w0 and w1
so that the linear function f(x)=w1 º x + w0
with input x and output y
best fits the data containing ground-truth values y’

–> how can we learn w0 and w1 from data?

Question 3

How do linear regression models learn the weights for function f?

Answer 3

minimize square errors of prediction
with respect to ground truth
for each data point:

Least Square fitting

for data point j: [yj’ - f (wj, w0, w1) ] ^2

Question 4

How does least squares fitting work?

Answer 4

1) define a loss (objective) function that is the sum of the squared errors over all data points

2) find the best-fit model parameters by minimizing the loss function with respect to those two model parameters
(first derivative –> closed-form expression for best-fit w0 and w1)

least-squares + linear model function: the resulting minimum of the loss function is GLOBAL (bc of the combo)
–> the model learns immediately the best-possible solution

Question 5

When can linear functions be used as a classifier?

Answer 5

When the data is linearly separable (and only then!)

Question 6

How do linear functions work as a classifier?

Answer 6

1) define decision boundary
f(x,w) = wx = w0 + w1x1 + w2x2

such that class 1: f(x,w) ≥ 0
and class 0: f(x,w) < 0

2) we can define class assignments through a threshold function

Question 7

What is the perceptron learning rule?

Answer 7

weights are adjusted by a step size that is called the LEARNING RATE.
by iteratively running this algorithm over your training data multiple times, the weights can be learned so that the model performs properly

–> solution is learned iteratively

–> does not imply that the model is learning something useful (eg dataset might not be suitable)

Question 8

Why can linear models not often be applied?

Answer 8

they have low predictive capacity
–> can only be applied to data that is linearly distributed

Question 9

How can linear models be fit for more capacity?

Answer 9

the base function can be changed to a polynomial:

f(x) = w0 + w1x + w2x^2 etc
Summe wix^i

the resulting regression problem is still linear in the weights to be found, therefore the same properties apply:

we can compute the parameters wi to minimize the loss with a closed-form expression. this is by default the best possible solution

Question 10

What can be changed for a polynomial linear function model in order to get more capacity?

Answer 10

p (which says how many weights and x we have. have to find the best p)

–> when p is too low, the decision boundary looks like a constant, if it’s too high it can also be not a good fit (underfitting vs overfitting)

Question 11

What does Occam’s Razor mean?

Answer 11

a model with fewer parameters is to be preferred

Question 12

What is the goal of any regression model?

Answer 12

to minimize the loss over the data, ie prediction errors

Question 13

What is a way to prevent a model from overfitting?

Answer 13

regularize the loss based on the learned weights

L’(x,w) = mean loss over N samples + regularization term

Question 14

What it the L2-norm?

Answer 14

||w||2 2 = w * w

Question 15

What happens when you add a L2 regularization term to a polynomial model?

Answer 15

The regularization term takes the weights and adds it to the loss function

–> the loss function cannot go as low as it wants to

with increasing alpha, all coefficients wi drop in magnitude, leading to smoother fits

Question 16

What is L2 regularization also called?

Answer 16

ridge regression

Question 17

What is L1 also called?

Answer 17

LASSO regression

least absolute shrinkage and selection operator

Question 18

What is the L1-norm?

Answer 18

||w||1 = |w|

Question 19

What does a regularization term consist of?

Answer 19

alpha and L2/L1-norm

alpha: regularization parameter

Question 20

What happens when you add a L1 regularization term to a polynomial model?

Answer 20

while L2 regularization modulates all coefficients wi in the same way,
L1 regularization aims to set less meaningful coefficients to zero

–> performs feature selection

tries to bring as many of the coefficients to 0 as it can

Question 21

How can we get closest to the minimum loss with a L2 norm in a 2D loss space spanned by w1 and w2?

Answer 21

we can only reach points on a circle to find the point that puts us closest to the global minimum

this would be inefficient if the global minimum is inside of the circle

Question 22

How can we get closest to the minimum loss with a L1 norm in a 2D loss space spanned by w1 and w2?

Answer 22

we can only reach points on a triangle defined by the two weights

Question 23

Which type of regularization should we use? L1 or L2?

Answer 23

• L2 regularization prevents the model from overfitting by modulating the impact of its input features in a homogeneous way
• L1 regularization prevents the model from overfitting by focusing on those features which seem to be most important

both can be deployed in any machine learning model that minimizes a loss function

Question 24

What are pros for linear models? (4)

Answer 24

• easy to understand and implement; resource efficient even for large and sparse data sets
• least squares method always provides best-fit results if the data is appropriate
• good interpretability due to linear nature of the model
• easy to regularize

Question 25

What are cons for linear models? (2)

Answer 25

• limited flexibility: data distribution must be brought into a form that is linear (regression) or linearly separable (classification)
• susceptible to overfitting if not combined with regularizer

Question 26

What is especially important when working with nearest-neighbor models?

Answer 26

Scaling of the data!

bc we need the distance in the model

Question 27

What are nearest neighbor models?

Answer 27

are non-parametric and simply rely on distances between data points

distances can be defined as metrics

nearest neighbor methods utilize distances between datapoints for classification and regression tasks

Question 28

What is a common distance metric in nearest neighbor models?

Answer 28

Euclidean distance

∂ = √∂x1^2 + ∂x2^2

Question 29

What is k-nearest neighbor classification?

Answer 29

k-nearest neighbor (knn) classifiers predict class affiliation of an unseen data point based on majority voting of its k nearest neighbors in a seen data set with ground-truth labels

are not trained in the general sense: distance of each unseen data point from all seen data points is calculated

Question 30

What is the impact of hyperparameter k on knn-classification?

Answer 30

k has an impact on how well the model generalizes to unseen data:
we have to perform a hyperparameter search

Question 31

What can nearest neighbor model also be implemented as?

Answer 31

as regressors that are able to interpolate between and smoothen available data
(only have to be aware that this exists)

Question 32

How do we regularize nearest neighbor methods?

Answer 32

is done through varying its hyperparameter k

a low k is susceptible to small-scale variations and noise

a high k may miss local details

Question 33

What are pros for knn classification? (3)

Answer 33

• easy to understand, implement and results are highly interpretable
• non-parametric
• works reliably even with small data sets

Question 34

What are cons for knn classification? (2)

Answer 34

• calculating distances computationally intensive for large data sets
• performs poor on sparse data sets; prone to the curse of dimensionality

Question 35

What is the curse of dimensionality?

Answer 35

number of data points should grow exponentially with data dimensionality

if parameter space is insufficiently sampled, the model does not have enough data points for training properly –> if we have too many features

Question 36

What is a decision tree?

Answer 36

a decision tree is a rule-based structure for prediction of scalar output from (potentially) multi-dimensional input data

Question 37

what are tree properties?

Answer 37

hyperparameters:

• tree depth (how many layers do we have)
• number of leaves (outputs, how many classes do we have)

Question 38

What decision tree learning? How do they learn?

Answer 38

a greedy divide-and-conquer strategy is adopted to train decision trees on a training data set in a recursive fashion

1) identify the “most important feature” (greedy)
2) split the samples across this feature (divide)
3) if all samples of a branch are of the same class, create a leaf, unless number of leafs has been reached, and stop
4) if not all samples of a branch are of the same class, recursively apply algorithm again to that branch until maximum depth reached

Question 39

What is the “most important feature” in decision trees?

Answer 39

generally, it means the feature that makes the most difference to the classification of a single sample

there are different implementations of this definition eg utilizing information entropy or other useful measures

Question 40

Can single decision trees generalize well?

Answer 40

No

Single decision trees typically generalize only to some extent due to their limited depth and size;
they have low model capacity

Question 41

How can decision trees be optimized so they perform besser?

Answer 41

ensemble methods - group the trees, which increases their capacity

by combining a large number of decision trees and letting them make decisions in an averaged vote or majority vote, we increase their capacity

Question 42

What are random forests?

Answer 42

Trees in a random forest are shallower than other decision tree models. the trees therefore act as “weak learners” (intentionally) that perform badly by themselves.

however, combining a large number of weak learners performs much better than individual trees.

the intuition behind is that weak learners “on average” compensate for their individual shortcomings.

Question 43

What are gradient-boosted tree-based models?

Answer 43

random forests (decision tree ensembles) that are built successively in such a way that every newly created tree compensates for the shortcomings of the previous trees

gradient-boosting refers to the fact that new base learners (individual devision trees) are fitted to the model’s pseudo-residuals, based on the gradient of the loss of the ensemble

–> loss decreases with each added tree

Question 44

What tasks can gradient-boosted models do?

Answer 44

they are very successful in regression and classification tasks
and still represent state-of-the-art in traditional ML

common implementations:
- XGBoost
- LightGBM

Question 45

What are pros of tree-based models? (4)

Answer 45

• extremely versatile and robust
• can be trained on small amounts of data
• non-parametric
• interpretability: tree-based models are to compute “feature importances”

Question 46

What are cons of the tree-based models? (1)

Answer 46

• decision boundaries and regression predictions may be discrete instead of continuous