Complexity/Selection/Regularization

Question 1

What is the bias-variance decomposition?

Answer 1

E[(Y - f̂_D(x))^2] = (Bias^2) + (Variance)

Question 2

What does high bias in a model indicate?

Answer 2

A bias towards a particular kind of solution (e.g., linear model), also known as inductive bias.

Question 3

What does high variance in a model indicate?

Answer 3

: The estimated model changes significantly when trained on different data sets, indicating overfitting.

Question 4

What is the VC dimension?

Answer 4

The maximum number of points that can be correctly classified by at least one member of a set of classifiers.

Question 5

What is the VC dimension of a linear classifier on R^p?

Answer 5

VC = p + 1

Question 6

How are Degrees of Freedom (DF) defined for an estimate ŷ = f̂(X)?

Answer 6

Back: df(ŷ) = (1/σ^2) Σ cov(ŷ_i, y_i) = (1/σ^2) tr(cov(ŷ, y))

Question 7

What is an intuition/interpretation of degrees of freedom in model complexity defined for an estimate ŷ = f̂(X)??

Answer 7

Degrees of freedom in model complexity represent the number of independent parameters that can be adjusted to fit the data, influencing the model’s flexibility.

Question 8

Relationship between complexity, bias, variance and total error

Answer 8

The higher the model complexity, the lower the bias, the higher the variance and total error does a convex in between.

Question 9

What does a good bias require?

Answer 9

Domain knowledge.

Question 10

What is the relationship between degrees of freedom, number of samples, number of features, and λλ in ridge regression?

Answer 10

In ridge regression, the degrees of freedom are influenced by the number of features and the penalty parameter λλ. As λλ increases, the degrees of freedom decrease, reducing model complexity and helping to prevent overfitting, especially when the number of samples is limited relative to the number of features.

Question 11

What is the PRESS statistic?

Answer 11

Predicted Residual Error Sum of Squares: PRESS = Σ(y_i - ŷ_-i)^2, where ŷ_-i is the prediction for the i-th sample when the model is estimated on all but the i-th sample.

Question 12

Higher model complexity effect on performance? What to do?

Answer 12

It will always lead to better fit on training data but not on test data therefore we need to select a model by estimating its performance for train and validation/test data.

Question 13

What is a method for validation?

Answer 13

Cross validation: estimate generalization error on different train/test splits.

Question 14

What is Leave-one-out Cross-Validation (LOO-CV)?

Answer 14

A method where the model is trained on all but one sample and tested on the left-out sample, repeated for all samples. The average prediction error is reported.

Question 15

What is k-fold Cross-Validation?

Answer 15

A method where data is split into k subsets, the model is trained on k-1 subsets and tested on the remaining one, repeated k times. Often k=5 or k=10 is used in practice.

Question 16

What is the relationship between expected prediction error and expected training error?

Answer 16

expected pred error = expected train error + constant (>0) * DF(model)

Question 17

What is a better selection criteria?

Answer 17

training error + complexity penalty. The larger the complexity (which tends to increase train error and lower generalization), the larger the penalty to keep it down!

Question 18

If two models fit the data equally well? which to select?

Answer 18

One with less complexity!

Question 19

What is the problem with cross validation?

Answer 19

If we test too many models, we will reach overfitting..

Question 20

What is the Bayes factor in model selection?

Answer 20

The ratio of marginal likelihoods: pr(x|m_i) / pr(x|m_j)

Question 21

What is the Bayes Information Criterion (BIC)?

Answer 21

BIC(x;m) = -2 log pr(x|θ̂,m) + p log(n), where θ̂ is the maximum likelihood estimate, p is the number of parameters, and n is the number of samples.

Question 22

What is the Fisher Information Approximation (FIA)?

Answer 22

FIA(x;m) = -log pr(x|θ̂,m) + (p/2)log(n/2π) + log C_m, where C_m is the geometric complexity

Question 23

What is the objective of l_k-penalized regression?

Answer 23

Minimize ω(θ) + λ||θ||_k^k, where ω(θ) is the loss function and ||θ||_k is the l_k norm of the parameters.

Question 24

What is the objective of l_k-penalized regression?

Answer 24

As the number of parameters increases past the interpolation threshold, test error first increases then decreases again, forming a double descent curve.

Question 25

What is implicit regularization in the context of gradient descent?

Answer 25

The gradient descent optimization process itself can act as a regularizer, especially in deep neural networks.

Question 25

How can increasing the number of features reduce model complexity?

Answer 25

When using a minimum l_2-norm estimator, increasing features can lead to implicit regularization.

Question 26

Why might overparameterized models be beneficial for complex data?

Answer 26

They can capture complex patterns while still benefiting from regularization, which is part of the success behind deep learning.

Question 27

How does the l_1 norm penalty (Lasso) differ from the l_2 norm penalty (Ridge) in regularization?

Answer 27

L_1 tends to produce sparse solutions (some coefficients exactly zero), while l_2 shrinks all coefficients but rarely to exactly zero.

Question 28

What is the “No Free Lunch” theorem in machine learning?

Answer 28

There is no single model that works best for every problem; the best model depends on the specific data and domain.

Question 29

What is Occam’s razor in the context of model selection?

Answer 29

If two models fit the data equally well, we should select the simpler one.

Question 30

How does the AIC (Akaike Information Criterion) differ from BIC?

Answer 30

AIC uses a fixed penalty of 2p, while BIC uses p log(n), making BIC more conservative for large n.

Question 31

What is the marginal likelihood in Bayesian model selection?

Answer 31

pr(x|m) = ∫ pr(x|θ,m)pr(θ|m)dθ, integrating over all possible parameter values.

Question 32

How can gradient descent act as an implicit regularizer?

Answer 32

It can bias the solution towards certain regions of the parameter space, often simpler or smoother solutions.

Question 33

What is the geometric interpretation of l_1 and l_2 regularization?

Answer 33

L_2 forms circular (2D) or spherical (higher dimensions) constraint regions, while l_1 forms diamond-shaped regions with corners on the axes.

Question 34

How does the effective degrees of freedom change as the ridge penalty λ increases?

Answer 34

The effective degrees of freedom decrease as λ increases, indicating a simpler model.