Regression

Question 1

What are 3 possible non-linear basis functions?

Answer 1

-Polynomial: ϕj(x) = x^j
-Gaussian: ϕj(x) = exp(-(x-μj)²/2s²)
-Sigmoidal: ϕj(x) = σ((x-μj)/s), with σ(x) = 1/(1+exp(-x))

Question 2

What is the geometrical interpretation of the least squares error?

Answer 2

The least squares solution is such that y^ is the orthogonal projection of y onto the subspace on the linear span of the features.

Question 3

How to perform regression with multiple outputs?

Answer 3

1. Use different sets of basis functions for each component of y (i.e. consider it as multiple independant regression problem).
2. A better solution is to use the same set of basis functions to model all components. Then, the weights are contained in a matrix instead of a vector:
   W* = (ϕ.T * ϕ)^-1 * ϕ.T * Y
   => wk = (ϕ.T * ϕ)^-1 * ϕ.T * yk
   N.B.: It requires to compute only one pseudo-inverse matrix.

Question 4

What is the gradient descent algorithm?

Answer 4

Repeat until convergence;
{
wj = wj - α * dL(w)/dwj for j = 0, 1, …
}
(α is the learning rate)

Question 5

What is the stochastic gradient descent (SGD) algorithm?

Answer 5

Repeat until convergence;
epoch{
for i in [1,N]:
wj = wj - α * d/dwj[(y^(x(i);w) - y(i))] for j = 0, 1, …
}

Question 6

What is a compromise between standard gradient descent and stochastic gradient descent?

Answer 6

Mini-batch gradient descent consists in considering k«N examples when calculating the loss function (instead of N for standard GD and 1 for stochastic GD).

Question 7

What is early stopping?

Answer 7

It consists in stopping gradient descent before minimal error is reached to avoid overfitting. To do so, we monitor the validation loss after each epoch. We stop GD when it doesn’t decrease for k epochs (“k-patience”).

Question 8

What is the closed form solution of ridge regression?

Answer 8

w = (X.T * X + λ Id)^-1 * X.T * Y

Question 9

What is the constrained optimization formula for L2 regularization (used in ridge regression)?

Answer 9

Find w that minimizes (Y-Xw).T * (Y-Xw),
subject to w.T * w <= η.
N.B.: η ∝ 1/λ

Question 10

What is the constrained optimization formula for L1 regularization (used in lasso regression)?

Answer 10

Find w that minimizes (Y-Xw).T * (Y-Xw),
subject to |w| <= η.
N.B.: η ∝ 1/λ

Question 11

What is the main property of convex functions?

Answer 11

Every (local) minimum is a global minimum.