Week 5: Regularization: Lasso and Ridge regression

Question 1

What does explicit regularisation amount to?

Answer 1

Modifying the cost function. ????????????????

Question 2

What is the idea behind regularisation?

Answer 2

(parametric model) To keep the parameters theta.hat small, unless the data really convinces us otherwise.

I.e., if a modlel with small values of the parameters theta.hat fits the data almost as well as a model with larger paramater values, the one with small parameter values should be preferred.

Question 3

Is Ridge regression the L1 och L2 regularisation?

Answer 3

The L2.

Question 4

Which values can lambda in take on in Ridge regr.?

Answer 4

lambda is greater or equal to zero (non-neg.)

Question 5

Describe the trade-off that lambda (regularisation parameter) in Rigde regression expresses?

Answer 5

Choosing lambda is a trade-off between the original cost function (fitting the training data as well as possible) and the regularisation term (keeping the parameters theta.hat close to zero).

Question 6

What does it imply for the (linear regression) model if we set lambda equal to zero in both Ridge and LASSO?

Answer 6

That we will have the original least squares problem, as the penalty term disappears.

Question 7

What does it imply for the model (paramaters) if we let lambda go to infinity?

Answer 7

That we will force all paramaters to zero.

Question 8

How do we choose the value of lambda in Ridge?

Answer 8

By cross-validation. Then selecting the lambda resulting in the lowest variance (MSE?).

It is possible to derive a version of the normal equations for the minimization problem in Ridge (the cost function).

Question 9

What is the difference between the regularized cost functions of Ridge and LASSO?

Answer 9

That we always have a close-form solution to the Ridge regression NE if lambda > 0 (since then X^TX + n lambda I_p+1 is then invertible).

For LASSO, we have to use numerical optimization to solve the cost function.

Question 10

What does it mean that LASSO favours sparse solutions?

Answer 10

It means that it tends to choose a lambda resulting in only a few of the parameters to be non-zero, and the rest to be exactly zero. It does so by “switching off” some inputs (by setting the corresponding parameter theta_k to zero).

Question 11

What does it imply for LASSO that it favours sparse solutions?

Answer 11

That, by “switching off” some inputs (by setting the corresponding parameter theta_k to zero), it can be used as an input (or feature) selection method.

Question 12

In practice, looking at a graph, what will be the difference between LASSO and Ridge?

Answer 12

That the Ridge (L2) will fit a model with small (but non-zero) coefficient values, whereas LASSO (L1) will fit a model where some coefficients are exactly zero and the rest are small.

Question 13

What is the goal of regularization in terms of E_new and E_train?

Answer 13

That the ultimate goal of explicit regularization is to slightly decrease E_train, hoping that it will generate a higher E_new.

Question 14

Why do we, generally, more often use regularization methods instead of dim. reduction techniques for categorical variables?

Answer 14

Because categorical variables aren’t well suited for dimension reduction techniques.

Question 15

Why is, usually, the intercept theta_zero not included in the penalty term?

Answer 15

Because we want to discipline the model of variation around average y.bar, not y.bar itself.

Question 16

Do we have to deman and rescale the inputs before performing regularization?

Answer 16

Yes, always.

Question 17

Which algorithm (numerical opt.) do we use when solving the NE for LASSO?

Answer 17

The Least angle regression (LARS) algorithm.

Question 18

What is the shrinkage factor in Ridge?

Answer 18

(1 + lambda)^-1.

We divide the LS.theta.hat_j by this factor to get the RIDGE.theta.hat_j.

Question 19

What will be the value of the LASSO coefficients if the LS coefficients are smaller than lambda (to begin with)?

Answer 19

Exactly zero.

Question 20

What do we try to minimize with an elastic net?

Answer 20

min (theta) * (1/n) || x theta - y ||^2_2 + lambda [(1-alpha) || theta || ^2_2 + (alpha) || theta ||_1

Question 21

What will lasso do if we have a model with only two, correlated input variables?

Answer 21

Arbitrarily set one coefficient to zero and hence exclude one of the variables.

Question 22

What is the elastic net?

Answer 22

It is a mixture of the lasso and ridge regularization. It poses a penalty that combines the two individual penalties. EN has an additional parameter (alpha).

Question 23

Explain the procedure of finding the ridge or lasso regression coefficients in linear regression using the coefficient budget perspective.

Answer 23

1) Find LS-coefficients
2) Draw contour lines
3) Set up budget in shape of circle or diamond
4) Extend contour lines until a contour barely touches the line of the budget constraint. The first line that barely touches is the combination of coefficients for your regularization model.

Question 24

What is the shape of the coefficient budget for lasso and ridge respectively? For p = 2? For p > 2?

Answer 24

For lasso its a diamond (for p > 2 a rhomboid) and for ridge its a circle (for p > 2 a sphere).

Question 25

What does the regularization path tell us that the orthogonal design doesn’t (for lasso)?

Answer 25

That lasso quickly results in a sparse model. Also which variables that are particularly important for predicting our output in the lasso model.

Question 26

What is the idea of the one S.E. (std error) rule?

Answer 26

To use it with cross validation (CV) and to select the most parsimonious model whose prediction error is not much worse than the minimum CV error. The chosen model will still be inside of one S.E. of the error.

Question 27

Which of ridge and lasso is a little bit better at reducing variance for a model where most variables are useful?

Answer 27

Ridge.

Question 28

Which of ridge and lasso is a little bit better at reducing variance for a model with many useless variables?

Answer 28

Lasso, as it can exclude variables fully by letting their coefficients equal zero.

Question 29

Explain how regularization affects bias and variance of a common linear regression model.

Answer 29

Without reg: low bias using training data → high variance.

With reg: a little larger bias to start off with → lower variance.
Regularization with man

Question 30

Which values can lambda take on?

Answer 30

Zero to infinity.

Question 31

Regularization for SIMPLE linear regression (using sq. error loss) boils down to what for i) ridge and ii) lasso?

Answer 31

To minimizing 1) sum of squared residuals plus lambda times the slope coefficient squared, 2) sum of squared residuals plus lambda times the absolute value of the slope coefficient.

Question 32

What is lambda?

Answer 32

A tuning parameter/a weight for trade-off.

Question 33

By how much do the ridge (L2) coefficients (linear reg) decrease?

Answer 33

By (1+lambda)^(-1)

Question 34

Why is it called L1 and L2 respectively?

Answer 34

Because we’re using the L1 and L2 norms respectively when setting up the penalty function.

Question 35

Name the two methods for graphically showing regularization.

Answer 35

1) Orthogonal design and 2) coefficient budget perspective

Question 36

Regularization for SIMPLE linear regression (using sq. error loss) boils down to what for i) ridge and ii) lasso? Explain as simple as possible.

Answer 36

To minimizing 1) sum of squared residuals plus lambda times the slope coefficient squared, 2) sum of squared residuals plus lambda times the absolute value of the slope coefficient.

Question 37

Draw the orthogonal design with some examples.

Answer 37

Set up 45 degree dotted line = would correspond to LS betas on y and x. Rotate for ridge and parallel for lasso.

Question 38

For adapt. Lasso, what will be the weight (strength of penalty) of an input with LS coefficient of 0.9 and 0.1 respectively?

Answer 38

1/0.9 = 0.1 and 1/0.1 = 10.