Linear Model Selection & Regularization

Question 1

Feature Selection for Linear Regression

Answer 1

1. Best subset selection, try combinations of all features (not feasible, just a theoretical optima). *Unless you have a really long time is ok
2. Forward Stepwise Selection, start with no var, keep adding one var at a time that maximizes fit. Then select the best of these models using CV error.
3. Backward Stepwise Selection - start with all vars, keep taking them away one by one. “” CV Error.
4. Regularization Techniques (Ridge, Lasso)
5. Feature Selection - Lasso even though is a regularization technique, does Feature selection

Question 2

Ways of selecting a model when you have tried a bunch of them

Answer 2

1. Use CV (Preferred)
2. Look at metrics such as AIC, BIC, etc or Adjusted Rsquared, which are goodness of fit metrics that pay a price for unnecessary variables. *Used these when computing power was in issue, nowadays use CV.

Question 3

Regularization Techniques for Linear Regression

Answer 3

1. Ridge - supply a regularization parameter that penalizes large coefficients (so better have a good one). Must be careful in selecting the regularization parameter. Also must center/scale variables prior to regularization.
2. Lasso - regularizes some coefficients to exactly zero so also performs feature selection. (preferred). Must also carefully select reg. parameter and scale/center vars.

Question 4

Disadvantage of Ridge Regression

Answer 4

Will include all variables P in the model, albeit will shrink coefficients of variables. This creates problems for model interpretation. Alternatively, Lasso will set some coefficients exactly to zero.

Question 5

Ridge vs. Lasso

Answer 5

Lasso is better when many of the vars really have no relation (zero coeff) with the outcome var.
Ridge will outperform if many of the vars infact are related to the outcome var.

Question 6

How to select tuning parameter

Answer 6

plot the K-fold CV Error against different values in the tuning parameter

Question 7

Dimension Reduction vs. Regularization

Answer 7

Regularization involves controlling the coefficients (with lasso, even setting them to zero which has the effect of reducing dimensions). Dimension Reduction transform the predictors into a smaller set of variables.

Question 8

Dimension Reduction

Answer 8

Transform dimensions into a smaller subset of linear combinations that substantially explain the variability of the all the original variables.

Question 9

Uses of PCA

Answer 9

Can be used for dimension reduction or unsupervised learning. Creates linear combinations of predictors that are uncorrelated w/eachother.

Question 10

Things to consider when doing PCA

Answer 10

center/scale prior to using PCA

Must select the # of Prin Components by plotting CV Error vs. # of components.

Question 11

PCA vs Partial Least Squares

Answer 11

PCA creates new vars that substantially explain the variability in the orignal vars.
PLS creates new vars that explain variability, but ALSO that are related to the response, kind of like a supervised PCA.
Like PCA, in PLS you must choose your # of features and center/scale. PLS IS NOT SUPERIOR TO PCA they are kind of a wash.

Question 12

What is high dimensionality

Answer 12

when p >= n. Linear and logistic regression doesn’t work well in thi sscenario.

Question 13

Interpreting models in high dimensionality

Answer 13

In this sceanrio, you cannot be confident that you have selected the “best” model, only that you have a “good model”. Also traditional measures of model fit dont dont apply (Rsq, P value, etc.)

Question 14

What gives you the most optimal subset of variables for a model (linear)

Answer 14

Best subset selection, available in the leaps library, regsubsets() function

Question 15

When do you want to use best subset selection

Answer 15

Best subset selection is computationally expensive, however when p < 10, then usually is ok to do best subset