week 3 & 4

Question 1

RSE

Answer 1

Residual Standard Error

= √ 1/ n − 2 RSS

Question 2

R-squared

Answer 2

R2 =TSS − RSS/ TSS

= 1 −RSS/ TSS

Question 3

TSS

Answer 3

total sum of squares.

= Σni=1 (yi − y ̄)^2

Question 4

βj

Answer 4

average

Question 5

Xj

Answer 5

holding all other predictors fixed.

Question 6

Interpreting regression coefficients

Answer 6

• The ideal scenario is when the predictors are uncorrelated — a balanced design
• Correlations amongst predictors cause problems
• Claims of causality should be avoided for observational data.

Question 7

Estimation and Prediction for Multiple Regression

Answer 7

• Given estimates βˆ0, βˆ1, . . . βˆp, we can make predictions using the formula
• We estimate β0, β1, . . . , βp as the values that minimize the sum of squared residuals
This is done using standard statistical software. The values βˆ0, βˆ1, . . . , βˆp that minimize RSS are the multiple least squares regression coefficient estimates.

Question 8

F-statistic

Answer 8

F = [(TSS − RSS)/p] / [RSS/(n − p − 1)] ∼ Fp,n−p−1

Question 9

Forward selection

Answer 9

• Begin with the null model — a model that contains an
intercept but no predictors.
• Fit p simple linear regressions and add to the null model the variable that results in the lowest RSS.
• Add to that model the variable that results in the lowest RSS amongst all two-variable models.
• Continue until some stopping rule is satisfied, for example when all remaining variables have a p-value above some threshold.

Question 10

Backward selection

Answer 10

• Start with all variables in the model.
• Remove the variable with the largest p-value — that is, the variable that is the least statistically significant.
• The new (p − 1)-variable model is fit, and the variable with the largest p-value is removed.
• Continue until a stopping rule is reached. For instance, we may stop when all remaining variables have a significant p-value defined by some significance threshold.

Question 11

qualitative,

Answer 11

These are also called categorical predictors or factor

variables.

Question 12

Extensions of the Linear Model

Answer 12

Removing the additive assumption: interactions and

nonlinearity

Question 13

hierarchy principle:

Answer 13

If we include an interaction in a model, we should also

include the main effects, even if the p-values associated with their coefficients are not significant.

Question 14

alternatives to least squares

Answer 14

• Prediction Accuracy: especially when p > n, to control the variance.
• Model Interpretability: By removing irrelevant features — that is, by setting the corresponding coefficient estimates to zero

Question 15

Subset Selection.

Answer 15

We identify a subset of the p predictors that we believe to be related to the response. We then fit a model using least squares on the reduced set of variables.

Question 16

Shrinkage

Answer 16

We fit a model involving all p predictors, but the estimated coefficients are shrunken towards zero
relative to the least squares estimates. This shrinkage (also known as regularization) has the effect of reducing variance
and can also perform variable selection.

Question 17

Dimension Reduction.

Answer 17

We project the p predictors into a M-dimensional subspace, where M < p. This is achieved by
computing M different linear combinations, or projections, of the variables. Then these M projections are used as predictors to fit a linear regression model by least squares.

Question 18

Best subset and stepwise model selection procedures

Answer 18

1. Let M0 denote the null model, which contains no
   predictors. This model simply predicts the sample mean for each observation.
2. For k = 1, 2, . . . p:
   (a) Fit all p k models that contain exactly k predictors.
   (b) Pick the best among these p k models, and call it Mk. Here best is defined as having the smallest RSS, or equivalently largest R2

Question 19

deviance

Answer 19

negative two times the maximized log-likelihood— plays the role of RSS for a broader class of models.

Question 20

Forward Stepwise Selection

Answer 20

1. Let M0 denote the null model, which contains no
   predictors.
2. For k = 0, . . . , p − 1:
   2.1 Consider all p − k models that augment the predictors in Mk with one additional predictor.
   2.2 Choose the best among these p − k models, and call it Mk+1. Here best is defined as having smallest RSS or highest R2

Question 21

Backward Stepwise Selection

Answer 21

1. Let Mp denote the full model, which contains all p
   predictors.
2. For k = p, p − 1, . . . , 1:
   2.1 Consider all k models that contain all but one of the
   predictors in Mk, for a total of k − 1 predictors.
   2.2 Choose the best among these k models, and call it Mk−1. Here best is defined as having smallest RSS or highest R2

Question 22

Estimating test error: two approaches

Answer 22

We can indirectly estimate test error by making an
adjustment to the training error to account for the bias due to overfitting.
We can directly estimate the test error, using either a
validation set approach or a cross-validation approach, as discussed in previous lectures.