10 Multicollinearity in MLR

Question 1

min|| 𝑿𝛽−𝑦2||
MLR solution, requires 𝑛 ≥ 𝑝

If 𝑿T𝑿 is not full rank:
* …
If the columns of 𝑿 are highly correlated: * …

Answer 1

No unique solution to normal equations.

Leads to unstable equation/plane
in the direction with little variability.

Question 2

Considerations in High Dimensions
While ..
Data sets containing more features than observations are often referred to a high-dimensional.
When the number of features 𝑝 is as large as, or larger than, the number of observations 𝑛, …
* It is too …
…. are particularly useful for performing regression in the high-dimensional setting.

Answer 2

𝑝 can be extremely large, the number of observations 𝑛 is often limited due to cost, sample availability, etc.

OLS should not be performed.

flexible and hence overfits the data.

Forward stepwise selection, ridge regression, lasso, and PCR

Question 3

Solutions to Multicollinearity
Subset selection
* …* (already discussed in the context of the linear regression)
Using derived input
* …
Coefficient shrinkage (regularization) => next class

Answer 3

best subset, backward, forward, stepwise selection of features

Principal component regression * Partial least squares

• Ridge regression
• Lasso (least absolute shrinkage and selection operator)..