Topic 1-4

Question 1

What is statistical learning?

Answer 1

Statistical learning refers to a set of approaches for estimating the relationship between a response variable Y and predictors X. This relationship is modeled as Y=f(X)+ϵ, where f(X) is the unknown function to be estimated, and ϵ is the error term representing noise.

Question 2

What are the names of
X and Y?

Answer 2

Y: Response variable (output or dependent variable)
X: Predictors (inputs, features, or independent variables)​.

Question 3

Pros/cons of the parametric and non-parametric methods:

Answer 3

Pros: Simple and interpretable; require fewer observations.
Cons: May fail if the assumed model is far from the true f
Non-parametric Methods:
Pros: Can model complex shapes of
f; more flexible.
Cons: Require more data; less interpretable​.

Question 4

For the best prediction performance, which of our tested statistical learning methods should we choose?

Answer 4

Select the method that achieves the lowest test MSE by balancing bias and variance​.

Question 5

What is the relationship between test performance and model flexibility

Answer 5

Test error decreases initially with flexibility (reducing bias) but increases beyond a certain point due to variance (overfitting)​.

Question 6

What is the RSE (residual standard error) useful for? And what is the formula?

Answer 6

For quickly seeing how much a prediction is usually off (the average standard error, measured in units of Y. So if RSE = 400, the average measurement is 400 units off.

RSE = Root(1/(n-2) * RSS)

Question 7

What are the 2 p values associated with multicolinearity?

Answer 7

The F-test and the T-test.
The F-test wether there is any non-0 parameter in the regression, if there is H0(all Bj is 0) is rejected
The T test give the amount of standard errors the parameter is from 0, if the parameter is a lot of std. error from 0, it likely is valuable to the response variable.

Question 8

Difference between Qualitative and Quantitative predictors

Answer 8

Qualitative = Race etc.
Quantitative = income

Question 9

With different coding schemes for Qualitative variables, what happens to the regression lines when interaction effects are implemented into the scheme?

Answer 9

They are no longer parallel to one another.

Question 10

What are assumptions of a simple linear regression?

Answer 10

Linearity: The relationship between
X and Y is linear.
Independence: Observations are independent.
Homoscedasticity: Constant variance of errors (ϵ).
Normality: Errors (ϵ) are normally distributed​.

Question 11

What are the five potential potential problems with a regression model (model diagnostics)?

Answer 11

• Non-linearity
• Non constant variance of error terms
• outliers
• High leverage points
• Collinearity

Question 12

What is collinearity

Answer 12

2 predictors are very closely related to one another, so its difficult to determine how they seperately affect the response

Question 13

Why is there a need for model diagnostics?

Answer 13

To see if any of the assumptions of the model are violated

Question 14

What are RSS, RSE, and Training MSE:

Answer 14

RSS (Residual Sum of Squares): Measures the total deviation of predicted values from actual values.
RSE (Residual Standard Error): An estimate of the standard deviation of ϵ, derived from RSS.
Training MSE: RSS normalized by the number of observations (adjusted for degrees of freedom). It is a measure of training error​.

Question 15

How to asses coeficient accuracy?

Answer 15

Standard errors (SE) of coefficients and their p-values indicate accuracy.

Question 16

How can you check if the model fitting is good?

Answer 16

RSS and RSE

Question 17

What are assumptions of a multiple linear regression?

Answer 17

Linearity: The relationship between
X and Y is linear.
Independence: Observations are independent.
Homoscedasticity: Constant variance of errors (ϵ).
Normality: Errors (ϵ) are normally distributed​.
No multicollinearity among predictors

Inclusion of all relevant predictors​

Question 18

What are solutions to the 5 problems a regression model might have?
- Non-linearity
- Non constant variance of error terms
- outliers
- High leverage points
- Collinearity

Answer 18

• Use nonlinear transformations of predictors
• Transform the response variable (y) to something nonlinear (log(Y))
• Check studentized residuals and remove plot
• Limit the values of X
• Use VIF to see the severity of collinearity, drop one of the predictors

Question 19

What are the assumptions of LDA?

Answer 19

The predictor variable is normally distributed under each response class

If there is more than one predictor, the predictors follow a multivariate normal distribution

All predictors have equal variance

Question 20

Why is the Bayes classifier the gold standard to compare other classifiers against?

Answer 20

Because it assumes perfect normality in the predictors, in the real world we do not know the distribution of our predictors. So the bayes classifer is purely theoretical

Question 21

What are the assumptions of Naive Bayes?

Answer 21

Within each class, the predictors are independent. This means that for each class you get a Function that is built up of functions for each predictor. Fk(x) = Fk1(x1) + Fk2(x2) + Fkp(xp)
k=class

Question 22

What is the main difference between LDA and QDA?

Answer 22

LDA assumes the predictor has a Class-specific mean and a shared variance. QDA assumes that the predictor has a class specific mean and a class specific variance.

Question 23

What are the performance measures for all classifiers?

Answer 23

Accuracy = (TN +TP) / (N + P)
Error rate = (FN +FP) / (N + P)
Sensitivity = TP / P
Specificity = TN / N

Question 24

What are the 2 extra performance classifiers for LDA and QDA? (Explain them in your head as well)

Answer 24

They are called the:
-ROC (receiver operating characteristics)
-AUC (Area under curve)

Both based on a curve Y-axis True positive rate (sensitivity = TP/P) and the false positive rate (1-specificity) .

Question 25

How to find the optimal K-value with KNN?

Answer 25

Cross validation

Question 26

What are the three CV methods?

Answer 26

• K-fold CV
• LeaveOneOut CV
• Validation Set approach

Question 27

What are the two Scenarios for Resampling Methods?

Answer 27

Model assessment: Estimating the test error rate when a test set is unavailable.
Model selection: Selecting the model with an appropriate level of flexibility​.

Question 28

List the pro’s and cons for each CV method

Answer 28

Validation Set:
Pros: Simple and quick.
Cons: High variance due to randomness in splitting data; training on fewer observations can lead to overestimation of the test error​.
LOOCV:
Pros: Uses nearly all the data for training; reduces bias.
Cons: Computationally intensive; requires refitting the model n times​.
K-Fold Cross Validation:
Pros: Balances bias and variance well; computationally efficient compared to LOOCV.
Cons: Still computationally expensive with very large datasets.