ML-06 - Model and feature selection

Question 1

ML-06 - Model and feature selection

How do you use validation data to check if your problem is due to high bias vs. high variance?

Answer 1

(See image)

Question 2

ML-06 - Model and feature selection

Which area is high bias and which is high variance?

Answer 2

(See image)

Question 3

ML-06 - Model and feature selection

If you have high bias, is your model underfit/overfit?

Answer 3

Underfit

Question 4

ML-06 - Model and feature selection

If you have high variance, is your model underfit/overfit?

Answer 4

Overfit

Question 5

ML-06 - Model and feature selection

If your model is underfit, do you have high bias or variance?

Answer 5

High bias

Question 6

ML-06 - Model and feature selection

If your model is overfit, do you have high bias or variance?

Answer 6

High variance

Question 7

ML-06 - Model and feature selection

What is the first tool to try for overfitting problems?

Answer 7

Regularization

Question 8

ML-06 - Model and feature selection

What does regularization prevent?

Answer 8

Overfitting.

Question 9

ML-06 - Model and feature selection

Rescribe the bias/variance as a function of the regularization lamba parameter.

Answer 9

Question 10

ML-06 - Model and feature selection

Describe how the error vs. training set size looks for a situation with a good bias/variance trade-off.

Answer 10

(See image)

Question 11

ML-06 - Model and feature selection

Describe how the error vs. training set size looks for a situation high bias.

Answer 11

(See image)

Question 12

ML-06 - Model and feature selection

Describe how the error vs. training set size looks for a situation with high variance.

Answer 12

(See image)

Question 13

ML-06 - Model and feature selection

What should you try if you have high variance? (3)

Answer 13

• Get more data
• Smaller sets of features (or smaller NN)
• Try increasing regularization lambda

Question 14

ML-06 - Model and feature selection

What should you try if you have high bias? (3)

Answer 14

• Get more features
• Feature engineering, add polynomial features
• Try decreasing regularization lambda

Question 15

ML-06 - Model and feature selection

What are the 3 steps of the ML design guideline?

Answer 15

1) Start with a small model (baseline) that’s quick to implement.
2) Decide if more data or features will help (guided by learning curves)
3) Error analysis, manually examine samples where model made errors

Question 16

ML-06 - Model and feature selection

In the ML design guideline, how do you perform error analysis? (3)

Answer 16

• Look at data your model predicted wrongly
• Look for systematic trends in the type of errors made
• Hypothesize what cues (features) could have helped

Question 17

ML-06 - Model and feature selection

What is feature selection in ML?

Answer 17

Selecting which features are necessary, e.g. because they are redundant or not correlated with the labels (e.g. ID column).

Question 18

ML-06 - Model and feature selection

What is the goal of feature selection?

Answer 18

Find an optimal set of features that results in a “best model” for a problem.

Question 19

ML-06 - Model and feature selection

For feature selection, what are the 4 classes of feature selection mentioned?

Answer 19

• Filter methods
• Wrapper methods
• Embedded methods
• Dimensionality reduction methods

Question 20

ML-06 - Model and feature selection

Describe the gist of feature selection filter methods.

Answer 20

From the set of all features, find a subset based on some selection criteria, e.g. correlation coefficient.

Question 21

ML-06 - Model and feature selection

If you use a selection criteria to reduce a set of features to a subset, what type of feature selection method is that?

Answer 21

Filter methods

Question 22

ML-06 - Model and feature selection

What does the correlation coefficient measure?

Answer 22

Linear relationships between two or more features relative to each other or the output label.

Question 23

ML-06 - Model and feature selection

How would you use the correlation coefficient to filter out unnecessary features?

Answer 23

If two features are correlated, you don’t need both.

Question 24

ML-06 - Model and feature selection

Describe how wrapper methods work.

Answer 24

Search
- Loop until stopping
- Generate a feature set
- Test performance
- Select best performer

Question 25

ML-06 - Model and feature selection

What are the most commonly used wrapper methods? (4)

Answer 25

• Exhaustive feature selection
• Forward/sequential feature selection
• backward feature elimination
• recursive feature elimination (RFE)

Question 26

ML-06 - Model and feature selection

Describe exhaustive feature selection.

Answer 26

Try every combination of feature subsets and use the best one.

Question 27

ML-06 - Model and feature selection

Describe forward/sequential feature selection.

Answer 27

An iterative method starting with a single feature, then add more.

Question 28

ML-06 - Model and feature selection

Describe backward feature elimination

Answer 28

Start with all features, remove the worst performers.

Question 29

ML-06 - Model and feature selection

Describe recursive feature elimination (RFE)

Answer 29

is a type of back feature elimination method, which removes the weakest feature (or features) recursively one at a time until the specified number of features is reached.

Question 30

ML-06 - Model and feature selection

What is RFE short for?

Answer 30

recursive feature elimination

Question 31

ML-06 - Model and feature selection

What is RFECV short for?

Answer 31

recursive feature elimination with cross validation

Question 32

ML-06 - Model and feature selection

What are embedded methods?

Answer 32

An iterative method where in each iteration, a method extracts the features that contributed the most.

Question 33

ML-06 - Model and feature selection

Describe how regularization is used for feature selection.

Answer 33

L1 - Penalizes features to change their importance in the model. Promotes sparsity, i.e. feature is not used.

L2 - Maintains all variables, but assigns importance to improve performance.

Question 34

ML-06 - Model and feature selection

Describe how dimensionality reduction methods are used for feature selection.

Answer 34

Analyses input and projects it into a lower dimension, meaning it transforms the inputs into a new form.

Question 35

ML-06 - Model and feature selection

What are 3 common dimensionality reduction methods?

Answer 35

• PCA (Principal Component Analysis)
• SVD (Singular Value Decomposition)
• LDA (Linear Discriminant Analysis)

Question 36

ML-06 - Model and feature selection

What is PCA short for?

Answer 36

Principal Component Analysis

Question 37

What is SVD short for?

Answer 37

Singular Value Decomposition

Question 38

What is LDA short for?

Answer 38

Linear Discriminant Analysis