Lecture 10 - Model Evaluation

Question 1

What does it mean when a model has a high bias?

Answer 1

The model does not match the training data closely enough to be useful.

Bias - Limited flexibility

Question 2

What does it mean if a model has a high variance?

Answer 2

It means that it matches the training data too closely.

Variance - sensitivity to specific sets

Question 3

What does it mean if we’re too “fit”?

Answer 3

If we are too “fit” then the model conforms too much to this one data set so we can’t generalize.

Question 4

What is the bias-variance trade off?

Answer 4

It is the effort to minimize two sources of error that prevents supervised learning algorithms from generalizing beyond their training set.

Question 5

What is the irreducible error?

Answer 5

The bias-variance decomposition is composed of tree terms: Bias, variance, and irreducible error.

error = bais + Variance + irreducible error

Question 6

How do you detect overfitting?

Answer 6

Use a separate set of holdout data. We spit the labelled data into two collections: training and evaluation

Question 7

What are the two important properties for detecting overfitting using holdout data?

Answer 7

The data were not used int he training so they cant have been memorized.

They have labels, so we can review models accuracy without labeling costs.

Question 8

What is cross-validation and why is it important?

Answer 8

It allows us to see how our model does, on average across a number of randomized trials.

This will tend toward the population average.

Question 9

Why do we need to be careful using train-test splits?

Answer 9

We don’t want to end up with all one class in training and not in evaluation.

Question 10

How do we make sure that we don’t end up with all one class in our training set?

Answer 10

Make several random splits.

We do this with k-fold cross validation where k can be 3, 5, 10 different splits.

Question 11

Describe underfitting?

Answer 11

It means our model has not captured the complexity present in our training data.

Question 12

You will see excellent performance on the training data and much worse performance on the test data if

Answer 12

The model is overfit

Question 13

If you see bad performance on both the train and the test sets

Answer 13

The model is underfit

Question 14

Training performance is almost always better than

Answer 14

test performance

Question 15

If you have overfit data?

Answer 15

Collect more data
Try a similer model
Apply regularization

Question 16

If you underfit the data?

Answer 16

Try more complex model

Try engineering additional features that may help explain

Question 17

Explain regularization in a sentence

Answer 17

Regularization is a way to penalize models as they become more complex to discourage over fitting during the training process.

Question 18

What does regularization encourage?

Answer 18

A “simpler” hypotheses

Question 19

What effect does regularization have?

Answer 19

Shrinking the coefficients of the model towards zero.

Question 20

Name two common approaches to regularization.

Answer 20

L1 and L2

Question 21

Describe L1 and L2 regularization.

Answer 21

In L1 regularization we add the absolute values

In L2 regularization we use the squared term

Question 22

Which regularization results in better predictions?

Answer 22

L2

Question 23

What is hyperparameter optimization?

Answer 23

Some models have parameters of their own.

Use use grid search over the possible values of parameters in conjunction with cross validation to discover the optimal combination of those parameters.

We learn how best to learn.