Lecture 7 - Model Selection, Evaluation

Question 1

What are the two goals of evaluating the prediction performance?

Answer 1

1. Model selection: almost all methods have parameters that need to be chosen
2. Model evaluation: After selecting your hypothesis, estimate how well it generalizes to new data.

Question 2

What does the loss function tell?

Answer 2

The price for predicting y’ when true value is y

Question 3

What is misclassification rate?

Answer 3

The average zero-one loss over data set, aka the percentage of falses: 10% falses => misclassification rate 0.1

Question 4

What is classification accuracy?

Answer 4

The fraction of correct instances, aka 1 - misclassification rate

Question 5

What is the generalization error?

Answer 5

Expected loss over the probability distribution of our data, aka the loss on average for random new instance

Question 6

What is the standard approach for generalization error?

Answer 6

Having a training and test data

Question 7

What are some estimators of generalization error?

Answer 7

• Training set error
• Test set error
• CV error
• Bootstrap error

Question 8

What is bias?

Answer 8

Expected difference between estimated and true error, ideally close to zero

Question 9

What is optimistic/pessimistic bias?

Answer 9

Optimistic bias: systematically estimates error to be smaller than it is
Pessimistic bias: systematically estimates error to be larger than it is

Question 10

What is variance?

Answer 10

How large the magnitudes of the differences tend to be on average.

Question 11

Can method with zero bias have large variance?

Answer 11

Yes, negative + positive errors cancel each other

Question 12

What is training set error (resubstitution error)

Answer 12

The average loss on the training examples.
Has high optimistic bias, since the model is chosen due to good fit to this particular data set

Question 13

What is testing set error (holdout estimate)?

Answer 13

Splitting data to training and test, then computing average loss on test set.

Question 14

Is training set error unbiased?

Answer 14

Only if testing a single hypothesis, testing multiple hyperparameters etc leads to optimistic bias

Question 15

What is the solution to avoid optimistic bias for test set?

Answer 15

Splitting the data to training-validation-test

Question 16

What is stratification?

Answer 16

Using strategy to make the data splits more similar to distribution of classes in each set.

Question 17

What is Cross-Validation and why is it used?

Answer 17

It is used because too small test sets can be unreliable. We want as much data as possible for testing and training, so we use CV to make “multiple” training and testing sets from the same data.

Question 18

What is leave-one-out cross-validation?

Answer 18

Using data of n instances, we use ith instance as a test set and n-1 as training sets, then we loop through all the n.

Question 19

Can the leave-one-out CV overfit?

Answer 19

No, because test instances are not part of the training set

Question 20

What is the benefit of leave-one-out CV?

Answer 20

Allows us to use the whole data set for both training and testing simultaneously?

Question 21

What is K-fold CV?

Answer 21

Same idea as with leave-one-out CV, but instead of having one for testing, it has one fold for testing and K-1 folds for training.

Question 22

What is the K in K-fold cross validation?

Answer 22

It is the number of how many folds the data is split into, NOT HOW MANY DATA POINTS THERE ARE IN ONE FOLD.

So for example 100 data points with k=5 => test set is 20

Question 23

What is the benefit of K-fold CV?

Answer 23

Much faster than leave-one-out.

Question 24

What is the disadvantage of K-fold CV?

Answer 24

Can have a small pessimistic bias, because the model evaluated can be slightly less accurate than the final model trained on all data

Question 25

What’s the benefit of M-times repeated K-fold CV?

Answer 25

Some variance in the K-fold can be reduced with repeating the random sampled folds multiple times.

Question 26

Can we do model selection with CV?

Answer 26

Yes, using the training folds we can choose hyperparameter values, feature sets etc. and then test the final model on independent test data. does not need validaition set.

Question 27

What is nested CV?

Answer 27

CV within CV. We split the data to K-fold, then we take one fold as a test data, and other folds test different parameters within the fold itself until testing with the test data.

Computationally very expensive

Question 28

What are some challenges in performance evaluation?

Answer 28

The data isn’t always iid (independent, identically distributed).

Question 29

What’s the issue with grouped data?

Answer 29

With grouped data it’s easy to make predictions, if parts of the group is in the training and part in the testing. Solution s to randomize data on whole group level, so all group data is either on the test or training set

Question 30

Why is time series data problematic?

Answer 30

Random split is not suitable, it is easy to predict stock price if we know the previous and next day of the predicted date

Question 31

What is Mean Squared Error? What is it’s baseline?

Answer 31

(y’ - y)^2.
Zero if correct
Sensitive to outliers
Baseline: is MSE smaller than predicting the mean of training set y values

Question 32

What is mean absolute error? What is it’s baseline?

Answer 32

|y’ - y|
Zero if correct, else linearly growing
Not as sensitive to outliers
Baseline: smaller than the median of training

Question 33

What is baseline for misclassification rate?

Answer 33

Majority voter

Question 34

What is cost matrix?

Answer 34

The consequence for one class might be different than for another class.

For example broken tea cup produced again -> c1

Mailing broken cup leads to mailing a new one, reputation costs etc -> c2

if c1 is a lot cheaper, it is better to predict broken more often incorrectly than not broken

Question 35

What is confusion matrix?

Answer 35

Classification results represented in a confusion matrix, where rows are true classes and the columns are predicted classes

Question 36

What does a binary confusion matrix look like?

Answer 36

On rows are predicted values, on columns actual values,

TP FP
FN TN

Question 37

How is the precision calculated? What does it tell us?

Answer 37

Precision = TP / (TP+FP)

Tells the proportion of relevant documents among those returned, can be maximized with classifying only those instances we are most certain are positive -> bad recall

Question 38

How is recall calculated and what does it tell us?

Answer 38

Recall = TP / (TP + FN)

Maximized if there are no false positives, so easy to maximize, classify everything as positive.

Tells us the proportion of relevant documents found

Question 39

How is F-score calculated?

Answer 39

2 * (Precision * Recall) / (Precision + Recall)

Question 40

What is TPR?

Answer 40

True Positive Rate (recall)
TPR = TP / (TP + FN)

Proporion of positives correctly identified

Question 41

What is FPR?

Answer 41

False Positive Rate
FPR = FP / (FP + TN)

Proportion of negatives incorrectly identified

Question 42

What is the ROC curve?

Answer 42

Receiver operating characteristic curve? Draws the true positive rate against the false positive rate

Question 43

What is AUC?

Answer 43

The area under ROC curve. It’s an indicator how well the classifier solves the problem, the larger the are the better the solution.

Question 44

What values AUC has?

Answer 44

Between 0 and 1. 1 = perfect ranking, all instances of positive class receive higher predicted values than those of negative class. 0.5 is the same as guessing, so AUC should be better than 0.5

Question 45

What does it mean that AUC is invariant to relative class distributions?

Answer 45

AUC should be same for inbalanced data sets with 90-10 positives and 50-50 positives

Question 46

What isi classical mistake for AUC?

Answer 46

Using discrete values 1 and 0 instead of TPR and FPR values like 0.6 and 0.4