Evaluation

Question 1

What is formal definition of overfitting?

Answer 1

A predictor F is overfit if we can find another predictor F’ where:

• E_train(F’) > E_train(F)
• E_gen(F’) gen(F)

Question 2

What is formal definition of underfitting?

Answer 2

Can find another predictor F’ with smaller E_train and E_gen

Question 3

How is E_train (training error computed)?

Answer 3

Question 4

How is E_gen calculated (generalization error)?

Answer 4

Question 5

How can we estimate E_gen?

Answer 5

Set aside test set and compute E_test (same way as E_train)

lim E_test = E_gen as the size of the test set -> infinity

Question 6

How can you compute the confidence interval for E_gen from E_test

Answer 6

Question 7

What do we use training/validation/testing sets

for?

Answer 7

• Training set: construct classifier
• Validation set: pick algorithm + tune hyper parameters
• Testing set: estimate future error rate

Question 8

How does cross-validation work?

Answer 8

• Randomly split data into k sets
• Test on one portion (train on k-1 others)
• Average error over all k folds
• Final classifier is trained on all date

Question 9

What is leave-one-out?

Answer 9

Cross validation where k = # of training instances

Question 10

What is the problem with leave-one-out validation?

Answer 10

Classes not balanced

Testing { 1 of A, 0 of B } vs training: { n/2 of B, n/(2-1) of A }

We would always predict B (most frequent), but we will always be wrong

Question 11

What does stratification do?

Answer 11

Keeps class labels balanced across training/testing sets

Question 12

How do you do stratification?

Answer 12

• Split instances by class
• Split class into K parts
• Assemple i^th fold by combining 1 part from each path

Question 13

What is true positive?

Answer 13

Classifier predicts positive, and it is positive

Question 14

What is true negative?

Answer 14

Classifier predicts negative, and it is negative

Question 15

What is false positive?

Answer 15

Classifier predicts positive, but they are negitive

Question 16

What is false negative?

Answer 16

Classifier predicts negative, but it is actuall positive

Question 17

What is the definition of classification error?

Answer 17

Question 18

What is the defintion of accuracy?

Answer 18

Question 19

What is the problem with classification error / accuracy?

Answer 19

Misleading when classes are unbalances

• Predict earthquake: unlikely so always predict no
• Decide if webpage is relevent: 99.999% are not so retreive nothing

Question 20

What is the definition of False Alarm Rate?

Answer 20

FP / (FP + TN)

Question 21

What is the definition of Miss rate?

Answer 21

FN / (TP + FN)

Question 22

What is the definition of Recall?

Answer 22

TP / (TP + FN)

Question 23

What is the defintion of Precision?

Answer 23

TP / (TP + FP)

Question 24

What is the problem with False alarm rate / miss rate / recall / precision?

Answer 24

Trivial to get 100% or 0% individually, must report them in pairs

Question 25

What evaluation measure would we use for event detection?

Answer 25

Cost = C_FP * FP + C_FN * FN

e.g. cost of evacuating with no earthquake vs cost of staying with earthquake

Question 26

What is the definition of F-measure?

Answer 26

2 / (1 / Recall + 1 / Precision)

Simular to accuracy but without TN

Question 27

What is a ROC curve?

Answer 27

Plot of TP vs FP as threshold varies

Question 28

What does a perfect and random classifer look like on an ROC curve?

Answer 28

Question 29

Whats are some problems with mean squared error?

Answer 29

• Very sensitive to outliers (because of the squaring)
• Sensitive to mean / scale (mean value might have lower MSE than a model which captures the pattern but the mean is off)

Question 30

What is the mean absolute error (MAE)?

Answer 30

Question 31

What is Median absolute deviation (MAD)?

Answer 31

med { |f(x_i) - y_i| }

Question 32

Whats the pros/cons of median absolute deviation (MAD)?

Answer 32

completly ignores outliers but cant take derivative

Question 33

What is definition of correlation coefficient?

Answer 33

Question 34

What does correlation coeffient capture?

Answer 34

Realtive ordering - usefull for ranking tasks