Evaluation Metrics

Question 1

In Hypothesis Testing, what is the null hypothesis?

Answer 1

Null Hypothesis H_0 states the assumption to be tested

Question 2

What are the two types of error in hypothesis testing

Answer 2

Type I Error “false positive”: Rejecting H_0 when it is in fact true

Type II Error “false negative”: Failing to reject H_0 when it is in fact false

Question 3

Give the formulas for misclassification rate and accuracy in Classifier Accuracy

Answer 3

MR = # incorrect predictions /
total predictions

ACC = # correct predictions /
total predictions

Question 4

What do confusion matrices summarise

Answer 4

the performance of an algorithm when compared with the real classes

Question 5

Give the formulas for ACC, TPR, FPR, TNR

Answer 5

ACC = TP+TN / TP+FP+TN+FN
TPR = TP / TP + FN
FPR = FP / FP + TN
TNR = TN / FP + TN

Question 6

Give the formulas for precision and recall

Answer 6

A = Retrieved and relevant
B = Retrieved results
C = Relevant Results
Precision = A / A+B
Recall = A / A+C

Question 7

What is a Precision-Recall (PR) curve used for?

Answer 7

• To study the output of a binary classifier
• Measure precision at fixed recall intervals

Question 8

Give the formulas for Balance Accuracy Rate (BAR) and Balance Error Rate (BER)

Answer 8

BAR: Mean of TPR and TNR
BER: Mean of FPR and FNR

Question 9

Give the formula of the F1 Measure

Answer 9

2 * Precision * Recall /
Precision + Recall

Question 10

What is a decision threshold and what is its most common value?

Answer 10

the value (theta) used to discriminate when selecting between a positive and negative outcome.
Most common value = 0.5

Question 11

What does a ROC plot visualise?

Answer 11

How the TPR and FPR change over many different thresholds.

Question 12

What is overfitting

Answer 12

Model is fitted too closely to the training data (including its noise). The model cannot generalise to situations not presented during training, so it is not useful when applied to unseen data

Question 13

Possible causes of overfitting

Answer 13

• Small training set
• Complex Model
• noise
• high dimensionality

Question 14

What is peeking and what can you use to avoid it

Answer 14

When the performance of a model is evaluated using the same data used to train it.

Avoid peeking by using a hold-out set

Question 15

What are some drawbacks to using a random split as the hold-out strategy

Answer 15

• Sometimes we don’t have the “luxury” of setting aside data for testing.
• Since it is a single experiment, the hold-out estimate of error rate can be misleading if we get an “unfortunate” split of the data.
• Even if we use multiple splits, some examples will never be included for training or testing, while others might be selected many times.

Question 16

Briefly explain the steps of k-Fold Cross Validation

Answer 16

• Divide the data into k disjoint subsets - “folds”
• For each of k experiments, use k-1 folds for training and the selected one fold for testing.
• Repeat for all k folds, average the accuracy/error rates.

Question 17

What is the validation set in the Three-Way-Hold-Out Strategy?

Answer 17

The subset of examples used to tune the classifier

Question 18

What is the main advantage of the Three-Way-Hold-Out Strategy?

Answer 18

It avoids a bias in evaluation of the model