ML-05 - Model evaluation

Question 1

ML-05 - Model evaluation

Describe the typical ML workflow. (7)

Answer 1

(See image)

Question 2

ML-05 - Model evaluation

Describe k-fold cross-validation.

Answer 2

(See image)

Question 3

ML-05 - Model evaluation

Describe the validation workflow.

Answer 3

(See image)

Question 4

ML-05 - Model evaluation

Describe the test workflow.

Answer 4

Question 5

ML-05 - Model evaluation

What is a requirement for using accuracy as a metric?

Answer 5

Your data has to be balanced.

Question 6

ML-05 - Model evaluation

What should you use in place of accuracy, if your data is imbalanced?

Answer 6

Precision, recall, F1 or ROC-AUC.

Question 7

ML-05 - Model evaluation

What is the formula for precision?

Answer 7

TP/PP = TP / (TP + FP) = True positives / predicted positives.

Question 8

ML-05 - Model evaluation

What is the formula for recall?

Answer 8

TP/P = TP/(TP + FN) = True positives / actual positives

Question 9

ML-05 - Model evaluation

Fill in the labels “Predicted” and “Actual”. (See image)

Answer 9

(See image)

Question 10

ML-05 - Model evaluation

Which one is used to calculate precision? (See image)

Answer 10

2

Question 11

ML-05 - Model evaluation

Which one is used to calculate recall? (See image)

Answer 11

1

Question 12

ML-05 - Model evaluation

What’s the formula for F1 score?

Answer 12

(See image)

Question 13

ML-05 - Model evaluation

What are some benefits of the F1 score? (2)

Answer 13

• 1 metric over 2 (P&R) makes it easier to make decisions.
• Automatically selects threshold values.

Question 14

ML-05 - Model evaluation

What is a drawback of the F1 score?

Answer 14

Difficult to interpret.

Question 15

ML-05 - Model evaluation

How can you visualize precision/recall? (3 mentioned)

Answer 15

• Confusion matrix
• ROC curve (from slides; technically TPR/FPR?)
• Precision/recall curve

Question 16

ML-05 - Model evaluation

What is ROC short for?

Answer 16

Receiver operating characteristic

Question 17

ML-05 - Model evaluation

What us AUC short for?

Answer 17

Area under curve

Question 18

ML-05 - Model evaluation

What is a ROC curve?

Answer 18

A plot of TPR vs. FPR.

Question 19

ML-05 - Model evaluation

What is ROC-AUC?

Answer 19

The area under a ROC curve.

Question 20

ML-05 - Model evaluation

What is TPR short for?

Answer 20

True positive rate

Question 21

ML-05 - Model evaluation

What is FPR short for?

Answer 21

False positive rate

Question 22

ML-05 - Model evaluation

How do you interpret the values in ROC-AUC, e.g. 0.9?

Answer 22

AUC = 0.9 means there is a 90% chance the model can distinguish between +/- classes.

Question 23

ML-05 - Model evaluation

What is PR-AUC short for?

Answer 23

Precision-recall curve area under curve.

Question 24

ML-05 - Model evaluation

What is a PR curve?

Answer 24

Precision recall curve.

Question 25

ML-05 - Model evaluation

How do you plot a PR curve?

Answer 25

A precision-recall plot has P/R as the axes.

It is plotted by changing the threshold value for e.g. a sigmoid output activation, then plotting the corresponding precision/recall values

Question 26

ML-05 - Model evaluation

How do you tune precision/recall to avoid false positives?

Answer 26

Tune for high precision, low recall.

Question 27

ML-05 - Model evaluation

How do you tune precision/recall to avoid false negatives?

Answer 27

Tune for low precision, high recall.