Chapter 4 Tour of Model Evaluation Metrics

Question 1

Choosing model evaluation metrics is challenging. This challenge is made even more difficult when there is a skew in the class distribution. Why? P 54

Answer 1

The reason for this is that many of the standard metrics become unreliable or even misleading when classes are imbalanced, or severely imbalanced, such as 1:100 or 1:1000 ratio between a minority and majority class.

Question 2

We can divide evaluation metrics into three useful groups; What are they? P 55

Answer 2

1. Threshold Metrics (e.g., accuracy and F-measure)
2. Ranking Metrics (e.g., receiver operating characteristics (ROC) analysis and AUC)
3. Probability Metrics (e.g., mean-squared error)

Question 3

When are threshold metrics used? P 55

Answer 3

Threshold metrics are those that quantify the classification prediction errors. That is, they are designed to summarize the fraction, ratio, or rate of when a predicted class does not match the expected class in a holdout dataset. These measures are used when we want a model to minimize the number of errors.

Question 4

Perhaps the most widely used threshold metric is … and the complement of classification accuracy called …. P 56

Answer 4

Classification accuracy, classification error

Question 5

There are two groups of metrics that may be useful for imbalanced classification because they focus on one class; they are … and … P 56

Answer 5

sensitivity(TPR)-specificity(TNR), precision-recall

Question 6

Sensitivity and Specificity can be combined into a single score that balances both concerns, called the G-mean. How is G-mean calculated? P 57

Answer 6

G-mean = √(Sensitivity × Specificity)

Question 7

Precision and recall can be combined into a single score that seeks to balance both concerns, called the F-score or the F-measure. How is it calculated? P 57

Answer 7

F-measure = 2 × Precision × Recall/( Precision + Recall)

Question 8

The F-measure is a popular metric for imbalanced classification. True/False P 57

Answer 8

True

Question 9

What is F Beta -measure? How is it calculated? P 57

Answer 9

The Fbeta-measure (or F β − measure) measure is an abstraction of the F-measure where the balance of precision and recall in the calculation of the harmonic mean is controlled by a coefficient called beta.
Fbeta-measure = (1 + β ^2 ) × Precision × Recall /(β^ 2 × Precision + Recall)

The choice of the β parameter will be used in the name of the Fbeta-measure. For example,
a β value of 2 is referred to as F2-measure or F2-score. A β value of 1 is referred to as the
F1-measure or the F1-score. Three common values for the beta parameter are as follows:
ˆ F0.5-measure (β = 0.5): More weight on precision, less weight on recall.
ˆ F1-measure (β = 1): Balance the weight on precision and recall.
ˆ F2-measure (β = 2): Less weight on precision, more weight on recall.

Question 10

What is one limitation of the threshold metrics (besides not specifically focusing on the minority class)? P 57

Answer 10

One limitation of the Threshold metrics is that they assume that the class distribution observed in the training dataset will match the distribution in the test set and in real data when the model is used to make predictions. This is often the case, but when it is not the case, the performance can be quite misleading. In particular, they assume that the class imbalance present in the training set is the one that will be encountered throughout the operating life of the classifier.

Question 11

Ranking metrics don’t make any assumptions about class distributions. True/False? P 57

Answer 11

True

Unlike Threshold metrics that assume that the class distribution observed in the training dataset will match the distribution
in the test set and in real data when the model is used to make predictions.

Question 12

Rank metrics are more concerned with evaluating classifiers based on how effective they are at … P 58

Answer 12

Separating classes

Question 13

How do rank metrics evaluate a model’s performance? P 58

Answer 13

These metrics require that a classifier predicts a score or a probability of class membership. From this score, different thresholds can be applied to test the effectiveness of classifiers. Those models that maintain a good score across a range of thresholds will have good class separation and will be ranked higher.

Question 14

What is the most commonly used ranking metric? P 58

Answer 14

The ROC Curve or ROC Analysis

Question 15

How’s a no skill model shown as ROC-AUC curve ?

Answer 15

A classifier that has no skill (a no skill model is one that can’t discriminate between the classes and would predict a random class or a constant class in all cases) will be represented by a
diagonal line from the bottom left to the top right.

Question 16

Although generally effective, the ROC Curve and ROC AUC can be optimistic under a severe class imbalance, especially when the number of examples in the minority class is small. True/False? P 59

Answer 16

True

Question 17

An alternative to the ROC Curve is the …curve that can be used in a similar way, although focuses on the performance of the classifier on the minority class. P 59

Answer 17

Precision-recall

again, different thresholds are used on a set of predictions by a model, and in this case, the precision and recall are calculated.

Question 18

How’s a no skill curve on precision-recall metric? P 59

Answer 18

A no-skill classifier will be a horizontal line on the plot with a precision that is proportional to the number of positive examples in the dataset. For a balanced dataset this will be 0.5.

Question 19

What are probabilistic metrics designed for specifically? P 60

Answer 19

Probabilistic metrics are designed specifically to quantify the uncertainty in a classifier’s predictions.

Question 20

When are probabilistic metrics used? P 60

Answer 20

These are useful for problems where we are less interested in incorrect vs. correct class predictions and more interested in the uncertainty the model has in predictions and penalizing those predictions that are wrong but highly confident.

These measures are especially useful when we want an assessment of the reliability of the classifiers, not only measuring when they fail but whether they have selected the wrong class with a high or low probability.

Question 21

Perhaps the most common metric for evaluating predicted probabilities is … for binary classification (or the…), or known more generally as … Another popular score for predicted probabilities is the …. P 61

Answer 21

log loss, negative log likelihood, cross-entropy, Brier score

Question 22

What’s the benefit of the Brier score compared to log loss when dealing with imbalanced datases? P 61

Answer 22

Brier score is focused on the positive class, which for imbalanced classification is the minority class. This makes it more preferable than log loss, which is focused on the entire probability distribution.

Question 23

What is log loss and Brier score formula? P 61

Answer 23

LogLoss = −((1 − y) × log(1 − yhat) + y × log(yhat))
BrierScore = (1 /N) sum( (yhati − yi) ^2)

Question 24

Can Brier score be used for multi-class problems? P 61

Answer 24

Although typically described in terms of binary classification tasks, the Brier score can also be calculated for multiclass classification problems.

Question 25

The differences in Brier score for different classifiers can be very small. How is this problem addressed? P 61

Answer 25

In order to address this problem, the score can be scaled against a reference score, such as the score from a no skill classifier (e.g. predicting the probability distribution of the positive class in the training dataset). Using the reference score, a Brier Skill Score, or BSS, can be calculated where 0.0 represents no skill, worse than no skill results are negative, and the perfect skill is represented by a value of 1.0.

Question 26

What’s BrierSkillScore formula? P 61

Answer 26

BrierSkillScore = 1 − BrierScore /BrierScoreref

Question 27

Although popular for balanced classification problems, probability scoring methods are less widely used for classification problems with a skewed class distribution. True/Fasle? P 61

Answer 27

True

Question 28

How can we choose metrics for our problem? P 62

JB how to choose metrics for imbalanced
classification

Answer 28

1. Perhaps the best approach is to talk to project stakeholders and figure out what is important about a model or set of predictions. Then select a few metrics that seem to capture what is important, then test the metric with different scenarios.
2. Another approach might be to perform a literature review and discover what metrics are most commonly used by other practitioners or academics working on the same general type of problem.

The second approach can often be insightful, but be warned that some fields of study may fall into groupthink and adopt a metric that might be excellent for comparing large numbers of models at scale, but terrible for model selection in practice.

Question 29

In a multi-class classification setup, ____ (micro avg/macro avg) is preferable if you suspect there might be a class imbalance.

External

Answer 29

macro-average

If your goal is for your classifier simply to maximize its hits and minimize its misses, micro-average would be the way to go.

However, if you valued the minority class the most, you should switch to a macro-average, where you would only get a 50% score. This metric is insensitive to the imbalance of the classes and treats them all as equal.

Ref

Ref 2

Question 30

How are TP,TN,FP,FN calculated for multiclass datasets?

External

Answer 30

In a 3*3 confusion matrix for multiclass data:

FN: The False-negative value for a class will be the sum of values of corresponding rows except for the TP value. ( cell2+ cell3 for setosa)

FP: The False-positive value for a class will be the sum of values of the corresponding column except for the TP value. (cell4+ cell7 for setosa)

TN: The True Negative value for a class will be the sum of values of all columns and rows except the values of that class that we are calculating the values for. (cell5+cell6+cell8+cell9 for setosa)

TP: The True positive value is where the actual value and predicted value are the same. (cell1 for setosa)

Answer