ML Metrics

Question 1

Offline metrics for classification models

Answer 1

Precision, recall, F1 score, accuracy, ROC-AUC, PR-AUC,
confusion matrix

Question 2

Offline metrics for regression

Answer 2

Mean squared error (MSE)
MAE
RMSE

Question 3

Offline metrics for ranking system

Answer 3

• MRR
• mAP
• nDCG

Question 4

Online metric for ad click prediction

Answer 4

• Click through rate
• Ad revenue

Question 5

Online metric for harmful content detection

Answer 5

• Number of reports
• Actioned reports

Question 6

Online metric for video recommendations

Answer 6

• Click through rate
• Total watch time
• Number of completed videos

Question 7

Types of Loss Functions

Answer 7

Mean squared error
Categorical cross-entropy loss
Binary cross-entropy loss

Question 8

Mean squared error

Answer 8

• Measures the difference between the predicted output and the true output
• Used to optimize the model parameters during training
• what we’re trying to minimize when we train a model

Question 9

precision

Answer 9

positive predictive value

probability a sample classified as positive is actually positive

TP/(TP+FP)

Question 10

recall

Answer 10

same as true positive rate
true positives / total positives
TP / (TP+FN)

sensitivity of the classification

Question 11

What’s the best metric when you have a large number of negative samples

Answer 11

Precision and recall

Precision is not affected by a large number of negative samples because it measures the fraction of true positives out of the number of predicted positives (TP +FP).

Precision measures the probability of correct detection of positive values while FPR, TPR, and ROC measure the ability to distinguish between classes.

Question 12

Highest value of F1

Answer 12

1.0 indicating perfect precision and recall

Question 13

Lowest value of F1

Answer 13

0 if either precision or recall are 0

Question 14

AUC range

Answer 14

0 to 1

Question 15

ROC

Answer 15

• true positive rate (recall) on the y axis
  false positive rate on the x axis
• captures the performance of a classification model at all classification thresholds (probability thresholds)
• does not depend on class distribution!
• receiver operating characteristic curve

Question 16

AUC

Answer 16

• area under the ROC curve
• used to evaluate a binary classification model
• Quantifies the ability of the model to correctly classify

AUC ranges from 0 to 1

Question 17

AUC of 0

Answer 17

A model that is 100% wrong

Question 18

AUC of 1

Answer 18

A model that is 100% correct

Question 19

What’s the best metric when you have a large number of positive samples

Answer 19

ROC is a better metric

Question 20

What metric should you use when detection of both classes is equally important

Answer 20

ROC

Question 21

F1

Answer 21

• used to evaluate the performance of a binary classification model
• combines precision and recall into a single measure
• harmonic mean of the precision and recall which provides a balanced measure of the model’s accuracy
• F1 is 0 if either precision or recall is 0

Question 22

true positive rate

Answer 22

aka recall
true positives / all positives
TP / (TP + FN)

Question 23

Offline Metrics

Answer 23

Score the model when building it
Before model is put into production (train, eval, and test datasets)
Examples of offline metrics: ROC, AUC, F1, R^2, MSE, intersection over union

Question 24

online metrics

Answer 24

scores from model once it is running in prod and serving

domain specific. things like click through rate or minutes spent watching a video.

Question 25

MRR

Answer 25

• mean reciprocal rank
• only considers the rank of the first relevant item
• not a good measure of the quality of the list as a whole

Question 26

mAP

Answer 26

• mean average precision
• good for ranking problems
• works well for binary relevance (relevant or irrelevant).
• For continuous relevance scores use nDCG

Question 27

nDCG

Answer 27

• winner, winner for ranking problems
• continuous relevance score
• shows how good the ranking is compared to the ideal ranking
• takes into account the position of the relevant item in a ranked list
  Ranges from 0 to 1. Higher values indicate better performance.

Question 28

nDCG acronym

Answer 28

normalized discounted cumulative gain

Question 29

Cross entropy

Answer 29

• how close the model’s predicted probabilities are to the ground truth label.
• CE is zero if we have an ideal system that predicts a 0 for the negative classes and 1 for the positive classes.
• The lower the CE, the higher the accuracy of the prediction.
• Good for ad click

Question 30

Normalized cross entropy.

Answer 30

• Normalized cross entropy.
• Ratio of our model’s CE and the CE of the background CTR.
• Low NCE indicates the model outperforms the baseline.
• NCE ≥1 indicates that the model is not performing better than the baseline.
• Good for ad click