C4

Question 1

accuracy is not a good measure in most cases of classification

Answer 1

classes are often unbalanced: high accuracy in one class might mean low accuracy in another class
--> use precision and recall

Question 2

precision

Answer 2

what proportion of the assigned labels are correct? (for one class)

Precision = |A * T| / |A| = tp / (tp + fp)

Question 3

recall

Answer 3

what proportion of true labels was assigned?

Recall = |A * T| / |T| = tp / (tp + fn)

Question 4

F-score

Answer 4

average of precision and recall

F1 = 2 * precision * recall / (precision + recall)

Question 5

what determines the quality of the classifier

Answer 5

• difficulty of the task: more categories = more difficult
• number of training items: more is better
• richness of the representation: more informative features are better

Question 6

analysis for classification

Answer 6

per-item evaluation: which are the most difficult items, one classifier may work better for one subset, another for another subset

per-category evaluation: what is the precision and recall per class (especially with class imbalance)

error analysis: confusion matrix

Question 7

evaluation for regression

Answer 7

mean squared error:

• difference between y_hat en y = error
• square each error (bigger error more punished)
• take the mean

Question 8

evaluation for rankings

Answer 8

option 1: proportion of correct answers at position 1

option 2: proportion of items that have the correct answer in the top n

option 3: the rank of the correct answer or the rank of the highest ranked relevant answer (multiple answers)