Part 2: Performance measures

Question 1

Confusion matrix

Answer 1

Classification for 2 or more classes
- Data set for which class is known.
- Model assigns 0 or 1 to record.
- Example text classification/sentiment analysis
\+ spam vs not-spam
\+ positive, neutral, negative
- Some probabilities between 0 and 1 -> threshold
- Also more classes

Question 2

False positive (FP)

Answer 2

Selected documents that are not relevant.

Question 3

False negative (FN)

Answer 3

Not selected documents that are relevant.

Question 4

Precision

Answer 4

The fraction of relevant and selected, compared to the selected. TP/(TP+FP)

• When precision < 1, not all retrieved are relevant, FP > 0
• When precision = 1, the FP = 0

Question 5

Recall

Answer 5

The fraction of relevant and selected compared to the relevant. TP/(TP+FN)

• Recall < 1, not all relevant are selected, FN > 0
• Recall = 1, FN = 0

Question 6

F1 score

Answer 6

(2 * precision * recall) / (precision + recall)

Maximal when precision + recall = constant.

Question 7

ROC curve

Answer 7

Graph represents the cumulative classes (0, 1) of the example explained above. Every time you switch from direction (from vertical to horizontal or vice versa), you have an error.
Best curve is when you have 1 switch between horizontal and vertical (angle 90 degrees).

Probabilistic classifier:
- Assume the model calculates a number between 0 and 1.
- All records are listed in decreasing order.
- We choose a cut off value to decide on class.
This is also common in fraud analysis: the cut off is the suspicious one.

Question 8

Comparing classifiers

Answer 8

• By using the probability we can order the records.
• This produces a ROC curve.
• Then choose the best cut-off value.