Lecture 8 - Performance measures

Question 1

Classification

Answer 1

Classify e-mails to spam vs inbox

• Given (labeled) examples of both document types
• Train a classifier to discriminate between these two
• During operation, use classifier to select destination folder for new email: Inbox or Spam folder?

Question 2

Steps in classification

Answer 2

Step 1: Split data into train and test sets
Step 2: Build a model on a training set
Step 3: Evaluate on test set

Question 2

Note on parameter tuning in classification steps

Answer 2

• It is important that the test data is not used in any way to create the classifier
• Some training schemes operate in two stages:
  
  • Stage 1: build the basic structure
  • Stage 2: optimise parameter setting
• The test data cannot be used for parameter tuning
• Proper procedure uses three sets: training data, validation data, and test data
  
  • Validation data is used to optimise parameters

Question 3

Making the most of the data

Answer 3

• Once evaluation is completed, all the data can be used to build the final classifier
• Generally, the larger the training data the better the classifier (but returns diminish)
• The larger the test data the more accurate the error estimate

Question 4

Types of outcomes

Answer 4

• Building models using training data is called Supervised Learning
• Interested in predicting the outcome variable for new records
• Three main types of outcomes:
  
  • a. Predicted numerical value, e.g., house price
  • b. Predicted class membership, e.g., cancer or not
  • c. Probability of class membership (for categorical outcome variable), e.g., Naive Bayes

Question 5

Evaluating Predictive Performance - Generating numeric predictions

Answer 5

• Interested in models that have high predictive accuracy when applied to new records
• Models are trained on the training data
• Applied to the validation data and
• Measures of accuracy then use the prediction errors on that validation set

Question 6

Prediction Accuracy measures

Answer 6

Question 7

Prediction Accuracy Measures pt2

Answer 7

Question 8

Prediction Accuracy Measures pt3

Answer 8

Question 9

Lift Chart pt1

Answer 9

• Graphical way to assess predictive performance
• In some applications, we are not interested in predicting the outcome of value of each new record
• But the goal is to search for a subset of records that gives the highest cumulative predicted values
• Compares the model’s predictive performance to a baseline model that has no predictors

Question 10

Lift Chart pt2

Answer 10

• In practice, costs are rarely known
• Decisions are made by comparing possible scenarios
• Example: promotional bailout to 1,000,000 households
  
  • Mail to all with response rate being 0.1% (1,000)
  • Consider a data mining tool that can identify a subset of 100,000 most promising households with response rate 0.4% (400)
  • Responses are 40% ut cost cost is 10%
  • → It might pay off to restrict to these 100,00
• The increase in response rate is called lift factor
• A lift chart allows a visual comparison

Question 11

Judging Classifier Performance - i.e., categorical variables

Answer 11

• Misclasification is when a record belongs to one class but the model classifies it as a member of a different class
• A natural criterion for judging the performance of a classifier is the probability of making a misclassification error

Perfect classifier makes no errors! but the real world has “noise” therefore cannot construct perfect classifiers

Question 12

Confusion / Classification matrix

Answer 12

• A matrix that summarises the correct and incorrect classifications that a classifier produced for a given dataset
• Rows and columns correspond to the predicted and true (actual) classes
• In practice, most accuracy measures are derived from this matrix
• Correct classifications: True Positive and True Negative
• Incorrect classifications: False positive - outcome incorrectly predicted as a yes / positive, False negative i.e., outcome incorrectly predicted as a no / negative

Question 13

Type I and II errors

Answer 13

Type I Error → False Positive

• Predicted positive but that is incorrect
• Example: predicted that a man is pregnant, but actually he is not pregnant

Type II Error → False Negative

• Predicted negative but that is incorrect
• Example: predicted a woman is not pregnant when actually she is pregnant

Question 14

Overall success rate / Accuracy

Answer 14

Number of correct classifications divided by the total number of classifications

Question 15

ROC curves

Answer 15

• Developed in 1950s for signal detection theory to analyse noisy signals
• Characterise the trade-off between positive hits and false alarms
• ROC curve plots TP (on the y-axis) against FP (on the x-axis)
• Plots the true positive rate against the false positive rate for the different possible thresholds of a diagnostic test

Question 16

Example ROC curve

Answer 16

Question 17

Using ROC for Model Comparison

Answer 17

Question 18

Limitation of Accuracy and Cost Matrix

Answer 18

Consider a 2-class problem: number of class 0 examples = 9990 and number of class 1 examples = 10

If model predicts everything to be class 0, accuracy is 9990/10000 = 99.9%. Accuracy is misleading because model does not detect any class 1 example

Question 19

Computing Cost of Classification

Answer 19

Question 20

Multiclass prediction

Answer 20

Question 21

Multiclass prediction - Kappa statistic

Answer 21

Question 22

Precision and Recall

Answer 22

Question 23

Precision and Recall formulas

Answer 23

Question 24

Determining Recall is difficult

Answer 24

• Total number of items / records that belong to a particular class is sometimes not available

Solutions:

• Sample across the dataset and perform relevance judgment on these items
• Apply different models to the same dataset and then use the aggregate of relevant items as the total relevant set

Question 25

F-measure (F1-measure)

Answer 25

Question 26

Python code

Answer 26