Data Mining - Lecture Performance Measures

Question 1

What do errors based on the training set tell us?

Answer 1

About the model’s fit.

Question 2

What do errors based on the validation set tell us?

Answer 2

The ability to predict new data.

These errors are called prediction errors.

Question 3

Which three types of outcome do we deal with in this course?

and can we also evaluate

Answer 3

1. A predicted numerical value
2. A predicted class membership
3. The probability of class membership

Question 4

How do we measure prediction accuracy for numerical prediction?

Answer 4

We use the error for each record and compute one of the following measures:

1. Mean absolute error (MAE)
2. Mean error
3. Mean percentage error (MPE)
4. Mean absolute percentage error (MAPE)
5. Root mean squared error
6. Lift Chart

Question 5

How do you compute the mean absolute error?

Answer 5

For each record you get the error. You disregard the sign before the value and you sum up all errors.

You multiply the sum with (1/n)

Question 6

How do you compute the mean error?

Answer 6

You sum up all errors (including the signs before the value). You multiply the sum of errors with (1/n).

Question 7

How do you compute the mean percentage error?

Answer 7

You divide each record’s error by the records actual value (Yi). You do this for all records and sum them up.

You multiply the sum with (1/N) and you do that * 100

Question 8

How do you compute the mean absolute percentage error?

Answer 8

Same as the mean percentage error, except in the first part you do not take the sign into considerations before the error value/Yi.

Question 9

How do you compute the Root mean squared error?

Answer 9

You square each individual error and sum them up. You multiply it by (1/n)/ The answer you put in a square root.

Question 10

What is a lift chart?

Answer 10

It is a chart that to compare a model with a baseline model with no predictions, to see which subset of records gives the highest cumulative predicted values.

On the x-axis you put percentage of samples.
On the y-axis you put the percentage where the model predicts the positive class well.

Then you can see how your model performs. It can be that for a small percentage of the sample you will get a more then average prediction/response rate.

Question 11

How can we evaluate classifier performance?

Answer 11

1. We make a confusion matrix.
2. Based on that confusion matrix we can compute accuracy, prediction, recall, F1
3. We can also make a ROC Curve based on the confusion matrix.

Question 12

What is a misclassification?

Answer 12

If your model puts records in the wrong class:

False Negative and False Positive

Question 13

What is a confusion matrix?

Answer 13

A matrix with the predicted classes on the x-axis and the actual classes on the y classes. It lists the TP, FP, TN, FN.
It reads like:

True Negative | False Positive
False Negative | True Positive

Question 14

What is a type I error?

Answer 14

False Positive

Question 15

What is a type II error?

Answer 15

False Negative

Question 16

How do you compute accuracy to evaluate classifier performance?

Answer 16

True Positive + True Negative / n

Question 17

What is a ROC Curve?

Answer 17

A diagram in which you can put multiple models to evaluate their performance.

On the x-axis you have False Positive Rate
On the y-axis you have True Positive Rate

Question 18

What happens if TP,FP in the ROC curve is (0.0)?

Answer 18

The model declares everything as a negative class

Question 19

What happens if the TP,FP in the ROC curve is (1,1)?

Answer 19

It declares everything as a positive class

Question 20

What happens if the TP,FP in the ROC curve is (1,0)?

Answer 20

Ideal situation

Question 21

What is the area under the curve in ROC in the random guessing situation and what is it in an ideal situation?

Answer 21

Random: 0.5
Ideal: 1.0

So your model should definitely be in between those two.

Question 22

What is a limitation of accuracy?

Answer 22

If you have many records in one class and zero in the other class, the accuracy can still be high, even though it is not putting anything in one class.

Question 23

How can you avoid this?

Refers to accuracy and it’s flaw

Answer 23

If you get a cost matrix as well.

• > You compute the accuracy
• > You also compute total cost by (TP * cost TP) + (FP * cost FP) + (TN * cost TN) + (FN * cost FN)

Question 24

What is the Kappa statistics in multiclass prediction?

Answer 24

You can use it if you have an actual predictor confusion matrix and a random predictor confusion matrix.
-> It measures the improvement compared to the random predictor

(success rate actual predict - success rate random predictor) / (1 - succes rate random predictor)

Question 25

What is Recall?

Answer 25

The ability of the model to find all of the items of the class.

(True Positive) / (True Positive + False Negative)

Question 26

What is Precision?

Answer 26

The ability of the model to correctly detect class items.

True Positive) / (True Positive + False Positives

Question 27

What is the F-Measure?

Answer 27

Takes into account both Recall as Precision

F = (2 / (1/R) + (1/P)