Classifier Evaluation Flashcards

1
Q

Classifier Evaluation

A

Usually advised that a classification algorithm (binary or multi-class) is evaluated. Applies to all classification algorithms whether learning is supervised or unsupervised, based on metaheuristics or others.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

The purpose of evaluations

A

is to determine how well the classification algorithm performs on some test data.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Classifier Evaluation using Test Data

A

The objective is to evaluate how the classifier performs on our test data. Use 80% of data for training and 20% for testing. Some algorithms also continuously use test data during training. The test dataset is input to the classifier and the output is given in a table.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

confusion matrix

A

We use a confusion matrix to summarize the results. It summarizes the performance of the classifier.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Binary Classifier Evaluation

A

Tp, Tn, Fp, Fn

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

True Positives (TP)

A

True positives are the cases when the actual class of the data point was true and the predicted was also true.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

True Negatives (TN)

A

True negatives are the cases when the actual class of the data point was false and the predicted was also false.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

False Positives (FP)

A

False positives are the cases when the actual class of the data point was false and the predicted was true.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

False Negatives (FN)

A

False negatives are the cases when the actual class of the data point was true and the predicted was false.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Recall or Sensitivity or True Positive Rate (TPR)

A

Number of items correctly identified as positive out of total actual positives. TP/(TP+FN).

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Specificity or True Negative Rate (TNR)

A

Number of items correctly identified as negative out of total actual negatives. TN/(TN+FP).

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Precision

A

Number of items correctly identified as positive out of total items identifiers as positive. TP/(TP+FP)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

False Positive Rate/Type I Error

A

Number of items wrongly identified as positive out of total actual negatives: FP/(FP+TN).

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

False Negative Rate or Type II Error

A

Number of items wrongly identified as negative out of total actual positives: FN/(TP+FN).

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

F1 Score

A

Harmonic mean of precision and recall given by:
2PrecisionRecall/(Precision+Recall)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Accuracy

A

Percentage of total items classified correctly: (TP+TN)/(N+P)

17
Q

K-Fold Cross Validation

A

Often test data is not available or can’t be used. How do we choose the learning parameters? The solution is to divide the training data randomly in folds and then use leave-one-out training. You then average the results over all folds and choose the best parameters. This is called model selection.