Measuring Performance

Question 1

What is Classification Accuracy?

Answer 1

number of correctly classified samples

/

total samples

Question 2

What is Classification Error Rate?

Answer 2

number of wrongly classified samples

/

total samples

Question 3

What is Recall?

Answer 3

TP

/

TP + FN

Question 4

What is Precision?

Answer 4

TP

/

TP + FP

Question 5

What are some measures we can use to measure performance in regression?

Answer 5

1. Root Mean Squared Error
2. Mean Absolute Error
3. Mean Absolute Percentage Error
4. Coefficient of Determination

Question 6

What is Root Mean Squared Error?

Answer 6

Take the average of the square differences, and then square root

Question 7

What is Mean Absolute Error?

Answer 7

Take the average of the absolute differences

Question 8

What is the Mean Absolute Percentage Error?

Answer 8

Take the average of ( the absolute differences divided by the true value)

Question 9

When is Mean Absolute Percentage Error useful?

Answer 9

When different classes in our output might give drastically different range of values.

E.g one output might be temperature, and the other might be kilo calories. Therefore we should normalise the error for each before summing them up

Question 10

What is the Bias Issue?

Answer 10

The idea that the accuracy of the training samples can be a poor estimator of the accuracy on unseen samples

Question 11

What is the Variance Issue?

Answer 11

The idea that the accuracy on a new set of test samples can still vary from the true accuracy, depending on the makeup of the test samples

Smaller set of test samples can result in a higher variance

Question 12

What is the Holdout Method?

Answer 12

Splitting the dataset into training and testing dataset

Question 13

What is Random Sub-Sampling?

Answer 13

1. Split the data in to K parts
2. Randomly select a fixed number for training and testing
3. Train the classifier from scratch using the training data, and then test to compute an error
4. Repeat the experiment from steps 1 and 2, K times
5. The final error is the average error of all the experiments

Question 14

What is K-Fold Cross Validation?

Answer 14

1. Divide the dataset into K partitions
2. For each of the K experiments, use K-1 partitions for training, and estimate the error using the remaining partition
3. The final error estimate is the average error of all K experiments

Question 15

What is an advantage of using K-Fold Cross Validation?

Answer 15

All the examples in the dataset are eventually used for both training and testing

Question 16

What is Leave One Out Cross Validation?

Answer 16

This is an extreme case of K-Fold Cross Validation.

Where if we have a dataset of N samples, we do N-Fold Cross Validation.

Question 17

What is Bootstrap?

Answer 17

For K experiments, select the training samples using sampling with replacement
And use the data that was never selected as testing data

The final error is the average error of the K experiments

Question 18

Comparing the errors of two models using Z-Tests

Answer 18

???

Question 19

What is F-1 Score?

Answer 19

2 * P * R

/

P + R

Question 20

What is the Variance Error?

Answer 20

This tells you how much the predictions changes under different realisations of the same model

Question 21

What is the Bias Error?

Answer 21

This is a measure of how the close the average prediction is to the true value, under different realisations of the same model.

Question 22

If I have high variance error, and low bias error, will I suffer from over-fitting or under-fitting?

Answer 22

Over-fitting

Question 23

If I have low-variance error, and high bias error, will I suffer from over-fitting or under-fitting?

Answer 23

Under-fitting