8-Model evaluation 1

Question 1

What is holdout evaluation strategy?

Answer 1

Partition the data into test and training data. Train only on training data, evaluate only on test data. Usually 80-20, 90-10 splits

Question 2

What are the advantages of holdout strategy?

Answer 2

Simple to work with and implement
High reproducability

Question 3

What are the disadvantages of holdout strategy?

Answer 3

Size of the split impacts model behaviour. Too many test instances, not enough learning. Too many training instances, not enough validation.

Question 4

What is repeated random subsampling?

Answer 4

Run holdout strategy multiple times on random set of training and test elements. Evaluate by averaging across the iterations.

Question 5

What is the advantage of repeated random subsampling?

Answer 5

Produces more reliable results than holdout strategy alone

Question 6

What are the disadvantages of repeated random subsampling?

Answer 6

Difficult to reproduce
Slower than holdout strategy
Wrong choice of training set, test set size can lead to highly misleading results

Question 7

What is cross validation?

Answer 7

Data is split into several partitions (m >= 2) and iteratively one partition is used as test data, while the other m-1 is used as training data. Evaluation metric is aggregated against the other 10.

Question 8

What are the pros of cross validation?

Answer 8

Very reproducible
Takes roughly the same time as repeated random subsampling
Every instance is a test instance for some partition
Minimises bias and variance of estimates

Question 9

What is the inductive learning hypothesis?

Answer 9

Any hypothesis found to approximate the target function well over a training data set will approximate the the target function well over unseen data

Question 10

What is error rate?

Answer 10

Fraction of incorrect predictions

Question 11

Why is error rate not ideal?

Answer 11

Some problems require us to penalise false negative errors, e.g. medical diagnosis. Others want to penalise false positive errors, e.g. spam classification

Question 12

What is accuracy?

Answer 12

Accuracy is 1 - error rate

Question 13

What is precision?

Answer 13

Precision is calculated on a specific label. It is calculated as TP / (TP + FP)

Intuitively it is “how often are we correct when we predict an instance is interesting?”

Question 14

What is recall?

Answer 14

Recall is calculated on a specific label. It is calculated as TP / (TP + FN)

Intuitively it is “how often do we correctly classify an interesting instance as interesting?”

Question 15

What relationship holds for precision and recall?

Answer 15

Usually precision and recall are in an inverse relationship

Question 16

What is the F score?

Answer 16

F1 is used when we want precision and recall to be high.

It is calculated as (1+b^2)PR/(b^2*precision + recall)

Question 17

What is ROC (Receiver Operating Characteristics) curve?

Answer 17

A graph comparing the rate of true positives and false positives. We want the rate to be in the bottom right triangle of the chart.

TPR = TP/(TP+FN)
FPR = FP(FP + TN)

Question 18

What is a contigency table?

Answer 18

A 2 by 2 table capturing the TP, FP, TN and FN

Question 19

What is a confusion matrix?

Answer 19

A n by n table capturing the results of a series of tests relative to an interesting class. The interesting class is the only label deemed as TP. All other classes are TN, even if matched incorrectly with the non-interesting class.

Question 20

What is macro-averaging?

Answer 20

The average of the precision (recall) values divided by the number of classes

Question 21

What is micro-averaging?

Answer 21

The sum of TP across classes divided by the sum of TP and FP across classes for micro-precision. Analogous for Recall

Question 22

What is weighted averaging?

Answer 22

The average of precision (recall) values weighted against the proportion of the class in the test data

Question 23

What is a baseline?

Answer 23

The baseline model is a naive method or model

Question 24

What is a benchmark?

Answer 24

A benchmark is an established rival technique that our model is pitched against

Question 25

What are types of baselines?

Answer 25

Random baseline: Assign a random class
Weighted random: Assign a random class based on proportions of classes in training data set
Zero-R: Assign the most likely class in the training dataset
One-R: Select one attribute and use it to predict an instance’s class. Test each attribute and select the one with the lowest error rate on the training data set

Question 26

What are the advantages of One-R?

Answer 26

Simple to understand
Simple to comprehend results

Question 27

What are the disadvantages of One-R?

Answer 27

Unable to capture attribute interactions
Biased towards high-arity attributes