B09 Evaluating Performance

Question 1

The goal of evaluating a ___________ is to have a better understanding of how its performance will _______ to future cases.

Answer 1

Classification Model Extrapolate

Question 2

A ____________ is a table that categorizes predictions according to whether they match the ________.

Answer 2

Confusion Matrix Actual Value

Question 3

The class of interest is known as the ________ class, while all others are known as _______.

Answer 3

Positive Negative

Question 4

The ____________ adjusts accuracy by accounting for the possibility of a correct prediction by chance alone.

Answer 4

kappa statistic

Question 5

Kappa values range from ___ to _____

Answer 5

0 (poor) - 1 (good)

Question 6

Kappa can also be though of as ______

Answer 6

Proportion of all instances where the predicted and actual values match.

Question 7

The _________ is defined as the proportion of positive predictions that are truly positive. A model with high _______ is trustworthy.

Answer 7

Precision

Question 8

The ________is a measure of the completeness of the results. A model with high _______has wide breadth.

Answer 8

Recall

Question 9

The _______ combines precision and recall into a single number using the harmonic mean. It provides a convenient way to compare several models side by side.

Answer 9

F-Measure

Question 10

The _______ of a model (also called the __________) measures the proportion of positive examples that were correctly classified.

Answer 10

Sensitivity; True Positive Rate

Question 11

The __________ of a model (also called the ____________) measures the proportion of negative examples that were correctly classified.

Answer 11

Specificity; True Negative Rate

Question 12

The __________________ is commonly used to examine the trade-off between the detection of true positives, while avoiding the false positives.

Answer 12

Receiver Operating Characteristic (ROC) curve

Question 13

The _______ treats the ROC diagram as a twodimensional square and measures the total area under the ROC curve.

Answer 13

Area Under the Curve (AUC)

Question 14

Most learners present performance measures during training. This is known as the __________ This metric is overly optimistic and cannot reliably measure future performance.

Answer 14

Resubstitution error

Question 15

The _______ method splits data into a ______ and ____ partition. At no time should the performance on the ____ dataset be allowed to influence the model

Answer 15

Holdout

Training

Test

Test

Question 16

Problems with the Holdout Method?

Answer 16

1. Each partition may have a larger or smaller proportion of some classes. This could lead to a class being omitted from the training data (resolved by stratified random sampling).
2. Some samples may have too many or few difficult cases, easy-to-predict cases, or outliers.
3. Substantial portions of data must be reserved to test and validate the model.

Question 17

A technique known as _________ is sometimes used to mitigate the problems with the holdout method. It uses the average result from several random holdout samples to evaluate a model’s performance.

Answer 17

Repeated Holdout

Question 18

_____ cross-validation randomly divides the data into ___ completely separate random partitions called ____

Answer 18

K Fold

K

Folds

Question 19

At the end of K-Fold Cross Validation, the __________ across all the folds is reported.

Answer 19

Average performance

Question 20

Some other Cross Validation techniques include:

Answer 20

5 Fold Cross Validation

Leave-one-out cross-validation

Random cross-validation

Stratified cross-validation

Question 21

Using sampling with replacement to form training set, _________ presents an alternative to cross-validation

Answer 21

bootstrapping

Question 22

Bootstrapping typically uses less data than crossvalidation for training, therefore, its test error will be _________

Answer 22

rather pessimistic