B09 Evaluating Performance Flashcards
Exam Prep
The goal of evaluating a ___________ is to have a better understanding of how its performance will _______ to future cases.
Classification Model Extrapolate
A ____________ is a table that categorizes predictions according to whether they match the ________.
Confusion Matrix Actual Value
The class of interest is known as the ________ class, while all others are known as _______.
Positive Negative
The ____________ adjusts accuracy by accounting for the possibility of a correct prediction by chance alone.
kappa statistic
Kappa values range from ___ to _____
0 (poor) - 1 (good)
Kappa can also be though of as ______
Proportion of all instances where the predicted and actual values match.
The _________ is defined as the proportion of positive predictions that are truly positive. A model with high _______ is trustworthy.
Precision
The ________is a measure of the completeness of the results. A model with high _______has wide breadth.
Recall
The _______ combines precision and recall into a single number using the harmonic mean. It provides a convenient way to compare several models side by side.
F-Measure
The _______ of a model (also called the __________) measures the proportion of positive examples that were correctly classified.
Sensitivity; True Positive Rate
The __________ of a model (also called the ____________) measures the proportion of negative examples that were correctly classified.
Specificity; True Negative Rate
The __________________ is commonly used to examine the trade-off between the detection of true positives, while avoiding the false positives.
Receiver Operating Characteristic (ROC) curve
The _______ treats the ROC diagram as a twodimensional square and measures the total area under the ROC curve.
Area Under the Curve (AUC)
Most learners present performance measures during training. This is known as the __________ This metric is overly optimistic and cannot reliably measure future performance.
Resubstitution error
The _______ method splits data into a ______ and ____ partition. At no time should the performance on the ____ dataset be allowed to influence the model
Holdout
Training
Test
Test
Problems with the Holdout Method?
- Each partition may have a larger or smaller proportion of some classes. This could lead to a class being omitted from the training data (resolved by stratified random sampling).
- Some samples may have too many or few difficult cases, easy-to-predict cases, or outliers.
- Substantial portions of data must be reserved to test and validate the model.
A technique known as _________ is sometimes used to mitigate the problems with the holdout method. It uses the average result from several random holdout samples to evaluate a model’s performance.
Repeated Holdout
_____ cross-validation randomly divides the data into ___ completely separate random partitions called ____
K Fold
K
Folds
At the end of K-Fold Cross Validation, the __________ across all the folds is reported.
Average performance
Some other Cross Validation techniques include:
5 Fold Cross Validation
Leave-one-out cross-validation
Random cross-validation
Stratified cross-validation
Using sampling with replacement to form training set, _________ presents an alternative to cross-validation
bootstrapping
Bootstrapping typically uses less data than crossvalidation for training, therefore, its test error will be _________
rather pessimistic
