Evaluating Model Performance

Question 1

Describe Accuracy

Answer 1

accuracy = number of correctly identified instances in the class / all instances in that class

Question 2

Describe a Confusion Matrix

Answer 2

A matrix with the predicted class counts (correct / in correct) on the row elements and the actual class count (correct / incorrect) on the columns

Question 3

What is Recall?

Answer 3

TP / (TP + FN) - Recall in this context is also referred to as the true positive rate or sensitivity

Question 4

What is Precision?

Answer 4

TP / (TP + FP) - precision is also referred to as positive predictive value (PPV);

Question 5

What is the F1 score?

Answer 5

F1 = 2 * (precision * recall) / (precision + recall). The F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst at 0:

Question 6

What is the Mean Absolute Error?

Answer 6

The mean absolute error takes the total absolute error of each example and averages the error based on the number of data points.

Question 7

What is the Mean Squared Error?

Answer 7

the residual error - or that is the difference between predicted and the true value, are squared.

Some benefits of squaring the residual error is that it automatically converts all the errors as positives, emphasizes larger errors rather than smaller errors, and from calculus is differentiable which allows us to find the minimum or maximum values.

Question 8

In model prediction what are the two main sources of errors that a model can suffer from?

Answer 8

Bias due to a model being unable to represent the complexity of the underlying data or variance due to a model that is overly sensitive to the limited data it has been trained on.

Question 9

When does bias occur?

Answer 9

bias occurs when a model has enough data but is not complex enough to capture the underlying relationships. As a result, the model consistently and systematically misrepresents the data, leading to low accuracy in prediction. This is known as underfitting. It is an overly simplified model. High error on training set. In regression that would mean low r^2, large SSE

Question 10

When does variance occur?

Answer 10

When we train a model, we typically use a limited number of samples from a larger population (the training set). If we repeatedly train a model with randomly selected subsets of data, we would expect its predictons to be different based on the specific examples given to it. Here variance is a measure of how much the predictions vary for any given test sample.

Some variance is normal, but too much variance indicates that the model is unable to generalize its predictions to the larger population from which training samples were drawn. High sensitivity to the training set is also known as overfitting, and generally occurs when either the model is too complex and/or we do not have enough data to support it.

We can typically reduce the variability of a model’s predictions and increase precision by training on more data. Would mean much larger error on test set than on training set.

Question 11

What can happen more likely when you use too few features?

Answer 11

High bias. Might need several features to fully describe what’s going on in the data but might only be using a subset of the necessary features. Thus, overly simplified model and high bias. Think of it as highly biases to too few features.

Question 12

What are patterns to identify high variance?

Answer 12

Using many features or carefully optimizing performance to a training set.

Question 13

What is K-Fold Cross Validation?

Answer 13

Split data into k bin sizes, Run K separate training / test runs picking each bin once and training on the other k-1 bins. Then average test set performances from the k experiments.

Question 14

What is the curse of dimensionality?

Answer 14

As the number of features or dimensions grow, the amount of data we need to generalize accurately grows exponentially

Question 15

What is a learning curve in machine learning

Answer 15

A visual graph that compares the metric performance of a model on training and testing data over a number of training instances.

Question 16

What would indicate bias in a learning curve?

Answer 16

When the training and testing errors converge and are quite high this essentially means the model is biased. No matter how much data we feed it, the model cannot represent the underlying relationship and therefore has systematic high errors.

Question 17

What would indicate variance in a learning curve?

Answer 17

When we have a large gap between the training and testing error this essentially means the model suffers from high variance. Unlike a bias model, models that suffer from variance can generally improve if we have more data to learn from or we can simplify the model representing the most important features of the data.

Question 18

Describe an ideal learning curve.

Answer 18

When both the testing and training curves converge and where the error is extremely low. That is the model is very accurate on unseen data.

Question 19

What is a model complexity graph?

Answer 19

A model complexity graph looks at how the complexity of a model changes the training and testing curves rather than the number of data points to train on. The general trend of is that as a model increases, the more variability exists in the model for a fixed set of data.

Question 20

What is the Simpson’s paradox?

Answer 20

Simpson’s paradox, or the Yule–Simpson effect, is a paradox in probability and statistics, in which a trend appears in different groups of data but disappears or reverses when these groups are combined. Berkley discrimination example.