7 - Model Evaluation Flashcards by Kaman Hung

What is the primary focus of model evaluation in data science?

To evaluate the usefulness of models in making predictions.

How well did you know this?

Not at all

Perfectly

What is the difference between evaluation and validation?

Validation ensures consistency between training and test data sets, while evaluation measures accuracy and error rates.

How well did you know this?

Not at all

Perfectly

What type of models are discussed in the context of evaluation measures in this chapter?

Classification models, specifically decision trees.

How well did you know this?

Not at all

Perfectly

What is a contingency table?

A table that summarizes the performance of a classification model by comparing predicted and actual outcomes.

How well did you know this?

Not at all

Perfectly

Define the terms TN, FP, FN, and TP in the context of a contingency table.

TN: True Negatives
FP: False Positives
FN: False Negatives
TP: True Positives

How well did you know this?

Not at all

Perfectly

What does accuracy measure in model evaluation?

The proportion of correct classifications made by the model.

How well did you know this?

Not at all

Perfectly

What is the formula for calculating the error rate?

Error Rate = 1 - Accuracy.

How well did you know this?

Not at all

Perfectly

What do sensitivity and specificity measure in classification models?

Sensitivity: Ability to classify positive records correctly
Specificity: Ability to classify negative records correctly

How well did you know this?

Not at all

Perfectly

How is precision defined?

Precision = TP / TPP.

How well did you know this?

Not at all

Perfectly

What does recall measure?

Recall is another term for sensitivity, measuring the proportion of actual positives captured by the model.

How well did you know this?

Not at all

Perfectly

What are Fβ scores used for?

To combine precision and recall into a single measure.

How well did you know this?

Not at all

Perfectly

What does F1 score represent?

The harmonic mean of precision and recall, with equal weighting.

How well did you know this?

Not at all

Perfectly

What is the method for model evaluation?

Develop the model using the training data set
Evaluate the model using the test data set

How well did you know this?

Not at all

Perfectly

What is the target variable in the clothing data example?

Response, coded as 1 for positive and 0 for negative.

How well did you know this?

Not at all

Perfectly

What are the three continuous predictors in the clothing data example?

Days since Purchase
# of Purchase Visits
Sales per Visit

How well did you know this?

Not at all

Perfectly

What does the accuracy of Model 1 indicate?

Model 1 has an accuracy of 0.8410 or 84.10%.

How well did you know this?

Not at all

Perfectly

What is the baseline performance for the All Negative Model?

Accuracy = TAN / GT.

How well did you know this?

Not at all

Perfectly

What does a specificity of 0.9541 indicate about Model 1?

The model correctly classified 95.41% of actual negative records.

How well did you know this?

Not at all

Perfectly

What is indicated by Model 1’s sensitivity of 0.2804?

Only 28.04% of actual positive records were classified as positive.

How well did you know this?

Not at all

Perfectly

How is precision calculated for Model 1?

Precision = TP / TPP.

How well did you know this?

Not at all

Perfectly

What does the F1 score of 0.372 signify?

It reflects the balance between precision and recall for the model.

How well did you know this?

Not at all

Perfectly

What are the steps to perform model evaluation using R?

Develop Model 1 using training data
Run test data through Model 1

How well did you know this?

Not at all

Perfectly

What command is used to create a contingency table in R?

table() command.

How well did you know this?

Not at all

Perfectly

What command is used to create a contingency table in R?

table() command

How well did you know this?

Not at all

Perfectly

How do you add row names to a table in R?

Use row.names() function

What does the addmargins() command do in R?

Adds a Total row and column

What are the costs associated with True Negative (TN)?

What is the cost of a False Positive (FP) in the context of a clothing retailer?

$10

What is the cost associated with a False Negative (FN)?

What does True Positive (TP) represent in terms of cost?

−$40

What is the adjusted cost matrix for the lender as shown in Table 7.6?

CostTN = 0, CostFP = 1; CostFN = 4, CostTP = 0

What is the importance of accounting for unequal error costs in modeling?

To improve profitability and model accuracy

What is the formula for Overall Model Cost when error costs are unequal?

Overall Model Cost = FP Cost + TP Cost

When comparing models, what is the most important evaluation measure when error costs are unequal?

Model cost per record

What does Sensitivity measure in model evaluation?

Proportion of positive responders captured

What does Specificity measure in model evaluation?

Proportion of nonresponders correctly identified

What is the relationship between Accuracy and Error Rate?

Error Rate = 1 − Accuracy

How is the profit per customer calculated for a model?

Profit per Customer = Overall Model Cost / GT

What does the term 'data-driven error costs' refer to?

Costs derived from analyzing existing data rather than assumed values

What was the mean Sales per Visit used to update the CostTP?

$113.58

What does the term 'contingency table' consist of?

Actual categories vs. predicted categories

What happens to model accuracy when unequal error costs are considered?

Accuracy may not be the best metric for comparison

What is the effect of having higher sensitivity in a model with unequal error costs?

Leads to better profitability

What does the F2 score emphasize in model evaluation?

Favors recall (sensitivity) over precision

What is the cost associated with the All Negative model?

$0 profit

What is the profit per customer for Model 1?

$4.97

What is the profit per customer for Model 2?

$10.87

What is the profit per customer for Model 3?

$12.44

Fill in the blank: The retailer's false negative cost is ______ times higher than the false positive cost.

four

True or False: Model 1 had the highest accuracy but was the least profitable.

True

What should data scientists do to evaluate their models effectively?

Use metrics that consider unequal error costs

What is the formula for Error Rate?

Error Rate = 1 − Accuracy ## Footnote This formula provides a way to measure the rate of incorrect predictions in a model.

What do sensitivity and specificity measure?

Sensitivity measures the true positive rate, while specificity measures the true negative rate ## Footnote These metrics are crucial for evaluating the performance of classification models.

What is the Method for Model Evaluation?

The method involves comparing predicted values against actual values to assess model performance ## Footnote This typically includes metrics such as accuracy, precision, recall, and F1 score.

Why was the All Negative model chosen as a baseline for Model 1?

It provides a conservative baseline for accuracy calibration, as it always predicts the negative class ## Footnote This helps in understanding the performance of more complex models.

When error costs are unequal, what is the most important evaluation measure for model selection?

The most important measure is often the cost-sensitive accuracy or the expected cost ## Footnote This accounts for the different costs associated with false positives and false negatives.

How might a naïve analyst erroneously prefer Model 1 to Model 2?

By only looking at overall accuracy without considering error costs or class distribution ## Footnote This can lead to misinterpretation of model effectiveness.

What metrics need to be calculated for the All Positive and All Negative models?

Evaluation metrics including accuracy, precision, recall, and F1 score ## Footnote These metrics provide a comprehensive view of model performance.

What variables are used to create a C5.0 model (Model 1) for predicting customer Response?

Days since Purchase, # of Purchase Visits, and Sales per Visit ## Footnote These features are critical for understanding customer behavior.

What is a contingency table used for in model evaluation?

To compare actual and predicted values of a response variable ## Footnote This helps visualize the performance of the model.

What is the purpose of creating a cost matrix in model evaluation?

To specify the costs associated with different types of errors (false positives and false negatives) ## Footnote This allows for more informed decision-making based on financial implications.

What is the 4x cost matrix?

A cost matrix where a false positive is considered four times as bad as a false negative ## Footnote This reflects the differing impacts of errors in certain business contexts.

What should be included in the Model Evaluation Table for Model 2?

Overall Model Cost and Profit per Customer, along with other evaluation metrics ## Footnote This table helps in comparing multiple models effectively.

What is the purpose of the simplified data-driven cost matrix?

To reflect more accurate costs based on actual data rather than arbitrary values ## Footnote This enhances the decision-making process in model evaluation.

What is the significance of comparing evaluation measures from different models?

It highlights the strengths and weaknesses of each model and informs better model selection ## Footnote This comparison is essential for improving predictive performance.

What features are used in the C5.0 model for predicting a loan applicant’s Approval?

Debt‐to‐Income Ratio, FICO Score, and Request Amount ## Footnote These features are key indicators of a loan applicant's creditworthiness.

What does a contingency table for Model 1 predict?

It compares the actual and predicted values of loan Approval ## Footnote This aids in assessing the accuracy of the predictions made by the model.

What is the purpose of calculating the mean for the Interest per loan applicant?

To set the negative value as the cost of a true positive in the cost matrix ## Footnote This establishes a data-driven approach to evaluating the model's performance.

What should be done after building Model 2 for loan approval?

Populate the Model Evaluation Table with evaluation measures using the data-driven cost matrix ## Footnote This allows for a structured comparison with Model 1.

How can one quantify the financial impact of using data-driven error costs?

By calculating the total profit made by the bank based on the evaluation of the models ## Footnote This provides insight into the economic benefits of accurate model predictions.